Keeping CALM: when distributed consistency is easy Hellerstein & Alvaro, arXiv 2019 The CALM conjecture (and later theorem) was first introduced to the world in a 2010 keynote talk at PODS. Beh… Article word count: 1444
Keeping CALM: when distributed consistency is easy Hellerstein & Alvaro, arXiv 2019 The CALM conjecture (and later theorem) was first introduced to the world in a 2010 keynote talk at PODS. Beh… Article word count: 1444
Keeping CALM: when distributed consistency is easy Hellerstein & Alvaro, arXiv 2019
The CALM conjecture (and later theorem) was first introduced to the world in a 2010 keynote talk at PODS. Behind its simple formulation there’s a deep lesson to be learned with the power to create ripples through our industry akin to the influence of the CAP theorem. It rewards time spent ruminating on the implications. Therefore I was delighted to see this paper from Hellerstein & Alvaro providing a fresh and very approachable look at CALM that gives us an excuse to do exactly that. All we need now is a catchy name for a movement! A CALM system is a NoCo system, “No Coordination.”
When it comes to high performing scalable distributed systems, coordination is a killer. It’s the dominant term in the Universal Scalability Law. When we can avoid or reduce the need for coordination things tend to get simpler and faster. See for example Coordination avoidance in database systems, and more recently the amazing performance of Anna which gives a two-orders-of-magnitude speed-up through coordination elimination. So we should avoid coordination whenever we can.
So far so good, but when exactly can we avoid coordination? Becoming precise in the answer to that question is what the CALM theorem is all about. You’re probably familiar with Brooks’ distinction between essential complexity and accidental complexity in his ‘No silver bullet’ essay. Here we get to tease apart the distinction between essential coordination, a guarantee that cannot be provided without coordinating, and accidental coordination, coordination that could have been avoided with a more careful design. In many cases, coordination is not a necessary evil, it is an incidental requirement of a design decision. One of the causes of accidental coordination is our preoccupation with trying to solve consistency questions at lower levels of the stack using storage semantics. There’s an end-to-end argument to be made that we need to be thinking about the application level semantics instead. The lower layers have a part to play, but we can’t unlock the full potential if we focus only there. As Pat Helland articulated in ‘Building on quicksand’, writes don’t commute, but application level operations can. It’s also only at the application level that we can trade coordination for apologies too.
Anyway, here’s the central question: What is the family of problems that can be consistently computed in a distributed fashion without coordination, and what problems lie outside that family? Monotonicity
Monotone speech. Monotony. Dull, boring, never a surprise or variation. Monotonic systems are similar, they only ever move in one direction. With monotonicity, once we learn something to be true, no further information can come down the line later on to refute that fact.
Consider deadlock detection in a distributed graph.
As the machines in the setup above exchange information about the edges they are aware of, then at some point the cycle involving T_1 and T_3 will come to light. No matter what other edges are later discovered, we know we have a deadlock. Deadlock detection in this system is monotonic.
Here’s another very similar looking distributed graph, but this time we’re interesting in reachability for the purposes of garbage collection.
More precisely, we’re interested in unreachability. Based on information to date, we may think that a certain object is unreachable. This property is not monotonic though – the very next edge we uncover could make it reachable again.
In the first instance, we were asking a “Does there exist?” ( \exists ) question. The presence of one positive example gives us an answer in the affirmative and additional positive examples don’t change that fact. In the second instance, we were asking a “Does not exist?” ( !\exists ) question. We can only answer a question like that once we’ve looked at every example. It’s the same for a “For all…” question ( \forall ).
With exactly the same graph, and exactly the same edge discovery algorithm, if we’d been interested in the property of reachability rather than unreachability, then that would have been monotonic. Once we know an object is reachable, finding out it is also reachable via a second path doesn’t change that fact. Unless… our system allows deletion of objects and edges. Then an object that was reachable can become unreachable again, and reachability is no longer monotonic.
What we’ve learned from these examples is that negation and universal quantification mess with monotonicity. We need the property that conclusions made on partial information continue to hold once we have the full information.
This idea of monotonicity turns out to be of central importance. It’s time to meet the CALM Theorem (Consistency as Logical Monotonicity): Consistency as Logical Monotonicity (CALM). A program has a consistent, coordination-free distributed implementation if and only if it is monotonic. The CALM theorem delineates the frontier between the possible (that’s the ‘if’ part) and the impossible (the ‘only if’ part). It’s most satisfying that this is also exactly the famous motivational poster message “Keep CALM and Carry On”. If you keep things CALM then you are always in a position to ‘carry on’ without needing to stop and coordinate.
Some more ‘C’ words
We’re making progress. We can now refine our original question to this: “What is the family of problems that can be computed in a monotonic fashion, and what problems lie outside that family?” So far we’ve been talking about CALM, consistency, and coordination. Now we need to introduce a couple more ‘C’ words: commutative and confluent.
Recall that a binary operation is commutative if the order of its operands makes no difference to the result. Addition is commutative, subtraction isn’t. Confluence as applied to program operations is a generalisation of the same idea. An operation is confluent if it produces the same sets of outputs for any non-deterministic ordering and batching of a set of inputs. Confluent operations compose: if the outputs of one confluent operation are consumed by another, the resulting composite operation is confluent. Hence confluence can be applied to individual operations, components in a dataflow, or even entire distributed programs. If we restrict ourselves to building programs by composing confluent operations, our programs are confluent by construction, despite orderings of messages or execution races within and across components. Confluent operations are the building blocks of monotonic systems. We still need to take care to avoid negation though (deletions, !\exists, and its alternative formulation: \forall). One creative solution for dealing with negation / deletions, as used for example by CRDT sets, is to keep a separate growing set of deletions alongside additions. In relational algebra terms we can have selection, projection, intersection, join, and transitive closure, but not set-difference. In terms of mutable state we can allow inserts, but not updates and deletes. The key insight in CALM is to focus on consistency from the viewpoint of program outcomes rather than the traditional histories of storage mutation. The emphasis on the program being computed shifts focus from implementation to specification: it allows us to ask questions about what computations are possible. CALM in practice
CRDTs provide an object-oriented framework for monotonic programming patterns. We really want to use them within a functional programming context though, or at least one that avoids bare assignment to mutable variables. Immutability is a trivial monotonic pattern, mutability is non-monotonic. Going further, the Bloom language was explicitly designed to support CALM application development: \* Bloom makes set-oriented, monotonic (and hence confluent) programming the easiest constructs for programmers to work with in the language. \* Bloom can leverage static analysis based on CALM to certify when programs provide the state-based convergence properties provided by CRDTs, and when those properties are preserved across compositions of modules. If you can’t find a monotonic implementation for every feature of an application, one good strategy is keep coordination off the critical path. For example, in the garbage collection example we looked at earlier, garbage collection can run in the background. Another option is to proceed without coordination but put in place a mechanism to detect when this leads to an inconsistency so that the application can “apologise”. The CALM Theorem presents a positive result that delineates the frontier of the possible. CALM shows that monotonicity, a property of a program, implies consistency, a property of the output of any execution of that program. The inverse is also established: non-monotonic programs require runtime enforcement (coordination) to ensure consistent execution. As a program property, CALM enables reasoning via static program analysis, and limits or eliminates the use of runtime checks. There’s plenty more good material in the paper itself that I didn’t have space to cover here, so if these ideas have caught your interest I encourage you to check it out.
Stress is such a commonly used word, but what does it really mean? How does stress affect us and what can we do about it? Stress can be the cause of so many health conditions, lost days of work, irritability with those we love, and general unhappiness. Certain herbs, vitamins, minerals and techniques can help calm your body so you are better able to deal with the stresses that come your way. Some of the more common symptoms of stress are: fatique, headaches, irritability, stomach aches, heartburn, rapid heart rate, high blood pressure, cramping, indigestion, insomnia, food reactions, hives, weight gain, weight loss, and lowered immune function. Stress can effect any one in any part of the body as we all deal with life differently; we all hold our stress in different parts of our body. Some supplements that can be useful for stress are: The B vitamins (B complex), vitamin C, Calcium-Magnesium, and EFA’s (essential fatty acids). EFA’s are oils like, flax, fish and 3-6-9 oil; they are the "good fats". Many essential nutrients are depleted when we are stressed, especial... Show more...
Stress is such a commonly used word, but what does it really mean? How does stress affect us and what can we do about it? Stress can be the cause of so many health conditions, lost days of work, irritability with those we love, and general unhappiness. Certain herbs, vitamins, minerals and techniques can help calm your body so you are better able to deal with the stresses that come your way. Some of the more common symptoms of stress are: fatique, headaches, irritability, stomach aches, heartburn, rapid heart rate, high blood pressure, cramping, indigestion, insomnia, food reactions, hives, weight gain, weight loss, and lowered immune function. Stress can effect any one in any part of the body as we all deal with life differently; we all hold our stress in different parts of our body. Some supplements that can be useful for stress are: The B vitamins (B complex), vitamin C, Calcium-Magnesium, and EFA’s (essential fatty acids). EFA’s are oils like, flax, fish and 3-6-9 oil; they are the "good fats". Many essential nutrients are depleted when we are stressed, especially Magnesium. This mineral helps to calm the muscles and nerves, making you less jumpy, less quick to anger, while reducing cramping and chocolate cravings. There are many great herbs you can take for stress: St John’s Wort, Kava Kava, 5-HTP and more; do not take any of these three if on an anti-depressant or anti-anxiety medication. Homeopathic remedies can also work well. Remedies such as, Calms Forte, Rescue Remedy by Bach Flower are good for calming during a crisis; Coffea Cruda works to help shut off your mind so you can sleep. There are many techniques we can use for stress: deep breathing, time out, counting, walking, a hobby, talking to a friend, exercise, watching a funny movie, dancing, singing and meditation. Deep breathing is practiced by expanding the whole torso while breathing in, filling the lungs with air, and then releasing it slowly. Take a class or read a book to help learn a hobby, meditation or an attitude change. Learn to release anger in a healthy productive way. Get a pet. Spending time with a pet has been shown to lower blood pressure, increase immune and a sense of well being. A cat’s purring is on the right decibel for healing, yet we still don’t understand how they purr. Watch your diet. Too much caffeine, sugar and white flour products (which immediately turn to sugar in the body) can make you tense and irritable. Going without food for hours causes your blood sugar to drop, making you light headed, nauseous and irritable. Eat more fruits, veggies, whole grains, efa’s and good quality proteins. Drink more pure water. Our genetics determine the form of disease and what part of the body is affected. Diet, and the amount of stress and how we let that stress affect us, determines whether we develop that disease. If we cannot change the stress, we have to learn how to live with it. You can learn to be happy while dealing with the everyday stress of life.
The two smartphone apps taking meditation mainstream exude a Zen vibe and trumpet nearly identical missions to increase health and happiness around the world.
But behind the blissful marketing mantras, Headspace and Calm are locked in a head-to-head fight to dominate the booming $1.2 billion meditation market.
“I would say we’re in mindful competition with each other,” says Michael Acton Smith, co-CEO and co-founder of Calm, based in San Francisco. “And they have six times as many employees as we do.” Calm has a staff of about 40, while Headspace employs 230.
“The irony is not lost on me,” says Rich Pierson, CEO and co-founder of Headspace, of the rivalry with Calm. But he’d rather focus on his Santa Monica, Calif.-based company’s authenticity, he says, which is drawn from co-founder Andy Puddicombe’s 10 years of studying meditation at Buddhist monasteries.
Andy Puddicombe, co-founder of meditation and mindfulness app Headspace, spent a decade studying meditation at Buddhist monasteries.
Andy Puddicombe, co-founder of meditation and mindfulness app Headspace, spent a decade studying meditation at Buddhist monasteries. Photo: Patrick T. Fallon/Bloomberg News
“If you were going to see a psychologist, you’d probably want to know where they trained and qualified. It’s the same with meditation,” Mr. Pierson says. Calm’s founders previously worked in online gaming and advertising.
Both startups are venture-backed, founded by charismatic British guys who moved to California. Both practice what they preach, offering officewide daily meditation sessions. And both apps have been downloaded more than 38 million times, with each hitting 1 million paid subscribers in June, the companies say.
“Headspace launched two years before us, and now we’re neck and neck,” says Dun Wang, Calm’s chief product and growth officer.
Founded in 2010, Headspace had dominated the meditation category until this year, when Calm caught up.
Calm got a boost from winning the 2017 iPhone App of the Year award from Apple ’s App Store last December.
“Since winning App of the Year, we seem to have a much higher growth rate than they do, and we’ll surpass them from now on,” Calm’s Ms. Wang says.
Calm has topped the category in both downloads and mobile revenue since last December, with revenue through October at $50.7 million, according to estimates from mobile-data firm Sensor Tower. Headspace, now in second place, saw revenue of $34.3 million, according to Sensor Tower. Both offer standard subscription rates of $13 a month. Annual subscriptions cost $96 a year at Headspace and $70 a year at Calm.
Michael Acton Smith, co-CEO and co-founder of meditation and mindfulness app Calm, at his company’s San Francisco headquarters.
Michael Acton Smith, co-CEO and co-founder of meditation and mindfulness app Calm, at his company’s San Francisco headquarters. Photo: Calm
A Headspace spokeswoman says Sensor Tower’s data didn’t account for subscriptions that come through its website and corporate partnerships, which would push its 12-month revenues “north of $100 million.” The company declined to provide year-over-year comparisons. Calm didn’t dispute Sensor Tower’s data.
The award is a touchy subject at Headspace. Apple does not reveal its selection methods or criteria.
“I think people are blowing App-of-the-Year way out of proportion,” says Ben Spero, a managing director at Spectrum Equity, a Headspace investor. “It’s good P.R., but it’s not determinative. It’s not that Apple was saying that Calm is a better app—they’re big fans of Headspace, too,” he says, pointing out that the App Store often features Headspace on its landing page. Apple declined to comment on the 2017 award.
The meditation industry—including studio classes, workshops, books, online courses and apps—is worth about $1.2 billion and growing, according to a 2017 estimate by Marketdata Enterprises. Studies show meditation can reduce stress and improve sleep.
Headspace vs. Calm: The Meditation Battle That’s Anything but Zen
The national meditation rate has tripled over the past five years. The number of U.S. adults meditating is on track to surpass the number of those practicing yoga in the next two years, according to an October report from the Centers for Disease Control and Prevention.
Businesses are eager to help employees tap into their higher consciousness in the hopes that it will improve productivity and reduce stress-related health-care costs. About 22% of companies started offering mindfulness meditation training in 2016, and another 21% planned to add it in 2017, according to a survey by The National Business Group on Health and Fidelity Investments.
All the attention has fueled a digital bonanza, with more than 2,000 new meditation apps launched over the past three years. Headspace and Calm account for about 85% of the revenue generated by the top 10 apps in the category, according to Sensor Tower data.
Headspace uses quirky cartoons to walk the user through meditation sessions, all guided by Mr. Puddicombe. In addition to programs for stress and anxiety, it offers sessions for sleep, work/productivity and fitness.
Calm lets listeners choose from several different meditation teachers and peaceful nature scenes. About half of Calm’s users use it primarily as a sleep app rather than a meditation app, the company says. They tune in to Calm Sleep Stories, which are bedtime stories narrated by actors including Matthew McConaughey, and Calm Music, which offers curated musical playlists.
Headspace is pushing for growth through corporate partnerships, last year hiring chief business officer Ross Hoffman away from Twitter , where he led global content partnerships.
Headspace launched Mindful Cab, a program of in-cab meditations, in November.
Headspace launched Mindful Cab, a program of in-cab meditations, in November. Photo: Getty Images
“We’re just fishing where the fish are,” Mr. Hoffman says. Over the past 18 months, he has signed deals with more than 300 companies to integrate Headspace subscriptions into their employee health and wellness benefits, including Unilever , Adobe , Genentech and LinkedIn.
Last March, Headspace announced a deal with the National Basketball Association to become the league’s official mediation and mindfulness partner. It involves branded content, promotions and about 7,000 subscriptions for players and staff, but no money exchanged hands, Headspace says.
Calm has largely ignored the business market, declining most corporate requests until recently, when some major companies started calling. Calm executives say such deals don’t add substantial revenue. Corporations typically negotiate bulk purchasing discounts of up to 50% off standard subscription fees.
“Calm has managed to overtake Headspace because they’ve been laser-focused without getting distracted by the types of vanity partnerships, like with sports teams, that look good in the press but don’t move the needle,” says Harley Miller, principal at Insight Venture Partners, a Calm investor.
Headspace responds that all its partners serve a strategic purpose. Seeing professional athletes practice mindfulness and meditation with Headspace will encourage young aspiring athletes to do the same, Mr. Hoffman says, especially with stars like Kevin Love of the Cleveland Cavaliers and retired league MVP Steve Nash extolling its benefits. (Neither is a paid spokesman.)
“We have the strongest brand,” Mr. Hoffman says, pointing out that morning talk shows and late-night comedy hosts regularly turn to Headspace’s Mr. Puddicombe to teach viewers about meditation.
Early reluctance aside, Calm is expanding into the corporate arena, often on the heels of its biggest competitor.
“Companies will approach us and say, ‘We got this offer from Headspace, do you guys want to put in an offer as well?,’ ” Calm’s Ms. Wang says. “It’s working out great for us. We don’t need to be the ones making the cold calls to win these projects.”
PricewaterhouseCoopers and Samsung were among Calm’s first corporate clients.
In October, American Airlines became its first airline partner. (Headspace already has content deals with 11 airlines, including JetBlue , Delta and United.) Calm’s soothing nature videos will play on all seat-back screens during boarding, putting the brand in the faces of more than 200 million passengers a year, according to the carrier.
Headspace isn’t taking this lying down. Its Headspace Health division is currently seeking Food and Drug Administration approval to become the first prescription meditation app for certain chronic illnesses. FDA approval would pave the way for Headspace programs to be covered by health insurance.
Calm’s leadership team is focused on achieving mass mindfulness, and on victory.
“We’re both growing the sector. This is such a huge market, there’s going to be room for a few different winners,” Mr. Acton Smith says. “We want Calm to be the number one, of course—and we wish the others the best for second place.”
Corrections & Amplifications Calm’s deal with American Airlines will put the company’s nature videos before more than 200 million passengers annually. An earlier version of this article incorrectly estimated the figure as 800 million. (Dec. 15, 2018)
over the last couple of years i’ve been making really really really quiet music to listen to when i do yoga or sleep or meditate or panic. i ended up with 4 hours of music and have decided to give it away.
Two New Jersey police officers -- who remained calm and professional while a former Hillary Clinton financial adviser ranted and raved during a routine traffic stop last month -- are being lauded for their “verbal judo” during the tense interaction.