upGrad helped me understand the importance of positioning. I wrote about Positioning on Growth Catalyst last year in this post. From the earlier post,

Once you have figured out the market need and the product that solves for it, the next challenge becomes how to position it into the market. If you are building reusable rockets 🚀 like Elon Musk, this isn’t a big challenge because you have virtually no competition.

But more often than not, you are building a better version of something that already exists. Or if you are working in a bigger company, you are trying to launch something similar with the assumption that a stronger distribution channel or 💰 will make your product successful. That is where positioning helps.

I was still puzzled though — there are companies doing pretty well while expanding beyond their initial positioning. There are numerous examples of that. The Tata Group definitely has been able to create a trustworthy brand and added various lines of products based on that. A more recent example would be Zomato, which started as a restaurant discovery app. Over time, it added delivery and the Zomato Gold subscription. Zomato has been doing well in those segments. It is no more a restaurant discovery app, which was the initial positioning of the app. There are other food delivery apps (Swiggy), and dining subscription apps (DineOut). But Zomato has been able to enter these adjacent markets and capture a sizeable share.

A good explanation for this phenomenon is that as consumers associate Zomato with food, they can do various things around food like subscriptions, delivery, etc. The same can be applied to other products. However, this theory breaks when we apply it to another giant, Amazon.

Amazon started as an online bookstore. Over the years, it expanded into an online marketplace. So far, the theory holds true. Then Amazon entered into Cloud with AWS in 2006 and has been doing various other things since then like Prime Video, Alexa, etc. The positioning of Amazon has transformed from an online store to a brand that does things in a certain way using its 16 leadership principles.

So the question remains — how does a single product with a strong positioning goes on to do things successfully that don’t fit its initial positioning? To answer this question, we will have to understand the concept of a Wedge.

The Concept of a Wedge

A wedge is an object with a pointed edge and a thick edge. A wedge is used to cut or split apart objects. Force is applied to the thick end of the wedge, and the wedge, in turn, applies force to the object along both of its sloping sides. This force causes the object to split apart.1

What happens if both ends of the wedge are thick? You won’t be able to split the object.

The concept works in a similar way when applied to products and markets. The initial product can be thought of like a wedge to penetrate the market. The force applied is the resources you have like the team, investments, product, etc. The thin edge of the wedge represents focus on a few use-cases to start with. If you go after the whole market, you won’t be able to make a difference because of the lack of focus. Once you create a product loved by a segment of the market, it’s easier to expand to adjacent use-cases.

Going back to Zomato’s example, the discovery of restaurants before Zomato was difficult. Zomato came and solved the problem. After doing that, Zomato could do various things like food delivery and subscriptions in the market because it had a strong team, a better understanding of market problems, and investments to fund projects.

In the same space, Swiggy entered by picking a different product wedge - delivery warm food to people sitting in the comfort of their homes. Over time, Swiggy has expanded into other delivery services. It launched Instamart and has started delivering groceries. It also launched Swiggy Genie, using which you can send specific items from one place to another.

By now, you might have gotten a good sense of wedge. It does not look quite different from positioning. So why do we need to introduce a new term?

Positioning Constrains, Wedge Liberates

The positioning is how end-users remember you. To retain your positioning, you might not want to do various things that confuse the users and dilute that positioning. That way, the concept of positioning constrains one’s thinking and might hurt the chances to win the market in the long run. A case in point here is the battle between Swiggy and Zomato. If Zomato had fixated itself as the restaurant discovery app, Swiggy could have won the delivery market and then entered into restaurant discovery as well since they have the required data from customers and restaurants to build the restaurant discovery. However, Zomato entering into food delivery helped them push off the competition by creating a stronger relationship with consumers while generating an additional source of revenue.

The concept of the wedge, on the other hand, allows you to think about other ways to gain/expand the business. If you think of the initial product as a wedge, you can be focused on that use-case until you hit a certain market share. Post that, you can naturally start thinking about how to build for adjacent use-cases and bring more users or revenue. From the previous examples, Amazon did so by continuously expanding into various markets, and so did Zomato and Swiggy.

Picking a Wedge

Starting is the difficult part, be it a startup or anything. Starting to work on the right thing is even more difficult. Picking the right wedge for your target market can make a lot of difference. So it’s important to spend some time there.

There are two major criteria to pick a wedge:

1. Find a value proposition which a customer segment needs

2. Find the constraints within which you and your team can deliver that value proposition

Most people when starting to build a product stop at #1. Let’s take an example and understand why both criteria are important. Before Swiggy started with food delivery, other players like TinyOwl, FoodPanda, Zomato, etc. had tried their hands on food delivery with no success. The value proposition from all of them was the same, but only Swiggy succeeded. The reason for this? Other players connected customers to restaurants but “outsourced" delivery to either restaurants or third-party logistics providers. This had a couple of problems. First, the cost of logistics was hard to reduce over time as it was outsourced. Second, the end-user experience was difficult to standardize since you didn’t have enough control over the logistics.

Swiggy was convinced that the only way to crack the food delivery market was to build an extensive logistics network, with a vast fleet of their own delivery boys and technology to back them. Swiggy also invested in a large sales force to bring and keep popular restaurants onboard. It also invested in building technology to make its logistics network more efficient. In short, they found the right value proposition and delivered the right solution to it. It’s important to do both.

Constraints to Find the Right Wedge

There are four constraints you can think about to create a wedge:

1. Geography

2. Category

3. Price

4. Behavioral segment

Geography — Starting in particular geography helps you gain leadership in that geography. It can become pronounced especially when the difference between two geographies is enhanced. That is the reason different companies are market leaders in different geographies. Here is the meal delivery market share of the online food delivery companies.

Geography isn’t a strong wedge as a better product can any day come and take over the market.

However, geographic constraint plays a crucial role in creating two-sided marketplaces. Swiggy started operating within a small region called Koramangala in Bangalore, rather than going to the whole of Bangalore. Uber constrained itself in New York before expanding itself. Expanding too fast in a two-sided marketplace can create problems with demand-supply matching.

Category — Amazon started with books, and then moved to sell everything online. A lot of other eCommerce companies started with fashion and then started selling other items. The category is a good constraint since focussing on a particular category can lead to cost leadership, differentiation, or both. Therefore, it’s hard for some other competitors to come and build a better experience.

Price — Offering something valuable for free, while figuring out something else to sell is another good wedge. A lot of disruptions start this way. Jio gave voice calling for free in India which was a paid plan for all other players. It focussed on monetizing through data and internet plans. It gave Jio a strong wedge in an already overcrowded telecom market.

Behavioral segment — Our society contains various types of people. For example, there is a vast difference in the way people think and behave in rural and urban areas. Going after an underserved segment is another important constraint you can put. An example of this is apps built for Bharat (tier II+ audience which doesn’t understand and speak English). Sharechat created Facebook for the vernacular, Bharat audience in India.

By now, we have understood the importance of wedges, and how to find the right wedge. We will conclude this post by covering the last topic - the right time to expand in a market.

The Right Time to Expand

One question that often appears once you start thinking about wedge and positioning: when is the right time to expand use-cases? The answer to this particular question is whenever you are done gaining a leadership position in the first set of use-cases and have significant gains to a distant second. If you haven’t gained the leadership in your initial market, going after other use-cases will lead to defocusing in the initial market. You might lose your positioning and market share in the mid-to-long run.

One startup in the Indian ecosystem that recently expanded beyond its initial positioning successfully is Meesho. Meesho started in 2015 by creating a new category of commerce in India - social commerce. It enabled small businesses and individuals to sell fashion and accessories within their network via social channels such as WhatsApp, Facebook, Instagram. Meesho created the inventory and logistics infrastructure for resellers. The resellers could gain a commission for the items they sold. The social commerce aspects of Meesho were clear from its play store description.

By 2021, it became a market leader in the social commerce space. According to a note by Bernstein released on September 27, 2021, Meesho was India's largest social commerce platform with over 13 million resellers, 45 million customers, and over 1 lakh suppliers.2 Most recently, Meesho raised $570 million from Fidelity, B Capital, others at a $4.9 billion valuation. Other social commerce platforms like GlowRoad have raised money at a valuation between $50-100 million and Meesho is far ahead of its competitors.

After gaining the leadership position in social commerce, it has now moved to a far broader market. Meesho wants to target 100 million small businesses and has started selling direct to customers, a plan that puts it against online retail majors Amazon and Flipkart in India.3 It is now changing its positioning from a reseller model to a B2C marketplace. Here is how the play store description of Meesho looks right now.

As you can see, its current positioning and model is similar to other eCommerce apps like Flipkart and Amazon :)

If you liked the post, subscribe now to receive regular posts from the Growth Catalyst.

Subscribe now

Have a good day,

Deepak

Twitter | LinkedIn

Subscribe now

Dear Reader,

It has been a month since my last post. Few readers reached out to ask whether I would continue writing. The worry is understood — many aspiring writers start dreading writing once the initial romance with the idea of writing starts wearing off. You can expect regular posts hereon because 10 months in, I am still in love with writing.

Before we get to the post, I request you to spend 2 mins on this survey form. This is very important for readers as the survey will help me create the writing roadmap this year. I got this suggestion from a good friend, Kali who is also a regular reader of my posts, thank you 🙏🏻

Enough chit-chat; let's get on to the post.

The retention of the product is a function of the user's repeat behavior, also called habits. Almost all of us have tried to build healthy habits by signing up for gym memberships or otherwise. Most of us also failed. The outcome shows that repeat behaviors are the hardest to form, which makes retention one of the hardest problems to solve.

Luckily, there are ways to manage and improve retention. I have already written a post around frameworks to fix retention. If you haven't read the post, I would recommend reading it. The post covers Fogg's behavior model, super apps, and hooked.

This post is primarily an extension of that post. We will cover

• How habits get built

• Switching costs and why they are so powerful in creating habits

• Time as an added dimension to Fogg's behavior model

The Science of Habit Building

Many years back, I read the book The power of habit by Charles Duhigg and absolutely loved it. Duhigg wrote about research done around habit building at MIT in the book. The researchers discovered that a simple neurological loop exists at the core of every habit, consisting of three parts: a cue, a routine, and a reward.

For example,

Cue or trigger — getting bored

Routine or action — opening Instagram

Reward — seeing the likes on your post, new photos from friends

The loop keeps repeating day after day every time you see a cue/trigger.

Researchers further realized that when the rewards are variable, i.e., they change every time you do the routine, the habits are more strong. It's like you opening Instagram multiple times after submitting a photo because you can't anticipate the # of likes and keep seeing different # of likes (variable rewards).

If you wonder why variable rewards create stronger habits, it has to do with what happens in your brain when you do the same action and experience variable rewards.

When the brain gets an unexpected reward, it releases dopamine. On Instagram, the momentary rewards come from the likes, comments, good photos, etc. But we can't predict the size of the reward nor the frequency of the reward. This means that you get a dopamine hit every time you open Instagram. Over time, you develop a dopamine dependency and keep coming back for the dopamine hit. For that reason, we get addicted to social media.

Another industry that employs variable rewards is casinos. Throwing dice and getting unpredictable monetary rewards gets gamblers addicted. They want to get the dopamine hit again and again, and that drives them to casinos.

How exactly do products use the science of habit building? We have read/talked about how social media uses the habit science to get people addicted. Let's talk about something different. From the Duolingo blog,

In January 2020, we tested three push notifications to encourage learners to make language learning their focus for the year. The winner?

That beat out "Reach your goals! Keep your resolution going with a lesson."

"Habit" also makes a difference in how learners receive those push notifications. Duo reminds you to practice — but only if you ask him to. Previously, the opt-in screen below focused on getting reminders in order to keep your streak, but when we tested copy around building a long-term habit, 5% more learners opted in.

Finally, there's the screen after you complete your first lesson. (Way to go!) After celebrating this milestone, we encourage learners to come back for more. Previously, this screen said "This starts your streak! Keep it going tomorrow!"

But when we tested "Reach a 7 day streak to start building a daily habit," learners came back in greater numbers. That’s what I call a win-win — and the start of a strong habit.

The cue in Duolingo could be a push notification, someone speaking in a different language, etc. The routine is opening the app and completing the lesson. The reward is creating or maintaining the streak.

Is there a way Duolingo introduces variable rewards and even creates a stronger habit loop? It can do that by making learning a social construct.

An app that does that well is Strava. Strava is a fitness app. You can add your social circle on Strava and see the fitness activities they are logging in your feed. You can give them likes/kudos for that. Over time, people would come back to see who gave them kudos and also keep doing physical activities to create a better social image.

If you are building a B2C product for daily/weekly usage, think about your product's cue-routine-reward loop. If not, wait for the next two sections.

Switching Costs

Switching costs play an important role when the product solves a particular need, and multiple products solve the same need. Switching costs ensure that the users, once acquired, will keep using your products and won't switch to your competitor.

One of the strongest switching costs is the network effect that we frequently see in social apps. As the network of your friends grows in a particular network, it's hard to leave it. Think WhatsApp, Insta, Snap, FB.

Beyond social products, other apps build switching costs differently. There are many different ways to build switching costs. If you need a framework to think about it, you can think about four factors — time, effort, psychological, money. A combination of these 4 creates switching costs. Here are some examples:

1. Data: Leaving one platform for the other forces users to let go of data or activity. Users don't want to lose that, and that creates a high switching cost. Depending on the use case, data can hit high on time, effort, and psychological factors.

   If you switch from Spotify to another music app, you'll lose your playlists.

   Android or Apple users will have to give up their purchased music tracks, apps, or movies if they want to move to another platform.

2. Learning Curve: If there is a learning curve for a product line, users will find it hard to switch to another product. It often happens with complex, enterprise products. The learning curve hits high on time and effort factors.

   Salesforce is a prime example of this. Teams once learning how to operate with Salesforce find it hard to switch to another software.

3. Brand/ Industry Standard: This one takes time and happens in both B2B and B2C companies. It hits high on psychological.

   In the 70s, the early IT industry projects often exposed decision-makers to the chance of losing their jobs because they would be held accountable for failure. Since IBM's size and quality in those days would guarantee a project's outcome, it seemed a logical, safe choice to go with the giant. Hence, the quote "Nobody ever got fired for choosing IBM" got popular.

   In B2C, Apple is a great example of it. Apple means premium and quality to a lot of users.

4. Exit costs: If the customer wants to exit the contract before time, he/she has to pay early termination fees. Telecom operators such as AT&T or Verizon used to charge up to $350 in early termination fees. Over time, most major carriers have eliminated the 2-year contract for consumers, so early termination fees (ETF) are quickly becoming a thing of the past. Exit costs hit high on the money factor.

   This is also seen in enterprise software where salespeople push for annual contracts, and there is a termination fee. It's no customer-friendly, and new-age SaaS subscription models are removing that.

5. Social Status/Influence: The rise of the Influencer industry has led to another major addition in switching costs. Once you have made your mark on a platform, you don't want to leave the platform and risk losing it. Examples — followers on Twitter, subscribers on YT, etc.

Time as an added dimension to B=MAP

I wrote extensively about B=MAT in the post, Simple Frameworks for Fixing Retention

From the post,

Let’s start with the basic question — what causes a particular behavior? The model that explains it is developed by BJ Foggs, who built the Behavior Design Lab at Stanford University. He and his team research human behavior and how you can influence it to achieve business goals. 

The Fogg Behavior Model shows that three elements should converge simultaneously for a behavior to happen: Motivation, Ability, and a Prompt. When a behavior does not happen, at least one of those three elements is missing.

With ample motivation and ability, the prompt/trigger shapes our behavior.

Once we start applying B=MAP to affect user behaviors, it becomes easier to find ideas of growth, conversions, etc. However, we often miss one important factor that affects retention most — time.

Motivation, ability, and our response to triggers change over time.

Take the example of MOOCs — the completion rate for MOOCs is quite low. Here is what the MIT research study around it says,

 One of the big knocks against MOOCs since their beginning was the low rate at which students completed the courses, even as defenders pointed out that many students took MOOCs for knowledge or edification, rather than for a credential. The critique stuck nonetheless.

Among all MOOC participants, 3.13 percent completed their courses in 2017-18, down from about 4 percent the two previous years and nearly 6 percent in 2014-15.

If we think about it, MOOCs have solved the ability problem. You can attend a MOOC any time, anywhere in the world, as long as you have a smartphone/laptop. People also have ample motivation to learn something that can help accelerate their careers. Once you start the course, MOOCs also send you timely notifications. So why is the completion rate so low?

Because motivation and ability vary over time, so a typical graph of mine used to look like this during my college semester.

Can MOOCs solve the problem? Yes. One simple way was to add payment and certification, which increases the students' motivation to complete. Among the "verified" students, 46 percent completed in 2017-18, compared to 56 percent in 2016-17 and about 50 percent in the two previous years.

Another way is to make the course limited time. A limited-time course creates a sense of urgency throughout the course. This is a reason why cohort-based courses are picking up. A live classroom with a community and limited time to complete has better completion rates. Add to that high fee the cohort bases courses charge makes exit costs high.

Throughout the user journey, we should be on a continuous lookout to add motivation, ability, and prompts over time. Solving it for once won't be enough.

With this, I will wrap up the post. Before you go, please fill this survey if you haven't already.

Thanks,

Deepak

Twitter | LinkedIn

Subscribe now

Conversion is a function of what people read and see on your page. From that perspective, three factors become very important for driving conversion.

The first one is confirmation bias which is our tendency to find information that confirms our existing beliefs/hypothesis. Confirmation bias massively affects the strategic decisions we take.

The second one is the information that can help people make decisions. The information can be shared through any format, be it text, images, videos, etc.

The third factor is copywriting, i.e. writing the text of product and marketing pages. Take it from David Ogilvy, one of the top marketers of all time

“When I write an advertisement, I don’t want you to tell me that you find it ‘creative.’ I want you to find it so interesting that you buy the product.”

We will discuss all three factors in this essay.

Confirmation Bias - The Mother of All Biases

Confirmation bias is our tendency to find information that confirms our existing beliefs/hypothesis. Here is an illustration of confirmation bias.

I have been there, have you? :)

Confirmation bias is especially strong for emotional issues or deep beliefs like those around politics and religion.

A lot of people I meet have heard about this bias somewhere, and they sometimes use it without fully understanding the power of confirmation bias. To apply it well in product management and growth, it’s important to understand it well.

So let’s start by discussing how confirmation bias happens. There are three ways it happens:

1. Biased search for Information - If you were to search “Are cats better than dogs?” in Google, you will get sites listing the reasons why cats are better.

   

   However, if you were to search “Are dogs better than cats?” Google will provide you the sites listing why dogs are better than cats.

   

   We should always keep this in mind when you researching on Google. Mind your words and the way you frame the sentences. On that note, Google should probably work on removing this bias :)

2. Biased Interpretation - We can interpret the same sentences in our own ways. An experiment conducted by Stanford University on the topic of capital punishment included both kinds of participants — those in support of and others against capital punishment. Scientists provided all subjects the same two studies. After reading the studies, participants still held their initial beliefs. They supported their reasoning by providing “confirming” evidence from the studies and rejected any contradictory evidence, or considered it inferior to the “confirming” evidence (Lord, Ross, & Lepper, 1979).

3. Biased Memory - We remember/recall information selectively that supports their beliefs. In one experimental study, participants were asked to read an extensive account of events in 1 wk of the life of a woman named Jane. Two days later, they used this previously learned information to test hypotheses about Jane's suitability for 1 of 2 jobs: either the rather extraverted job of real estate salesperson or the rather introverted job of a research librarian. Those assessing her as a salesperson better recalled extroverted traits while the other group recalled introversion traits (Snyder & Cantor, 1979).

Reading around confirmation bias makes it sound bad, but it’s not all that bad. We need confirmation bias in our lives to operate. Confirmation bias is a reason why we love music so much. Daniel Levitin wrote in This Is Your Brain on Music

As music unfolds, the brain constantly updates its estimates of when new beats will occur, and takes satisfaction in matching a mental beat with a real-in-the-world one.

Confirmation bias also helps us in processing information efficiently in our everyday lives. We are also exposed to limitless information through the internet nowadays. We can’t process it all if we had to analyze all the pieces of information.

It also protects our self-esteem and makes us feel confident by reassuring us around our beliefs. We can’t always keep second-guessing ourselves, it’s not practical.

In a way, confirmation bias is a two-edged sword. Being aware of this bias helps to throw away bad ideas.

Where does it have to do with conversion?

So far we have discussed what confirmation bias has to do with us on a personal or professional level. How does it help in conversion?

Let’s assume for a minute that we don’t understand the audience, their beliefs, and their assumptions well. It will be hard to create a pitch that they can relate to in that case. It can backfire as well if we end up challenging their assumptions/beliefs where they leave without hearing the whole thing.

On the other hand, if you understand your audience well, and your message sits right with them, they might give you a chance to explain what you have to offer and some of them will convert.

Let’s take an example here. Suppose you are building a course for working professionals in India. Should you build the one where you can offer a certification/degree or the one where you are advocating that degrees are useless?

In India, we have always seen college degrees as a good thing. Trying to convince the audience otherwise may backfire. upGrad.com is an example of this which offers online college degrees to working professionals — Move up in your career with specializations from top universities. Udacity, on the other hand, offers — The latest digital skills, within reach. Time will tell which one will succeed in India, but confirmation bias is in favor of upGrad.

This isn’t to say we shouldn’t help people overcome their biases. But don’t lean on it to improve conversions without investing heavily in changing people's perceptions. Go after it if you have the time and resources.

Helping People Make Decisions

To help people make decisions, we have to think about what factors affect/influence the decision making. We will discuss some of the important factors here.

Social Proof

Social proof was identified in 1935 by Muzafer Sherif, a social psychologist in an experiment discussed below.

Subjects were seated in a dark room and asked to stare at a pinprick of light a few feet away from them. They were asked to estimate how much the dot was moving. In reality, the light was stationary but the autokinetic effect (an observation that stationary objects appear to move when there is no second point of reference to compare them to) led people to believe it was moving.

A week or so later, the experiment was repeated but this time in groups. People were asked to shout out their estimates of the movement and Muzafer Sherif observed that the group would tend to converge on an estimate – even if these estimates were vastly different from those that they had given in the first part of the experiment. The experiment was then repeated again with the individuals on their own and their estimates tended to remain with the group output rather than with their own original estimate.

In short, social proof demonstrated that we are likely to converge on a course of action (signups, payments, etc) when others are doing the same.

Social proof is popular in product circles because it’s very intuitive. Almost all products do it knowingly or unknowingly. For example,

• Ratings and reviews on Booking.com, Amazon.com

• # of people using the app on Playstore

• # of subscribers to a newsletter/blog

• Testimonials from other customers on SaaS products

Social proof is used extensively because it is pretty useful in providing trust markers. A user thinks - “If people like me have benefitted from a particular service and are happy, my chances of making a wrong decision reduced significantly.” Users don’t have to do a lot of research/analysis which saves a lot of time.

In fact, the most powerful growth channel is built on social proof — referrals. We tend to trust a referral from a friend or family member over anything else.

The Paradox of Choice

We believe that more choices are always better. The paradox of choice states that while we might think that more choice would make us happy, the reality is that it often doesn’t. Sheena Iyengar, Professor of Business at Columbia Business School, did this study on people purchasing jam,

In a study on the effects of choice on motivation, researchers set up a jam tasting booth at an upscale grocery store. On some days, they displayed a limited selection of 6 flavors; on others, they put out a much more extensive assortment of 24 flavors. Customers were allowed to taste as many jams as they wanted. They were then given a coupon, valid for one week, that entitled them to a small discount on the jam. The coupons were marked with code numbers so that the researchers would know which customers had cashed them in. In the end, 30% of customers who visited the smaller booth came back to buy some jam, compared to a meager 3% of customers who saw the booth with the larger selection.

Despite the increased sense of freedom that comes with more choices, we often grow tired of making choices because

1. Lots of analysis among multiple choices leads to decision fatigue, so we simply stick with the status quo or delay our decisions.

2. Even if we make the decision, we get into a post-purchase ‘angst’ where we worry about whether the decision we made was the right one.

This leads us to another question. Should we simply reduce the choices available to consumers? Not necessarily. In some cases, more choice is preferred, and in others, it is not. Less choice is preferred when:

• People are unsure of their preferences

• People are not familiar with the options

• The options are similar with no clear better option

• The options are difficult to evaluate or compare

• When choosing a more utilitarian, necessity product – as opposed to a more luxury or hedonistic product

Sheena Iyengar points to four effective strategies that can help in decision-making:

1. Reduce the number of choices

2. Make the consequences of each choice clear 

3. Structure and categorize choices

4. Start with the easy choices before moving to the more complex choices

Each of these four strategies makes it more likely that we will feel happy about our choice. VWO published a study with Lyyti that simplified their pricing page from four columns to three. They didn’t eliminate any pricing tiers. Instead, they categorized the features better [strategy #3] and allowed you to select a free trial on any of the plans.

The variation ended up increasing visits to their free-trial page by 93.71%:

One way to quickly check whether your pricing pages are leading to analysis-paralysis is to show it to few people around and ask them to pick an option they like most. If they struggle with making a choice, it’s time to rethink the page.

Scarcity

We tend to like things that are harder to obtain. Scarcity is the psychological bias that makes us place a higher value on things that are scarce than those in abundance.

Scarcity is extremely powerful and fairly easy to implement. And the reason it’s so effective is that it combines multiple biases into one: loss aversion, social proof, and anticipated regret.

Loss aversion — If we don’t act upon a scarce product, it basically means we’ll lose both the product itself in the short run but also our freedom to choose it in the long run.

Social proof — Usually, products become scarce when the demand is high. Once that happens, it implies that other people bought it in the past so it must be valuable and we should seize the opportunity.

Anticipated regret — When facing a decision, we anticipate the associated regret we might experience. Deciding to act now is our attempt to try and eliminate that possibility.

One might argue that scarcity forces users to make a decision, but If we fail to tell people about the scarce product in time, it is just as bad. As long as we are presenting the facts and we are putting out the real numbers, it shouldn’t matter.

Scarcity can be of three types and you should check what can apply to your product.

1. Time-limited scarcity

When the time has a limit, it creates a deadline that makes people act before the time is up.

Examples: Limited-time deals on eCommerce websites like the lightning deal on Amazon

2. Quantity-limited scarcity

Quantity-limited scarcity is considered more effective than time-limited scarcity because the end of the supply is unpredictable, depending exclusively on-demand rather than time.

Examples: On Booking.com, they show the number of rooms left

3. Access-limited scarcity

It refers to limited access to features like information, groups, or spaces. Research showed that censorship made people place a higher value on the restricted features than those that were not because exclusivity made them feel special.

Examples: Tinder uses the “Elo” ranking system to rate members based on desirability and invites the top ones to join the closed version called Tinder Select.

Access limited scarcity was also used by OnePlus to create a waitlist.

Priming

Imagine you’re driving to buy some groceries on a sunny afternoon in Bangalore. The traffic is peak and you take around an hour to find a parking spot. You’re already in a foul mood when you enter the supermarket. What happens next? On your way out, you take out your frustration on the person at the billing section.

So what happened? It’s a phenomenon called Priming. You aren’t alone in experience this, every one of us has been there multiple times a day.

Priming is a bias whereby exposure to one stimulus (delay in finding the parking space) influences our reaction to the next stimulus (interaction with the supermarket employee).

Priming has strong effects on our behavior. One of the most fascinating studies around it was published by John Bargh.

In the first experiment, a group of participants was primed with stereotyped words related to elderly people, and the control group was given a list of neutral words. The researchers discovered that the former set walked slowly after the experiment as compared to the control group.

The second experiment revealed that participants that had been primed with words related to rudeness interrupted the experimenter on average after five minutes. Of those people who were primed with polite words, 82% never interrupted at all!

From the experiments, it’s clear that we behave a certain way because we are primed to do so.

So what’s happening here? Basically, when a person is primed, the information in their memory is temporarily activated, which then influences their response to the next stimulus.

Interestingly, it has a direct application in user experience and conversion. When users visit your website, everything they see and read can prime them in one direction or other. Just look at Audi’s page versus Maruti's page.

In the first fold, Audi has words like future, beyond, experience, progress, etc.

In the first fold, Maruti has multiple images around EMIs, features, care/support, etc.

Similarly, we can prime visitors’ expectations and their overall perception of the website through copy, colors, and images.

Copy

Talk to your customers in a way they’re familiar with. Open your ears to the words/phrases they use on an everyday basis. Incorporate those in your copy.

You can also analyze your competitor’s website for inspiration. What are some of the most frequent themes and words they use and why do they use them?

Colors

Colors have different meanings and can be used to prime emotions. Be aware of your target group and their understanding of colors and the emotions they elicit.

Images
Use pictures to prime your customers. You can prime emotions that come with the purchase of your product or you can prime a desirable action that requires the purchase of your product. Let’s look at the Airbnb home page for example. It makes you long for a holiday.

Authority

As human beings, we have evolved over the millions of years as a herd, led by a leader. The leaders made the big decisions and the rest followed suit.

As a result, we have a deeply rooted sense of duty to follow authority — celebrities, politicians, experts. This is known as authority bias — if someone in authority is speaking something, they must be right.

Websites/apps use trust seals of easily recognizable and reputable brands. Well-known logos and brands are remembered more often and make a user feel secure while making payments.

Authority bias is also used to create strong brands through association with authorities. Nike builds a strong durable sports brand by associating itself with super athletes.

Anchoring

Nothing is cheap or expensive by itself but compared to something.

Anchoring is a particular form of priming effect whereby initial exposure to a number serves as a reference point and influences subsequent judgments

Once you’ve seen a $150 burger on the menu, $50 sounds reasonable for the meal. At the supermarket, a $1,000 bag makes a $50 T-shirt look cheap.

A popular way of anchoring is used by websites in price presentation to influence price perception.

Here’s an example from The Economist that Dan Ariely talks about in his book Predictably Irrational. Take a close look at those prices.

If the Economist.com web subscription costs $59, and the one-year subscription to the print edition of The Economist costs $125, then the fair price of the print and web subscription combined should be $184.

But it’s not. It is also $125.

In reality, the stand-alone print subscription of $125 is a decoy, i.e. no one is going to buy it. It does two things — first, it anchors you to a higher price to make $59 look like a cheaper option. Second, it makes print+web subscription of $125 look like a steal deal.

Decoy pricing or anchoring should be used carefully since it can backfire when the value you are offering doesn’t match the price.

Copywriting

For a long time in the history of marketing, copywriting mattered most. From Oglivey to Hopkins, everyone has written about the importance of copywriting. Part of the reason was that you could advertise in newspapers and magazines where people were mostly reading text.

With the internet and TV, videos emerged as a key advertisement format. However, copy still remains crucial. When you think about the famous commercials of Apple on TV - the 1984 super bowl ad or Think Different, it is the copy that mattered most. Take the one that Steve Jobs loved and was used in Think Different,

“Here's to the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes. The ones who see things differently. They're not fond of rules. And they have no respect for the status quo. You can quote them, disagree with them, glorify or vilify them. About the only thing you can't do is ignore them. Because they change things. They push the human race forward. And while some may see them as the crazy ones, we see genius. Because the people who are crazy enough to think they can change the world, are the ones who do.”

While designing web pages and apps, copywriting becomes important in texts and videos. Why? Because a good copy converts. Here are six pointers to help you write a good copy for your products.

1. Framing

The same sentence can be framed in a positive or negative manner.

• Negative framing: Don’t be late.

• Positive framing: Arrive on time.

Research shows that negative frames reduce comprehension and degrade the impact of your message (Jacoby, Nelson, & Hoyer, 1982). So you should use positive frames for the majority of your sentences.

2. Serial Position Effect

The serial position effect refers to the finding that people remember the items presented at the beginning, or at the end of the list, relative to items presented in the middle.

For example, immediately after the presentation of a nine-item word list, individuals with normal memory ability might recall about 70% of the first three words, 60% of the last three words, but only 40% of the middle three words.

How can you use this? While listing the benefits, present the most important benefits in the beginning or at the end.

3. Bizarreness effect

Anything strange or unusual that causes interest or amusement has a higher recall.

As explained by Seth Godin of Purple Cow:

“If you travel on an airline and they get you there safely, you don’t tell anyone. That’s what’s supposed to happen. What makes it remarkable is the service is so unexpected (they were an hour early! they comped my ticket because I was cute! they served flaming crêpes suzette in first class!) that you need to share it.”

4. Using active over passive voice

Active sentences are more persuasive. Why? Hosman (2002) explains

“Sentences with more complex grammatical structures [are] more difficult to understand or comprehend. This comprehension difficulty could affect the persuasion process negatively, presumably because comprehension is an antecedent to persuasion…” (pp. 373)

Active sentences increase processing fluency. Because readers can digest your message more easily, they feel better.

So where do you go from here after reading this long post? Keep it for reference and take a landing page of a website you love and see where they are using one or more of these methods. Over time, you will start getting inspiration for your own products.

Here is a summary if you are trying to remember it all

1. Don’t fight confirmation bias unnecessarily. Understand your consumer’s needs, beliefs, and assumptions.

2. Help consumers make decisions — social proof, scarcity, authority, priming, a clear set of choices, anchoring

3. Copywriting — positive framing, bizarreness, serial position effect, using active voice

With this, I conclude this week’s post. Next week, we will discuss using psychology to retain users.

Sincerely,

Deepak

Twitter | LinkedIn

PS: If you liked this post and are liking the Growth Catalyst Newsletter so far, share it with friends or give me a shoutout on LinkedIn, Twitter. A little act of kindness will help reach this newsletter to those who can benefit from it 🙏🏻 😊

Much thanks to Divya and Abhishek for helping spread the word 🙏🏻

Subscribe now

In 1930, a psychologist at Harvard University called B.F. Skinner made a box with a lever and placed a rat inside it. As the rat moved around, it would accidentally knock the lever and, when it did, a food pellet would automatically drop into the box. Over time, the rat learned to go to the lever and press it: the reward made the rat learn this behavior. Skinner proposed that the same principle could be applied to humans. He called it operant conditioning, through which a voluntary behavior is modified by reward or punishment. This box with a lever became known as the Skinner box.

Gambling casino operators were among the first ones to use the operant conditioning principles for profit. They carefully designed casinos optimizing the physical environments, slot machines, unpredictable reward systems to get a number of players in a zone where they push buttons without caring much about anything. Some of these players are so absorbed that they urinate in cups so as not to break the flow. Unfortunately for the casinos when you leave the casino, you leave their Skinner box.

Though casinos don’t affect everyone, elections do. Almost a century later in 2013, Robertson worked under the supervision of Epstein, B.F. Skinner’s last doctoral student, to begin a series of behavioral experiments to address an unsettling question: Could a search engine be used to sway the minds of voters?

In an article titled “The 21st Century Skinner Box”, Robertson wrote about his findings.

The idea was simple. You can influence someone’s decisions based on the order in which you present the information. In the digital world, controlling the presentation of information becomes even easier. By arranging information, web-based platforms can influence which hotel you book, the music you rate, and, the emotional language you use on social networks.

So what would happen if people were exposed to biased search rankings—search results arranged to favor one political candidate or another?

As most of my readers live in India, you will certainly relate to it. They conducted one of the experiments on real, undecided Indian voters in the midst of India’s 2014 Lok Sabha elections, the largest democratic election in history.

On average, they were able to shift votes by 20 percent or more, with shifts as high as 72.7 percent occurring in some demographic groups (e.g., unemployed males from Kerala). But the biggest shock was that 99.5 percent of subjects in the India experiment showed no awareness of the experimental manipulation—the biased search rankings.

As most elections are won at a narrow margin, a 20% shift would mean changing the outcome of the whole election. It goes on to show the powerful effect of social media and search engines on our lives.

Robertson called this phenomenon the Search Engine Manipulation Effect (SEME) and published the findings in the journal Proceedings of the National Academy of Sciences. An example of social media influencing elections is the US still investigating the impact Facebook had on their 2016 presidential election results.

We don’t need to think of as one-off things that come once every four years like elections. One could say that the daily apps we use are a lot like digital Skinner boxes. They provide you with a sense of control. But they also influence your thoughts and choices. Examples could be the gamification of work that keeps Uber drivers on the road longer than they would be otherwise or notifications from an addictive game that keeps bringing you back and wasting hours on a stretch. Most people appear to be unaware of the impact apps have on them.

Unlike behavior scientists of the past who used to do experiments on hundreds of participants, the apps have an enormous amount of data and they can use this data to conduct experiments on the fly — the shape and color of the buttons you press, the timing of each notification you receive, and the content of every piece of information that reaches you.

If you have read this post so far, you may be inclining towards the dark powers of apps. That is because of the way I have presented the information. This is why there is so much scrutiny around the big tech these days because the press writes the story I have written so far. But the skinner box is a form of magic that can be used both ways. More importantly, the apps we use every day have created a huge impact on the world.

• Google and Facebook enable us to find information quickly, connect with like-minded people, and discover content that we’ll enjoy—saving us valuable time that we can spend elsewhere finding it.

• Apps like calm and headspace have used notifications and reminders to get people to meditate on a regular basis, which helps them.

• Education tech platforms have employed gamification tactics like points, badges, and leaderboards to help students learn more.

• Running apps like Nike Run Club use gamification to give you a sense of achievement and build healthy habits.

However, we shouldn’t ignore the problems of Tech as well. In fact, showing the world that they can affect billions of users with the right accountability and transparency could be tech’s route to redemption.

Hopefully, I don’t need to convince you now on why PMs and growth teams need to learn user psychology. If one principle of behavioral psychology can create such an impact, both good and bad, we need to learn and understand this subject while designing products.

If I had to explain Psychology, the perfect analogy to the user psychology would be the ‘Force’ from Star Wars.

The Force is a mysterious energy field created by life that binds the galaxy together. Harnessing the power of the Force gives the Jedi, the Sith, and others sensitive to this spiritual energy extraordinary abilities, such as tricking minds, and seeing things before they happen.

We can use behavioral science to build successful products, or simply improve the odds of succeeding it. We need to understand what motivates customers: their needs, desires, hopes, fears, and frustrations. Behavioral psychology helps us understand that.

In the next 3 posts, we will cover interesting things around psychology.

• Designing the right pages using principles of psychology (this one)

• Converting users using principles of psychology

• Retaining users using principles of psychology

Let’s now understand how to design the right page using the principles of psychology.

Designing the Right Page

There are two key ingredients of designing the right page — visual design and copywriting. The visual design includes elements like button placement, colors, layout, etc. Copywriting, as the term suggests, is the text you place on the page. As copywriting helps in conversions a lot, I will cover principles used in copywriting in the next post around conversion.

In this one, let’s discuss the principles that can improve the visual design

1. Gestalt Principles

The human brain is wired to see structure, logic, and patterns. It helps us make sense of the world. In the 1920s, a group of German psychologists developed theories around how people perceive the world around them. They are known as Gestalt principles. “Gestalt” is German for “unified whole”

For example, what do you see when you look at this image?

If you’re like most people, you see a triangle. Some of the gamers out there can see pac-men. We see the triangle because our brains see something that makes sense to us and that we understand.

Figure-ground

The figure-ground principle states that people instinctively perceive objects as either being in the foreground or the background. They either stand out prominently in the front (the figure) or recede into the back (the ground).

Knowing this principle well, you can do usability tests to figure what stands out in front of your designs on a particular page.

Similarity

The principle of similarity states that when things appear to be similar to each other, we group them together. And we also tend to think they have the same function.

For example, websites use the similarity principle to distinguish different sections by putting them in different colors. They also use the color blue to distinguish links from regular text and green/yellow buttons to indicate call-to-action. See Github's different sections for example. We can use the principle of Similarity in navigation, links, buttons, headings, call to action, and more.

Proximity

The principle of proximity states that things that are close together appear to be more related than things that are spaced farther apart.

Source: Andy Rutledge

We can apply the Proximity principle pretty much everywhere from navigation, cards, galleries, and banners to lists, body text, and pagination. See an example of two-page structures, the second one uses the proximity principle to help users focus by putting similar things together.

Common region

The principle of the common region is highly related to proximity. It states that when objects are located within the same closed region, we perceive them as being grouped together.

The Common Region principle can hold together many different elements keeping them unified by the use of line, color, shape, and shadow.

Continuity

The principle of continuity states that elements that are arranged on a line or curve are perceived to be more related than elements not on the line or curve.

The linear arrangement of rows and columns are good examples of Continuity. We can use them in menus and sub-menus, lists, product arrangements, carousels, services, or process/progress displays.

Closure

The principle of closure states that when we look at a complex arrangement of visual elements, we tend to look for a single, recognizable pattern.

When presented with the right amount of information, our brain will jump to conclusions by filling in the gaps and creating a unified whole. This way we can decrease the number of elements needed to communicate information, reducing complexity and making designs more engaging.

We can use the Closure principle in Iconography, where simplicity helps with communicating meaning, swiftly and clearly.

Symmetry

Symmetrical elements tend to perceived as belonging together regardless of their distance, giving us a feeling of solidity and order.

Symmetry can be used for galleries, product displays, listings, navigation, banners, and any content-heavy page.

Borrowed images and ideas from few great pieces on Gestalt principles, including this one 🙏🏻

If you are new to design as a PM or marketer, keeping Gestalt principles in mind will improve your design manifold.

2. Law of Least Effort, or Zipf's Law

Most people are inherently lazy, and if not lazy, want to do things efficiently. They want to take the path of least effort or resistance. This is called the law of least effort.

For example, people who want to order from a restaurant or to make a reservation will often look for a phone number but will give up if they can't easily find it prominently displayed on the landing page. If you operate a restaurant, put your phone number in the top right corner of every page on your site. And if you want people to email you instead of calling, make your email address highly visible instead.

If people can’t easily get the information they need (such as your phone number to make a restaurant reservation), they take the path of least resistance and move on to a competitor or another option.

3. Perceptual Selection

Perceptual selection is individuals focussing on certain elements instead of others. What makes people focus on certain elements and not others. There is a good overlap between Gestalt principles and perceptual selection. The basic principles of perceptual selection are

1. Principles of Size

Generally, the bigger is the size of the button, the higher is the probability that it attracts the attention of the user. Similarly, letters of higher sizes in newspapers or books are first selected for reading. 

2. Principles of Intensity

A loud sound, strong odor, or bright light are noticed more as compared to a soft sound, weak odor, or dim light. It is, for this reason, the CTAs like ‘add to cart’ or ‘buy now’ are made of bright colors.

3. Principles of Repetition

Repetition increases people’s sensitivity or alertness to the stimulus. Advertisers use this principle by repeating advertisements of the same product to attract people’s attention.

4. Principles of Novelty and Familiarity

Either a novel or a familiar external situation can get the attention. New objects or events in a familiar setting or familiar objects or events in a new setting draw better attention.

5. Principles of Contrast

Something which stands against the background, or which is unexpected receives more attention. People wearing purple shoes get more attention :)

6. Principles of Motion

A moving object draws more attention as compared to a stationary object. An icon that moves when the user comes to the page will get the attention of the user.

To apply these principles, on every page you should ask what you want your users to do and then use the principles of perceptual selection to highlight those actions.

4. The Aesthetic-Usability Effect

Users are more tolerant of usability issues when they find an interface visually appealing. This aesthetic-usability effect can mask UI problems and can prevent issue discovery during usability testing.

Ever wondered why many people love the Cred app despite several usability issues. It’s the aesthetic-usability effect at play there.

5. Competing pressures model

Another interesting theory from Psychology is the Field Theory developed by Kurt Lewin. He mentioned that human behavior is influenced by two pressures:

• Promoting pressures — reasons to do something

• Inhibiting pressures — reasons not to do something

As Product Managers, we often focus on promoting pressures by telling people about the benefits but not enough on reducing friction.

For example, making payments easier by adding more options almost always improves conversion because it solves for inhibiting pressure in a transaction. An interesting derivative of this theory is the Psyched framework written by Darius Contractor.

6. Cognitive Load or Hick’s law

Hick’s Law is a psychological concept that says people will take a longer amount of time to decide if they have more choices. When people have to make decisions and are presented with a lot of information, they have to think, and this raises cognitive load. Therefore, it makes more sense only to present a limited number of choices at any given time as users advance through your interface.

An example of this is Amazon doing it with ‘1 click ordering’. Here is a good analysis by Wharton on why it was a gamechanger.

As many readers ask about books that have helped me build knowledge in specific areas, sharing that for behavioral psychology. My own interest in psychology started in college, and here is the top 10 list of books I can remember that have helped me. In descending order of importance, they are :

1. Thinking fast and slow

2. Influence: The Psychology of Persuasion

3. The Power of Habit: Why We Do What We Do in Life and Business

4. Man’s search for meaning

5. Predictably Irrational: The Hidden Forces That Shape Our Decisions

6. Nudge: Improving Decisions About Health, Wealth, and Happiness

7. The Design of Everyday Things - how people see and interact with the world

8. Hooked: How to Build Habit-Forming Products

9. The 22 Immutable Laws of Marketing: Violate Them at Your Own Risk

10. Atomic habits

In the end, I will conclude this post by saying this :)

Don’t move to the dark side 🙌

Sincerely,

Deepak

Twitter | LinkedIn

PS: If you liked this post and are liking the Growth Catalyst Newsletter so far, share it with friends or give me a shoutout on LinkedIn, Twitter. A little act of kindness will help reach this newsletter to those who can benefit from it 🙏🏻 😊

Much thanks to Pushpak and Aditi for pointing out some spelling mistakes. Somehow Grammarly got turned off while editing and I didn’t notice 🙏🏻

Subscribe now

Happy New Year 2021,

Hope this year brings happiness and well-being to humanity :)

I published this post on the substack on 27th Dec. I was also trying to shift the newsletter URL from https://deepaksingh.substack.com/ to https://www.growth-catalyst.in/. And due to some technical issues during the process, I could not send emails after shifting. So this post couldn’t go through email to your Inbox.

I shared the post on social media channels like Twitter and LinkedIn though and some of you may have read it. The issue with email got resolved this week only. So sending it this week for most of you who haven’t read it yet :)

From the next week, here is the schedule of the posts in the next couple of months, spaced at 2 weeks each:

• User Psychology - Concepts to Convert Users with Real Examples [24th Jan]

• User Psychology - Concepts to Retain Users with Real Examples [7th Feb]

• Masterclass to Gamification - Part I [21st Feb]

• Masterclass to Gamification - Part II [7th March]

• Masterclass to Gamification - Part III [21st March]

Off to the post,

Airbnb recently did an IPO of $100 Bn. One of the key stories from the early days of Airbnb is what Paul Graham wrote in his essay “Do Things That Don’t Scale”. In this essay, Paul Graham advises startups to do things that don’t scale while building the startup initially. And the most common unscalable thing founders have to do at the start is to recruit users manually. You can't wait for users to come to you. You have to go out and get them. He wrote in the essay around AirBnB,

You'll be doing different things when you're acquiring users a thousand at a time, and growth has to slow down eventually. But if the market exists you can usually start by recruiting users manually and then gradually switch to less manual methods. 

Airbnb is a classic example of this technique. Marketplaces are so hard to get rolling that you should expect to take heroic measures at first. In Airbnb's case, these consisted of going door to door in New York, recruiting new users and helping existing ones improve their listings. When I remember the Airbnbs during YC, I picture them with rolly bags, because when they showed up for tuesday dinners they'd always just flown back from somewhere.

Founders don’t have a choice when it comes to user research in the early days of building and growing the product. The market and insights are super important to the founders to build a great product. And once founders set themselves down that path, no one can know about the users and the market better than them. Over time, it just becomes a habit and a superpower.

Is there a scalable way to do user research? Companies hire research teams over time who go to the market to bring back insights. However, product and growth teams still have to invest a lot of time going through raw recordings to get meaningful insights. In my opinion, user research never scales really. It is one of those unscalable things founders, PMs, and growth teams have to keep doing day-in, day-out. The market changes with such swiftness and without mercy that companies get obsolete every other day. To stay relevant, we have to invest a lot of time talking to consumers, going through market reports ourselves. Insights are hard to delegate/outsource :)

Hopefully, I have made an interesting case for why PMs and growth teams should care about user research🤔 In this post, we are going to cover the most important aspects of user research. But first a word of caution!

The world of user research is vast and might make you feel like an outsider. It made me feel so. Just look at this chart to understand why I am saying so :) Don’t spend more than 20 seconds.

The chart gives anxiety to a lot of folks I know. But as steve Krug wrote in his book,

“Fortunately, much of what I do (research) is just common sense, and anyone with some interest can learn to do it.”

We are going to cover what’s vital for product and growth teams in this team in simple terms. So this essay is helpful to anyone who is remotely interested in user research.

More specifically, we are going to cover four things

1. Market research versus user research

2. Why do user research?

3. 7 factors that affect the user research outcomes

4. What kind of user research is useful in different stages of product development?

5. Where and how do you start? (for smaller teams)

Market research versus user research

It is also important to understand what user research isn’t. People often confuse market research with user research.

Market research is useful in bringing business insights into market needs, size, competition, and pricing.  It uses quantitative methods and has a heavy focus on numbers. The researchers look at large sample sets to understand the average age of potential users, their income level, and other general characteristics.  Market research is more general than user research.

User research is mostly qualitative in nature. You can use data from your product or users to validate/invalidate the insights. It operates on a small subset of people to learn about who they are and what they want. We will be focussing on user research in this essay leaving market research for some other day.

The importance of user research

I have already made a strong case for user research. Let’s get into more details on where all it helps as some of these use cases you may not be familiar with

• Exploring the problem, understand why and not just what the problem is

• Getting behaviors of users and not opinions

• Validating assumptions and remove biases

• Discovering usability problems in product

• Creating personas by classifying different kinds of users you meet. Changing design, copy, experience according to these personas.

• Creating a shared understanding of user personas and their problems within internal teams like design, product, business, and engineering.

7 factors that affect the user research outcomes

A lot of big decisions are taken based on user research. But like any method, user research also has its own set of limitations. The decisions affected by these limitations wouldn’t turn out to be good decisions.

Let me put another point to explain this even further. All companies know that user research is important. They also have user research teams, but even then they fail to uncover customer problems and build products that don’t add much value to customers. Why is that? Because of the process and limitations of research methods matters. The quality of decisions made is as good as the limitations.

Hence it’s important to understand and remove these limitations wherever possible. There are 7 limitations —

1. Stated vs actual preference - In the last essay, we mentioned the ask-observe framework.

There is a key difference between asking and observing a customer. Asking provides you the stated preference of the customer, i.e. what people talk about/state. A lot of times, these stated preferences are different from actual preferences.

The actual preferences are revealed by observing what the customers actually do. They are also known as revealed preferences as they get revealed as you observe customers.

You may have seen multiple instances of stated versus revealed preferences in your day-to-day life. Like someone who says they like you a lot, but then go ahead telling lies about you to others.

Stated preferences are great for marketing products. But revealed preferences are the key to building products. What people say is more important in the marketing department. What people do is very important in product departments.

2. Environment - The environment in which the research is conducted is pretty important. People feel more comfortable in their own homes as compared to labs. Usually, the research should be conducted as naturally as possible.

3. Incentives — Almost all studies offer incentives for people for studies. It is important to be careful in places this can backfire.

Suppose you are designing a survey where you are paying people to complete a survey. In order to save time, people might start ticking random options without reading options.

Google designed surveys beautifully by creating a survey where they insert a fact-based question in between. The question can appear randomly and check whether the user is honestly answering the question. What a beautiful way to solve the problem with incentives, though it doesn’t solve it for all use cases.

4. Sample size — Depending on the user research method, the sample size of the TG varies. A usability study requires 6-8 participants. After 6-8 participants of a particular persona, the usability issues start getting repeatable. A survey requires >500 participants to contain error rates within 5%.

5. User’s relationship with the interviewer — Your Mom, family, and friends will often tell your idea is a good one as they want to cheer you up. Take as many unknown users in your research if possible. From the book Mom’s test,

They say you shouldn't ask your mom whether your business is a good idea, because she loves you and will lie to you. This is technically true, but it misses the point. You shouldn't ask anyone if your business is a good idea. It's a bad question and everyone will lie to you at least a little . As a matter of fact, it's not their responsibility to tell you the truth. It's your responsibility to find it and it's worth doing right .

6. Extent of context transfer — A picture is worth a thousand words. A working product is worth a thousand pictures. Does the user fully understand the context of your question when you ask them? If not, you may get the wrong answer.

Henry Ford is famous for the quote “If I had asked people what they wanted, they would have said faster horses.” Unless you show people what a car is and it can take them faster, you may not get the right assessment.

7. The researcher — The last and the most important limitation is the researcher himself/herself. The insights of the researcher are limited by the researcher’s ability to improvise in interviews, connecting with users and their needs, connecting with business/company and their needs. For this reason, sometimes founders and PMs are the best researchers. They are the ones working on it day-in, day-out. How can someone working for a couple of days, conducting some interviews get better insights than them?

If you go through solving all the limitations of the user research, you will quickly realize that it’s not possible to scalably solve these as there are a lot of nuances.

Stages of Product Development and User Research

As Lao Tzu says, “To attain knowledge, add things every day. To attain wisdom, remove things every day”

The wisdom lies in doing the research on a regular basis, rather than learning all these fancy terms from fancy books.

The best products get built, not by adding a lot of features, but by removing the clutter and solving a few core needs of the users. Take Apple, Netflix, Twitter, Uber as few examples. But saying no to features requires conviction, and the conviction gets stronger as you spend more time with consumers.

To do user research throughout product development, you have to understand different kinds of user research.

1. Usability Testing — Useful at all stages of the product. In usability testing, you let the users use a product and record the session to analyze where they struggle while using the product and what they love about the product. Usability tests should be done on every product. It also becomes useful when done on high fidelity prototypes before getting the product into development. You can also let users use your competitor’s products to figure out what aspects they have done well with and come easy to users.

   

   Usability testing can be done moderated or unmoderated. In a moderated study, the facilitator can gently nudge a quiet participant to share more about what he's doing. In an unmoderated study, you can ask a user to think aloud, but there is no one there to remind her if she doesn't do it.

   Nowadays, even remote usability testing is possible where usability testing is done via screen-sharing. Unmoderated studies can be done remotely through platforms like usertesting.com.

   Useful for: All stages, especially useful for prototypes, competitive analysis

2. User Interviews — Contextual interviews are highly effective in the exploratory phases of the product. There are two cases in particular. If you are trying to build a new product, contextual interviews help you connect with the users. If you already have a product and are struggling to figure out why something isn’t working out, user interviews are highly effective. You can do these interviews in person, video calls, or over voice calls.

   

   Useful for: Building a new product, figuring out why something is/isn’t working out

3. Survey and Questionnaires — Once you have done user interviews, you can quantify whether the problems exist with the bigger base by doing surveys/questionnaires. It is usually a bad idea to directly start with a survey. Qualitative methods are much better suited for answering questions about why or how to fix a problem, whereas quantitative methods do a much better job answering how many and how much types of questions

   Useful for: Quantifying the hypothesis validation, removing the sample bias

There are other popular methods that aren’t relevant or useful for your product most of the time like

— Eyetracking to see how users scan a product/page. It is good to understand how people read a page via eye-tracking. Here are some eye-tracking patterns. Eyetracking requires a sophisticated setup and quite time-consuming. It isn’t a method that produces a lot of insights apart from eye behavior. Here is how a sample eye-tracking heatmap looks like

— Focus groups are bringing 8-12 people in a room and getting them to discuss various aspects of the research. Focus groups are prone to groupthink and that’s why their insights are always questionable. Groupthink is a psychological phenomenon that occurs within a group of people in which the desire for harmony or conformity in the group results in an irrational or dysfunctional decision-making outcome.

— Diary study is used to understand the long-term behavior of the user. Users are asked to keep a diary and note their thoughts/usage patterns around a product. The data is self-reported. Diary studies have a high churn as a lot of people forget to do it over time. You need to offer incentives for diary studies, but as we discussed incentives can affect the outcome in unexpected ways. If done well, diary studies can help you how users ’ behavior attitudes change over time for your product.

— Card sorting is a method used to help evaluate how information should be presented on a site. In a card sorting session, participants organize topics into categories that make sense to them. You can use actual cards, pieces of paper, or sticky notes to do card sorting.

If you want to do one of these methods mentioned above, you can hire an agency to do so, or let UX research teams do it. If you are interested in reading about more methods quickly, you can read this article on UX research methods by Neilson Norman Group.

What you should focus on doing heavily are — usability testing, user interviews, and surveys.

Where do you start?

The quickest way to conduct user research is

— If you stay near your users and have access to them, you can go where your users are, talk and observe. I usually prefer to go to cafes and show the new design to a few people there and talk to them after buying coffee for them. This is provided the customers for the product also come to these cafes. If you are building a product for housewives, this method may not work.

— If accessibility is the problem, you can hire an agency to recruit these users and conduct the research with researchers/designers/business teams. A simple google search along with location like “user research agencies in Bangalore” will give you few good names.

If you want to do the user research quickly and know few users, there is one thing you must absolutely master before starting research — asking the right questions.

Asking the right questions

Let’s say you want to understand “why do people shop online?”

1. Ask questions which is easy to answer — If you ask users why they shop online, it’s a hard question to answer. So you have broken this question into smaller questions that are easy to answer.

2. Don’t ask leading questions”, i.e. questions that influence answer — When you try to rush to answers from users, you end up asking leading questions. A leading question is “How anxious do you feel while shopping online?”

3. Ask about specific instances — Specific answers are usually more accurate and insightful than generic ones, and one of the ways you can get specific answers is by asking about past instances when something happened. An example is “Tell me how you felt the last time you tried to buy something online and the purchase failed.”

4. Ask open-ended questions — Open-ended questions have a detailed answer. These cannot be answered with a simple 'yes' or 'no', and require the respondent to be detailed.

Putting an example below for you to understand and remember

Say you want to understand “why do people shop online?”

Ask questions which is easy to answer

“How do you buy different products?”

“What kind of products do you buy online?” —> “Why?”

“What kind of products you don’t buy online?” —> “Why?”

“What do you like the most about online shopping?”

“What do you hate about online shopping?”

Don’t ask leading questions

Change “How anxious do you feel when shopping online?”

to “Describe your online shopping experience last time”

Ask about specific instances

“Tell me how you felt the last time you tried to buy something online and the purchase failed.”

Ask open questions

Instead of “do people around you buy online?”

ask them “how do people around you purchase products?”

Final Words

I would leave you with 2 criteria that will help in deciding what important things have come in these user research sessions. A lot of time user research sessions lead to a lot of analysis-paralysis because you are trying to read too much into what every user. The actual meaningful insights are loud enough for everyone to hear and agree upon and can be triangulated. Let me explain

1. Loud insights - Building products on loud behaviors/insights from the research is always advisable. Loud insights are obvious to everyone.

   If there are some insights where everyone is interpreting them differently, you should either do more research to clarify or forget about them for the moment and act on the loud insights for the time being.

   If there are no loud insights, the research was conducted poorly and you should redo it. In absence of loud insights and triangulation, we end up doing this :)

   

2. Triangulation — Triangulation means using more than one method to collect data on the same topic. User research is just one source of data. Use product metrics, user psychology, and business understanding to back the user research. Suppose users are facing usability issues with a particular sign-up form, looking at the drops that happen on the form help you substantiate the usability testing.

Few words around consistency — It’s better to talk to 5 users every month than talking to 60 (5x12) in a month and then forget for another 12 months. There are two reasons why it is better

1. The first one is that consumer behavior changes rapidly. By talking to users every month, you are able to observe these changes.

2. The second reason is that your own understanding of the product and questions around change over time. So the research methods get more effective over time.

I will share some good resources next week that can help you to do a detailed study around interviews, usability testing, and surveys.

With this post, I wrap up Research and Experimentation: How to generate ideas and improve the metrics. Read the other 4 posts around this if you haven’t in the archives.

Sincerely,

Deepak

Twitter | LinkedIn

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

In 2012, a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. Developing it wouldn’t require much effort—just a few days of an engineer’s time—but it was one of the hundreds of ideas proposed, and the program managers deemed it a low priority. So it didn’t get implemented for more than six months, until an engineer, who saw that the cost of writing the code for it would be small, launched a simple A/B test.

Within hours the new headline variation was producing abnormally high revenue, triggering a “too good to be true” alert. Usually, such alerts signal a bug, but not in this case. An analysis showed that the change had increased revenue by an astonishing 12%—which on an annual basis would come to more than $100 million in the US alone—without hurting any of the key user-experience metrics. It was the best revenue-generating idea in Bing’s history, but until the test was done, its value was under-appreciated.

The above anecdote appeared in the Oct’17 HBR issue in an essay written by a Microsoft employee and an HBS professor. I picked this particular anecdote because it defies a particular school of thought when it comes to a/b testing — the belief that making small changes can’t make a big difference for the product.

As the Bing example and numerous others show, small changes can make a big difference. This isn’t to say that we will always find big wins that way. We need to work on the bigger ones as well. So how do we get these ideas and identify which of them is going to work?

Let’s focus first on getting the ideas. If we get enough ideas, some of them are going to work.

Step 1 - Generate Ideas: Ask-Observe Framework

You can follow a simple framework to generate ideas. I call it the ask-observe framework.

Here is how it looks

Operators = {your team, competition, people who have solved a similar problem in the same or different domain, industry leaders who can reason from first principles}

You can do four things to generate ideas:

1. Ask the customers

2. Ask the operators

3. Observe the customers

4. Observe the operators

Let me explain them one by one

Ask the Customers

You can ask customers through various methods to validate hypotheses and generate insights.

• On-site surveys: Some of the best insights come when you ask your users questions while they are using the product. A scalable way of doing this on a continuous basis is to add contextual surveys in your app/website as visitors browse around. For example, an on-site survey might ask visitors who have been on the same page for a while if there’s anything holding them back from making a purchase today. If so, what is it?

• Customer interviews: Customer interviews are amazing. Nothing can replace getting on the phone and talking to your customers. Be it the problem definition, understanding motivations, or anything else, you can get to know everything through customer interviews.

• Surveys: You can also do full-length surveys that go out to your TG. The full-length surveys should be defined differently for different TG. Focus on defining your customers, defining their problems, defining the concerns they had prior to purchasing, why they bought from you, and identifying words and phrases they use to describe your product.

Observe the Customers

Observing the customer - while they are using your app and even when they are off the app in their own world - is a good way to generate insights.

• User testing. User testing is asking users to perform a few tasks on the app/website and recording the sessions for further reference. These sessions bring out a lot of usability issues and that’s why they are also known as usability tests. Steve Krug’s book Don’t Make me Think is a great primer to usability tests.

• Analytics : This is where most teams spend their time looking for insights. I have covered in earlier posts on how to use segmentation and cohort analysis along with AARRR can help you uncover beautiful insights from data. Do note that data will mostly tell you about what is wrong. To understand why it’s wrong, you have to go to customers and ask/observe them through usability tests, surveys, interviews etc.

• Technical analysis: Page speed matters a lot to most of your users. As Google published in 2017, bounce rate is impacted by page load on websites.

  

  One should start by getting data around these page speed and latency across browsers and devices, and we will quickly find some areas of improvement. One false negative that happens with page speed and latencies is that sometimes, you may improve the average page speed, only to realize that it isn’t affecting bounce rate. The reason might be that it’s still bad enough for the cohort who were bouncing off earlier. So use segmentation and analytics wisely. To measure if technical improvements are making any difference, you should always go for an A/B test and not rely on absolute improvements in page speed/latency numbers.

• Session replays: Session replays are similar to user testing, where you record sessions on your website and watch as your actual visitors navigate your site. What do they have trouble finding? Where do they get frustrated? Where do they seem confused? It’s a pretty good thing to do in early days. Checkout hotjar or crazyegg which help you do it if you haven’t. One thing to watchout here is the page load time of your website. The JS loading with these tools can create high page load time which can lead to high bounce rate. Be careful there.

• User reviews: Look through user reviews from social media, play store, app store, etc. The easiest yet most obvious thing to miss. Replying on those reviews gives you a good way to staying connected with users.

• User journeys of different personas: Your product is built for multiple personas. For example, the same landing page can’t cater to a teacher and students for an educational website even if both will type the same URL and come to your website. Look at how they are using the product. Understanding this will give you good ideas on how to improve the product for different personas.

Ask the Operators

Operators = {your team, competition, people who have solved a similar problem in the same or different domain, industry leaders who can reason from first principles}

• Product walkthrough — You can review your product page by page with your team and identify where you can make the experience more relevant, clear, and value-adding—and where you can reduce friction. Add an hour to your calendar every couple of weeks to do so.

• Brainstorming on questions — All the good hypotheses to test arises from the right questions. Spend a good time defining the problem statement and move on to ask the Who, What, Why, When, Where, How, How Much. Please avoid any discussion or answers.

• Talk to other teams working on similar problems — If your growth is stalled, talk to a few senior leaders who led growth for years. They would show you a direction. Talk to other growth professionals on the phone on a regular basis. Exchange ideas, share experiments and discuss what worked, what didn’t, and why you think you got the results you did. Adapt ideas to your own product.

Observe the Experts

Observation would be mostly done on experts not directly accessible to you - your competition, industry leaders, etc.

• Competitor products — Your competition is an expert you have to beat. Use competitor products, look through their updates and reviews. Always try to understand why they built something and whether something is an experiment or a full rollout. Don’t copy them blindly.

• Product comparison — This is similar to product walkthrough. Everyone picks one or two products from your competitive and noncompetitive space that they use actively. Walkthrough their product flow and discuss what they do well, what they could do better, and how you can adapt it to your own product and audience.

• World-class products — Study how world-class products are built. What changes are they making and why so? A lot of literature is available around top companies and products. As a vast majority of people use them, many of their product principles in some form can be applied to your own.

• Communities — Join good online communities/newsletters around your industry. Producthunt, reddit, hackernews are some of the good communities in tech.

• Study world-leading operators — You don’t have to reinvent the wheel around everything you do. You can avoid a lot of pain by observing how world-leading experts operate. This can be done by reading and listening to them. A lot of excellent operators don’t share their thoughts. You can start by following operators who share their thought like Lenny Rachitsky, Andrew Chen, Paul Graham, Ben Horowitz, Keith Rabois, etc. I have lists on my twitter account like this one - Best to Follow where I keep adding people in different areas.

Step 2 - Consolidate Ideas : Product Themes

By now, you have a list ideas. It’s time to consolidate them. And the best way to consolidate ideas is through product themes.

Examples of themes can be acquisition, activation, retention, revenue, referral, delight, engagement etc. Themes help you identify which of the problems are most hurting your user experience. For example, if your churn rate is low then you don't have much to gain by focusing on retention. If your bounce rate is high, you can focus on activation.

Pick few themes every quarter and work on them. In a particular quarter, you shouldn’t be working on more than 2-3 themes.

By the way, you can also pick the themes first and then generate ideas around this. Kind of moving from Step 2 to Step 1.

So now, you have themes and ideas. It’s time to prioritise the ideas.

Step 3 - Prioritize Ideas: RICE Framework

RICE is an acronym for the four factors used to evaluate each idea: reach (R), impact (I), confidence (C), and effort (E). Intercom talks widely about it.

Reach

Estimate how many people will be affected by the idea you are proposing.

Reach should also include the freuqency of usage. So if you have few ideas around payment, it will affect paying customers and # of transactions.

Impact

Estimate the impact on an individual person. This number multipled by reach gives you overall impact the idea may have on users/business.

The impact is difficult to measure precisely. So you can use a scale: 5 for “very high impact”, 4 for “high impact”, 3 for “medium”, 2 for “low”, 1 for “very low”.

Putting an impact numbers is an objective way of quantifying your gut feeling. Further, it helps you evaluate in future how ‘off’ were you from the real impact.

Confidence

Confidence answers whether the idea will work or not. Most of the startups fail because there is high enthusism about bad ideas. You can use a scale to rate confidence as well: 5 for “very high confidence”, 4 for “high”, 3 for “medium”, 2 for “low”, 1 for “very low”.

The confidence should come from the data/research you have for the idea. If you don’t have data/research to back it up - don’t put it in high confidence just because you are enthusiastic about it.

Effort

The final step is adding the effort from the team: product, design, and engineering.

The effort can be estimated as a number of “person-months” or “person-weeks”. A person month is the work that one team member can do in a month.

Now we calculate the prioritisation score for each idea and sort them out.

Score = Reach * Impact * Confidence/ Effort

Try it in some of your projects and see how it works :)

To summarise the post, the way to generate good ideas for experimentation is to

— Pick product themes

— Use the ask-observe framework to generate ideas

— Prioritise using the RICE framework

Quick Announcement: Last week, I came across a dilemma. I had the post ready but wasn’t happy with the quality of it. The key argument for writing a post every week is a story I read from the book Art and Fear.

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality.

His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: fifty pound of pots rated an “A”, forty pounds a “B”, and so on. Those being graded on “quality”, however, needed to produce only one pot – albeit a perfect one – to get an “A”.

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the “quantity” group was busily churning out piles of work – and learning from their mistakes – the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.

So writing and posting every week helps with the craft of writing :)

However, a key principle I want to hold on while writing is to deliver value in every post. Thousands of people reading it every week means hundreds of hours spend on these posts. The quality has to trump quantity now — so I have decided to write a post once every two weeks.

Give me a shoutout if you feel it’s a good decision by dropping a message on LinkedIn, Twitter or Email. Convince me otherwise if you feel so. Hope to get some compelling arguments for both sides :)

Next post, I will get into the detail of user research — one method that trumps all other ways of product development. Watch out for the post.

Sincerely,

Deepak

For the new ones here, do check out the other posts that I have written if you haven’t

1. Referrals: The Holy Grail of Growth

2. Measuring Product-Market Fit

3. Decoding Growth - Where is Growth Coming From

4. How to Engage Users Through Gamification

5. Understanding Positioning Through a Practical Lens

3,500+ smart, curious folks have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

At the heart of any management theory lies the art and science of decision making. In a world filled with opinions and data more than ever, it is hard to convince everyone on decisions that we make as PMs or marketers.

Jeff Bezos, on the other hand, has made many good decisions over the last 3 decades. So it’s refreshing to hear Bezos talking about the decision-making framework in his annual letter to shareholders in 2016. Here is what he wrote,

Some decisions are consequential and irreversible or nearly irreversible – one-way doors – and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation. If you walk through and don’t like what you see on the other side, you can’t get back to where you were before. We can call these Type 1 decisions.

But most decisions aren’t like that – they are changeable, reversible – they’re two-way doors. If you’ve made a suboptimal Type 2 decision, you don’t have to live with the consequences for that long. You can reopen the door and go back through. Type 2 decisions can and should be made quickly by high judgment individuals or small groups.

As organizations get larger, there seems to be a tendency to use the heavy-weight Type 1 decision-making process on most decisions, including many Type 2 decisions. The end result of this is slowness, unthoughtful risk aversion, failure to experiment sufficiently, and consequently diminished invention. We’ll have to figure out how to fight that tendency.

Big companies have ample talent and resources to create new products and markets. But slowness, unthoughtful risk aversion, failure to experiment sufficiently, and consequently diminished invention leads to them failing in the long run.

But even smaller companies suffer from the problem that he talks about spending too much time on Type 2 decisions. They focus on incremental improvements that don’t make a difference in their survival for a couple of years time.

If you are a senior leader/CEO and understand this with the power to change the culture, it is difficult to manage two competent, smart people with different ideas. You want to empower both and don’t want to operate on your own biases.

That is where doing the a/b tests help

• Type 2 decisions: when A/B testing is applied to Type 2 decisions, it’s very easy to walk back through the door as Bezos suggests by simply turning off B and returning to A.

• Type 1 decisions: A/B testing helps you understand when a Type 1 decision is needed. If you’re having trouble coming up with a way to test an idea and have the feeling that you won’t be able to roll back the test without consequence, you might be dealing with a Type 1 decision.

Examples? Introducing CTA at the top of the fold versus the end of the page is a type II decision. Testing variants of pricing is a Type I decision, and you should be extremely careful in doing so. Usually, # of type I decisions is much larger than type II, so you don’t have to scratch your head every day 😊

Every company does A/B testing in some way or another. The same way everyone focuses on the culture. But doing it the right way requires something more than hiring someone, and paying for a tool. It requires the right process and principles. We discussed what goes wrong with a/b tests in the last post. They will serve as the principles. In this one, we will discuss what’s the right way of conducting a/b tests. If you follow these methods even as a checklist, it will definitely help you improve the overall decision quality of the teams involved.

If you master them/ have already mastered them, I look forward to meeting you someday and learn :)

Steps to doing tests in the right manner

The whole process can be divided into three parts — before, during, and after the test.

Before the Test

STEP 1: Start talking to customers, stop listening to experts

Most experts offer you something dangerous - the confidence to make a decision based on some anecdote or examples from another company. Nothing wrong with that except you might run into three core problems:

• What works for other products won’t necessarily work for yours because your business model, product, culture, or customers are different from anyone else’s.

• Markets/ Customers/ Distribution channels are changing faster than ever. Things that worked for you or some competitor yesterday will likely be saturated and far less effective today. Most of the advice is outdated that way

• There are no silver bullets. No easy secrets to reach the stars. If it were that easy, everyone would do it :)

Instead, start talking to your customers you are building it for. It will give you the right kind of information and confidence to make decisions. In the next post, I will discuss at length how to talk to customers and generate ideas/insights. Subscribe here to receive it in your inbox if you haven’t already

Subscribe now

STEP 2: What are the key problems/questions?

If we talk to your customers, we will be able to come up with a list of customer problems

The unknowns of a product can also be framed as either a question or a problem. For example,

Low conversion rate as compared to the industry standard is a problem

High bounce rate on the landing page is a problem

User not able to find the right information to make the purchase is a problem

Though it’s not a prevalent problem, some teams jump from problems to ideas directly like,

Problem: Low conversion rate as compared to the industry standard

Ideas: we should offer discounts to the first time users, we should change the copy and image on the landing pages, etc.

Don’t do it. There is a subtle, yet powerful step in between — asking questions. All problems can be framed as questions

Why do we have a low conversion rate compared to the industry?

Why do we have a high bounce rate on this page?

Why are users not able to find the right information to make the purchase?

You will see how it helps in the next step.

Optional with step #2: Identify product themes

You can combine the problems into product themes. It helps you pick areas to focus on. This is especially important for startups/ small companies. You shouldn’t do experiments all over the place at once. In a quarter, focus on moving metric around one theme — acquisition, activation, retention, revenue, referrals. This way, the experiments will build on each other, create insight and depth of understanding in an area. They can also be connected to the broader product strategy.

STEP 3: Build questions - hypothesis- experiments

When you ask a question, you can come up with an exhaustive list of hypotheses for that question. Jumping directly from problems to ideas is usually incomplete that way.

Sorry for writing maybe so many times 😁 I want to ensure that you remember that all these are hypotheses and don’t be too sure of yourself.

Now you can move to the last part — defining the experiments to test these hypotheses.

One key issue that you may face at this step is too many hypotheses. Since everyone loves their hypotheses and ideas to test it, the only solution is to ask people to back it either with data or customer interviews.

If you are having trouble generating hypotheses, my next post on how to generate hypotheses/ideas will help you :)

STEP 4: Define success metrics up-front

Good experiments define success up-front. There are three benefits of defining success up-front:

1. You will properly instrument the metrics associated with the definition of success. Bad experiments get restarted because the right metrics weren’t instrumented from the beginning.

2. Bad experiments add many metrics to the dashboard, then cherry-pick the ones that improve. The teams will do analysis in endless ways to find a positive data point that can be used to justify the experiment. As the saying goes “If you torture data enough, it will confess to anything”. Avoid this at all costs.

3. Defining success upfront becomes a learning opportunity. Defining success upfront requires deep thinking about the problem and hypothesis. Once the experiment is over, you see the gap between what you predicted and what actually happened, and that builds your product/growth instincts.

This step also comes with a disclaimer: your org should be forgiving of failure to some extent. You can’t hold people accountable if the success metric doesn’t move as predicted. That said, no org is comfortable with 100% failure rates. Establish what level of failure is acceptable. If you are looking for benchmarks, 50% is a good place to start.

STEP 5: Track counter-metrics

Consider trade-offs while designing experiments. The whole product is a system and if you change some part of the system, it can affect other parts that aren’t directly visible.

For example, if you move up registrations, the registration-to-conversion (counter metric) may go down. Monetization features and ads often lead to some loss of retention and engagement (counter metrics).

Badly designed experiments focus on their own success metrics. They treat every improvement as a win, even if that win came at a cost elsewhere in the system.

Good experiments define counter metrics to make sure it doesn’t cost you more than you are gaining as a system.

During the Experiment

Avoid things mentioned in the earlier post

• Short-term and long-term behavior

• Dividing Users into Uniform Cohorts

• Statistical Significance and How Long The Tests Should be Run

After the Experiment

As long as we don’t torture data to confirm our biases, we should be fine. Keeping these pointers in mind also helps.

Improvement in one part of the funnel

Improvement in one part of the funnel doesn’t lead to an improvement in the whole. Sometimes, you see the signups increase which doesn’t lead to an increase in conversions.

It’s a very normal thing. Because you changed something, people got interested in signing up. But this may or may not lead to any improvement in conversions/revenue.

This is especially true about transaction apps like eCommerce, SaaS, etc. Attention apps like Facebook or Instagram operate differently. Their increase in sign-ups mostly leads to good improvement in metrics down the funnel-like time spent and retention. Similarly, a one-time purchase doesn’t lead to changes in retention.

This is why you shouldn’t equate wildly popular Facebook/Google growth stories around the color and size of a button to a transactional app.

Newness effect

I thank Ram, a friend and colleague, for bringing this point to my notice. The “Newness” effect is users clicking on the variant more often because it’s new. For example, 10% at the start of the experiment, tapering off over a few days. It matters because we might make a decision to go ahead with a variant which has a higher conversion rate compared to control in the first week, but lower than control beyond that.

From Google,

Consider a simple example:

• One original, one variant

• Variant true conversion rate (CvR) in long term: 1%

• Original (constant ) CvR: 3%

This could result in performance that looks something like this over time:

Most tools show the average conversion rate (in red). Note that the average conversion rate takes a very long time to approach the true conversion rate of 1% (day 8 in the graph).

Some tools like Google Optimise provide the solution to it by looking at how consistent/variable conversion rates are over time. They take it into account while showing the results. If you using some tool for a/b testing, is important to see whether the tool accounts for the newness effect.

Consequential changes (type I decisions or major launches)

You should monitor the success metrics and counter metrics for some time even after 100% rollout.

You should also keep an active eye on customer sentiments on social media/ reviews beyond the usual metrics.

Best Practices: A Tool of Learning to be a Super Product/Growth Person

A/B tests should be used as a tool for continuous learning. To do that, we need to document the questions, hypothesis, results of the experiment.

Documentation

You could use google sheets for documentation, or some of the advanced tools like Notion. Sharing this with other groups should be mandatory.

Learnings

“What do we do next with what we learned?” is a good place to start. Simply looking at metrics won’t help with learning, rather asking these question and documenting them helps

— Why did it work or not work?

— Does this prove or disprove our hypothesis?

— Was it a good test of the hypothesis? Any alternatives that we missed?

— If it failed, is there a chance that it could be successful if we invest more?

— If it succeeded, should we do more on this to make it even more successful?

I started this post with Bezos's framework for decision-making. I will wrap it with one of the paragraphs in the letter where he talks about experiments.

One area where I think we are especially distinctive is failure. I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there. Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. This long-tailed distribution of returns is why it’s important to be bold. Big winners pay for so many experiments.

We should remember this when you think about experiments next time — go for the upside, minimize the downside. Be bold wherever possible 💪

Sneak Peek into the next post

In the next post, we will cover

• Methods to generate questions/hypotheses/ ideas around for experimentation

• Debates around a/b tests: Algorithms in a/b, should tests be used to resolve conflicts, local maxima problem, etc.

Did I miss something? Please share your feedback/advice directly on LinkedIn/ Twitter. Alternatively, give a shoutout if you really liked it so that this free resource can reach others for learning 🙏🏻

Subscribe here to receive the weekly newsletter in your inbox so you don’t miss it

Subscribe now

See you next week,

Sincerely,

Deepak

The last few weeks have been crazily busy, but that shouldn’t deter us from writing and learning. That said, you can always provide me some extra motivation by sharing it on social media if you like the post 😉

If you don’t like it, please share your advice/feedback on how to improve it 😁

Off to the topic — in the last post around A/B testing, we covered the basics. It’s time to dive into the really interesting parts.

More than 3,500 people have subscribed to the growth catalyst newsletter so far. So consider subscribing to receive it in your Inbox👇

Subscribe now

If you aren’t familiar with the newsletter, you can read about it here

A/B tests are nothing new.

In the 1980s, Pepsi launched an advertising campaign to remove Coke from its No. 1 “cola-spot”. The ad showed that in a double-blind taste test, consumers preferred a sip of Pepsi vs a sip of Coke nationwide. They called it the Pepsi Challenge.

This blind test was a way of a/b testing two variants of soda, Coke and Pepsi.

Coke panicked. They put a lot of time, effort, and money to launch a sweeter version of Coke and called it New Coke. But sales went down for Coke, and Pepsi recorded the highest growth during New Coke’s first month.

Interestingly, Coke’s consumers were furious too. Over 400,000 letters of complaint came to the company. As it turns out, people only preferred a ‘sip’ of Pepsi because it was sweeter. But in their homes, while sitting in front of the TV and consuming a whole bottle of Cola, most people still preferred the old Coke and not something sweeter like Pepsi.

Eventually, the old flavor was brought back as Classic Coke and Coke has maintained a large market share even till today. In a 2013 article, Slate wrote “Coke won the cola wars because great taste takes more than a single sip.”

This is a great example of the limitations of scientific tests. They might reveal something, which when extrapolated might not be true at all. In a sip, they might prefer Sweeter Cola Pepsi but when it comes to consuming the whole bottle, they would go for Coke.

Such limitations fail the companies and are quite expensive. Hence when it came to writing about A/B tests, I preferred starting with the limitations.

Limitations of A/B Tests

We are all born with limitations, so why do we expect otherwise from A/B tests. Maybe because we don’t understand them fully, or maybe because the experts use the term ‘scientific’ too often with A/B tests. And how dare we question something scientific? 😊

So I am here to tell you that A/B tests have limitations, three in particular. These limitations arise mainly due to the way we run tests and the way we interpret them.

1. Short-term vs Long-term

It is hard to judge the right long-term behavior of the users. As most A/B tests only run only for a couple of weeks, they are almost always telling about the short-term behavior, but not the long-term ones.

An example of this which all of us have faced in our lives is the sales pitch. Sometimes, a company will use a sales pitch to create a sense of urgency for users, making false claims, and converting them into paid users. In a couple of business cycles, metrics like conversion and revenue would go up.

But give it a long enough time, and the problems with conversion will start appearing due to bad brand perception the sales team has created. In this situation, you can avoid this by devising counter-metrics like customer satisfaction rate along with conversion rates.

But normally it’s hard to do which is why I have put it in the limitations section. An example could be Facebook which is one of the biggest advocates of A/B testing. They have done some amazing things for the world through multiple products. But as for the blue Facebook (facebook.com), the product would have evolved into something different and our generation would still have loved it if they hadn’t relied solely on short-term CTRs on PNs, monetization, and engagement metrics.

Generally speaking,

A notification with a high CTR in the first month

A post with a high # of likes

A promise on the landing page that converts users at 2x rate but is hard to deliver in future

All of these will have a terrific success rate with A/B tests in the test duration (2-4 weeks), but whether they will create success for your product in the long-run is a hard question to answer and worth pursuing.

2. The Future Cohort of Users

“If you build it, they will come” is a half-truth of product development.

“If you build it and can reach some of them who love it, the rest will follow” should be the motto ✌️

The second limitation of A/B tests is that they don’t account for a future cohort of users. You can’t reach the users who haven’t downloaded and using your app right now. This is why PMs and marketers are puzzled when a strong hypothesis doesn’t work in tests.

The mistake that we make here is extending the results of an A/B test indefinitely into the future. In the future, the cohorts for which the test will work might join. This is especially true if you are in the early stages of product-market fit and haven’t reached all kinds of users. Look at this graph we discussed in GTM Strategy: Understanding Channels and figure out if you have covered all kinds of users

Consumers display different behaviors when it comes to the adoption of a new technology/product.

Specifically, the consumers can be divided into 5 parts on the basis of the ease of adoption. He calls them innovators > early adopters > early majority > late majority > laggards. There is a chasm between early adopters and the early majority, and if a product can’t cross this chasm, it will fail. Hence the name of the book “Crossing the Chasm”.

What we should be doing instead is looking at a hypothesis and understanding why it didn’t work.

Tests can often fail because of seasonality, or because you missed one tiny nuance, or because you don’t have the set of users for whom it will work.

You can avoid this to some extent by keeping a backlog of all tests and revisiting them periodically. Re-testing ideas should be an inherent part of a/b testing. 

3. The Unchanging World View

A/B tests also assume an unchanging world view. The same people who were uncomfortable with snapping few photos a few years back are making Tik-Tok videos now. The world of technology and associated user behavior changes fast. We should always keep that in mind.

Some of the worst ideas of the 1990s internet bubble have worked very well in the last decade. Grocery shopping where WebVan burned $800 million trying to deliver fresh groceries to your door in the 90s, we have success stories like Swiggy, Bigbasket, Amazon Fresh, and Instacart today. Internet-based currency and pets.com which failed miserably then, are doing pretty well now with cryptocurrency and chewy.com.

The bigger, billion-dollar ideas will take decades to work because they require a big shift in behavior and technology. The smaller ones that you are experimenting with might work in some form in a year from now on the same cohort of users.

The world is constantly changing, and that’s why re-testing becomes important to remove this limitation.

Beyond limitations, there are few things to keep in mind before starting an A/B test. There are a couple of assumptions that we make while reading the results of A/B tests. We should ensure those assumptions are right, or else we will end up looking at wrong results and making decisions — this would be worse than not doing a/b tests at all. It is important to maintain the sanctity of a test — the next section covers two important points.

The Sanctity of a Test

1. Dividing Users into Uniform Cohorts

When you are comparing the results of an a/b test, you assume that the group that saw version A is similar to the group that saw the version B.

While assigning randomly to a wide userbase, it may happen that a particular set of users are over assigned to one version.

Imagine building a gym app for a gym you run. One of the features of this gym app is that users can log their meals. Everyone who has a gym membership can log in to the app. There are two sets of users on this app —

• gym rats who come to the gym every day

• gym enthusiasts who get the membership but don’t visit often

Your goal is to improve the meal log rates so that you can help more members get to the right meal and exercise.

So you launched a reminder notification that reminds users every day at 9 pm to log their meals. You want to a/b test this notification. A month into the test, here are the results.

So you are now puzzled that the meal log rate has gone down because of notification. You check the significance of results and they look statistically significant.

So should you conclude that the notification is leading to a lesser # of meal logs? Not necessarily. Let’s check the segments now

As you can see, the meal log rate in both segments has increased to 90% and 41.9%, but due to the wrong random assignment, version A is showing better results than version B.

Dividing consumers into uniform cohorts is very important to maintain the sanctity of the test.

One quick way to measure whether your tool is diving users into uniform cohorts is doing an AA test.

An A/A test puts two exactly identical versions against each other instead of two different versions under A/B testing. If your testing tool is working well, there should be no difference between your control and variation versions. If there is a difference, time to stop all tests before you correct it.

Another thing you should ensure is tight exposure groups. Users in the control group shouldn’t be exposed to the change at any time during the test. If that happens, it will spoil the data.

2. Statistical Significance and How Long The Tests Should be Run

We all understand that reaching a statistical significance is important. Seeing the results early on before they reach significance and concluding either way, is no different than watching the first 10 minutes of the game and concluding which team will win the match. You may be right in few cases because of sheer dumb luck, but that’s not scientific by any measure.

Most of the teams I know hold this rule of statistical significance pretty well.

However, there is another question on similar lines — when should we stop the test? The simple answer is “when it reaches significance”.

This isn’t the right answer.

From Evan Miller on How Not to Run A/B Tests

The significance calculation makes a critical assumption that you have probably violated without even realizing it: that the sample size was fixed in advance. If instead of deciding ahead of time, “this experiment will collect exactly 1,000 observations,” you say, “we’ll run it until we see a significant difference,” all the reported significance levels become meaningless. This result is completely counterintuitive and all the A/B testing packages out there ignore it.

He argues that whenever a test reaches statistical significance, we can’t call it. So when do we call it complete? We need to answer two questions to find the right answer.

Question 1: How many users are needed for an A/B test?

We need to determine the sample size for the test before conducting the test, also known as the predetermined sample size. This can be done by using many tools out there and requires a baseline conversion rate and minimum detectable effect (MDE) as inputs.

• The baseline conversion rate is the current conversion rate.

• MDE is the smallest effect that we want to detect.

Let’s take for example a calculator from Evan Miller. Say we have a conversion rate of 3% and want to MDE of 5% relative change. The users needed for the test are 204,493 as shown in the image below from Evan Miller calculator.

I will not go into details but remember that 0.2 and 0.05 are conventional values for β and α and should be used in experiments. You can read more about it here

What if we want the MDE to be 2%?

So using the baseline conversion and MDE, we should find out the minimum sample size and complete the test only when the minimum sample size has been attained. If it does, the test isn’t valid.

Question 2: How long should we run the test?

As long as you need to reach a predetermined sample size. But the longer a test is running, the more susceptible it is to sample pollution. For example, visitors might be deleting their cookies and end up re-entering the A/B test as a new visitor.

Letting your test run for too long bad as well. Usually, it shouldn’t exceed a month.

If you are a small startup with 100s of visitors a day, you probably won’t reach the sample size in a month if your MDE is small. So focus on launching big things and other optimizations. We will cover more about this in the user research post.

Is there a minimum time for which the test should be run?

As a rule of thumb, it is dependent on the consumer cycle.

Take an example of engagement apps like Tiktok and Instagram, you can conclude the tests early as soon as you reach the pre-determined sample size because the engagement cycle for most users completes in hours and days.

On the other hand, transacting apps like Amazon might need more time because they need to account for all buyers including “I need to think about it” buyers, all the traffic sources, and other anomalies like the day of the week, etc. In transacting apps, a good rule of thumb that Shopify provides in two full business cycles.

I will end this post with one last point that should be remembered at all times while designing and conducting the experiments. It is as important as the other points mentioned above.

Bonus Point - Finding Causation instead of Correlation in Tests

From the book ‘Experimentation Works’,

Between 1500 and 1800, about 2 million sailors died of scurvy. Today we know that scurvy is caused by a lack of vitamin C in the diet, which sailors experienced because they didn’t have adequate supplies of fruit on long voyages. In 1747, Dr. James Lind, a surgeon in the Royal Navy, decided to do an experiment to test six possible cures. On one voyage he gave some sailors oranges and lemons, and other alternative remedies like vinegar. The experiment showed that citrus fruits could prevent scurvy, though no one knew why. Lind mistakenly believed that the acidity of the fruit was the cure and tried to create a less-perishable remedy by heating the citrus juice into a concentrate, which destroyed the vitamin C. It wasn’t until 50 years later when unheated lemon juice was added to sailors’ daily rations, that the Royal Navy finally eliminated scurvy among its crews. Presumably, the cure could have come much earlier and saved many lives if Lind had run a controlled experiment with heated and unheated lemon juice.

The good thing with A/B tests is that they are controlled experiments in general. The problem that happens sometimes is in the interpretation as it happened in the case above.

How do we make the right interpretation? An experiment should be simple enough that cause-and-effect relationships can be easily understood. If we include too many variables in a test, it will be hard to learn about what exactly caused the shift in the behavior.

This is one of the reasons large-scale redesigns are problematic. Even when the results are in the right direction, they fail to add any significant learning. This isn’t to say you should do redesigns if needed, but test a few of the strongest hypotheses around redesign first.

What if you can’t find the reason why it’s working?

What happens when a particular thing works and improves conversion, but you aren’t able to find why? It shouldn’t stop you from reaping the benefits.

Google tested multiple variations of the color blue until they found the perfect shade of blue to increase conversions at Google. From the Guardian,

"About six or seven years ago, Google launched ads on Gmail," Google UK's managing director Dan Cobley explained. "In our search we have ads on the side, little blue links that go to other websites: we had the same thing on gmail. But we recognised that the shades of blue in those two different products were slightly different when they linked to ads.

"In the world of the hippo, you ask the chief designer or the marketing director to pick a blue and that's the solution. In the world of data you can run experiments to find the right answer.

"We ran '1%' experiments, showing 1% of users one blue, and another experiment showing 1% another blue. And actually, to make sure we covered all our bases, we ran forty other experiments showing all the shades of blue you could possibly imagine.

"And we saw which shades of blue people liked the most, demonstrated by how much they clicked on them. As a result we learned that a slightly purpler shade of blue was more conducive to clicking than a slightly greener shade of blue, and gee whizz, we made a decision.

"But the implications of that for us, given the scale of our business, was that we made an extra $200m a year in ad revenue."

It’s hard to find the reason why a particular shade of blue works. What we know is that it leads to $200 m in revenue a year, period.

With this, I will wrap this week’s post.

Sneak Peek into the Next Post

The next post “This is the Way” will cover

• Steps to doing tests in the right manner

• Using A/B tests as a tool of learning to be a super product/growth guy

• Debates around A/B tests

Receive it in your inbox by subscribing if you haven’t

Subscribe now

Stay tuned and have a good day :)

Sincerely,

Deepak

Youtube | Twitter | LinkedIn

The first video in a series of videos around career, product, and growth is out 🔥.

Everything significant in our lives starts with a goal. So we should set it right. For that reason, I have chosen the topic “7 mistakes people make while setting goals/OKRs” 🎯

Here is the video —

Coming to the topic,

This is the first post on experimentation and user research topic. In this post, we will cover an introduction to a/b tests, key terms, types of a/b tests, and how to conduct them. In future posts, we will cover

Where and How Not to Do Experiments. Here we will cover AA tests, separating out assumptions and other things that help you do the A/B tests the right way.

How to Generate Ideas of Experimentation. It is important to get the right ideas for experimentation. One of the key methods here is user research which we will detail in the next post.

Doing User Research the Right Way. Benefits and pitfalls of user research methods like usability tests, contextual interviews, focus group discussions, etc.

So where do we start for this post?

It’s the US election season, so let’s start there! 😁

In 2008, Barack Obama made history. He was the first African-American US president to be elected. But he had a unique challenge when he was running for the nomination back in 2007 — most people hadn’t heard of him. Obama's campaign had to get support and raise money from all these people.

Dan Sirokar, who later co-founded Optimizely, was the Director of Analytics for the Obama 2008 campaign. Dan had been a product manager at Google before, and his role in the Obama campaign was to maximize voter registration, email sign-ups, and donations through the official website by leading a team of software engineers and analysts.

One of the key things that worked for the Obama Campaign was to experiment with images and words that appealed to the voters and donors. The experiment tested two parts of the first page visitors saw: the “Media” section at the top and the call-to-action “Button”.

They tried four buttons and six different media (three images and three videos).

Button variations

Media variations

The results were stunning. The winning media+CTA variation had a sign-up rate of 11.6% as compared to the original page which had 8.26%. It resulted in additional 2,880,000 email addresses, which translated into an additional $60 million in donations. The winner page was,

If you are old enough to remember Obama’s key message that election, it was “Yes, We Can Change”. Not far from winning variation on the website? 😊

Join 3,000+ smart, curious folks who have subscribed to the growth catalyst newsletter so far by subscribing👇

Subscribe now

What’s A/B Testing?

A/B testing is nothing fancy. It’s testing different variations of elements of a website or app like image, text, web page, sign-up forms, payment flows, etc.

The term A/B Testing denotes ‘testing whether should we go for option A or option B?’. Options A and B are variations of the elements.

Once you understand this, it’s a no brainer that all software products should do A/B testing to ensure they build the best version of their websites/apps for the users.

So where do we start? By understanding a few fancy terms so that you don’t feel like a muggle when people discuss A/B testing in a room.

Key Terms of A/B Testing

There are 4 key concepts you need to understand when it comes to A/B testing.

1. Control and variation

2. The goal of the test

3. The null hypothesis

4. Sample size, and confidence level

Control and Variation

As the term suggests, control is the current version available to all users on the product.

Variation is what we want to change in the product and test whether it’s better than the control.

The Goal of the Test

The goal of the a/b test is what matters to you and the reason why you are doing the a/b test in the first place. For example, the goal of the test in Obama’s campaign was increasing sign-ups. The goal should always be quantifiable.

Null Hypothesis

You will hear these daunting terms often with engineers or data scientists. Let me explain in simple words.

The null hypothesis assumes there is NO (null) difference between control and variation in achieving the goal. For the A/B test to be meaningful, the null hypothesis needs to be proven false by showing that one version is better than the other.

How do we prove that one version is better than the other? Enters sample size and confidence level.

Sample Size, and Confidence Level

We have already discussed sample size in designing surveys for product-market fit.

The sample size is the size of the population you have surveyed. The higher the sample size, the higher the confidence that your results will hold true for the entire population.

The same applies to websites, the higher the number of users who have been through the A/B test, the more confidence you can have whether one variation is better than the other. Let’s take an example.

Suppose 5000 people have been through your A/B test. The goal is to increase sign-ups. You have split the users 50-50 in control (A) and variation (B). The results you get is

A — 2500 users — 50 sign-ups

B — 2500 users — 55 sign-ups

Here you can see that B is converting at 2.2% (55/2500), whereas A is converting at 2%. So B looks better. But how confident are we that B is better than A?

You can use this calculator to find the confidence level — https://abtestguide.com/calc/

What if we increase the sample size to 500000

A — 250000 users — 500 sign-ups

B — 250000 users — 5500 sign-ups

Variation B is still converting at 2.2% (5500/250000), whereas A is converting at 2%. What about the confidence level?

What does 95% confidence mean? It means that in 95 of 100 cases, this variation will perform better.

Usually, when do A/B tests, we are looking for 95%+ confidence.

As you can see, the larger the traffic on the website, the quicker you can get the results.

How to Do A/B Tests

More often than not, your company might be using one or other tools of a/b testing. Larger companies built an in-house solution to a/b testing. Smaller ones pick one of the tools available in the market. Some of the popular ones are

1. VWO

2. Google Optimize

3. Optimizely

All of the three are market leaders in this segment, and you can try either of them.

When Doing A/B Tests

Beginner’s Mind

A/B testing requires a beginner’s mind, i.e. openness to test out ideas that may be unconventional.

An example of this came when the popular game Sim City 5 was launch. Electronic Arts, the owner of the game, wanted to A/B test different versions of its sales page to identify how it could increase sales exponentially.

Most marketers know that advertising an incentive will result in increased sales. So they offered 20 percent off a future purchase for anyone who bought SimCity 5 on the pre-order page.

The variation didn’t have this incentive.

The variation performed more than 40 percent better than the control. Avid fans of SimCity 5 weren’t interested in an incentive. They just wanted to buy the game.

Testing against conventional wisdom requires a beginner’s mind ✌️

Start Small

The variation should always be made live to a small % of users.

If you are a small-to-midsize company, you can start with 10% of users and see how the variation performs wrt control. if the performance is good, you can gradually keep increasing the variation to 20%, 50%, and eventually 100%.

Larger companies like Facebook and Google have enough users that they can start at 1% and gradually increase it.

You can determine where to start by looking at the population size. If 1% of your users are large enough to give you high confidence results within days or weeks, you should go for 1% first. If not, go for a higher %

What Can You A/B Test, aka Types?

Before we get into the types, it’s important to understand how websites/apps work for non-technical readers. In simple language,

In order to understand how websites work, try to remember the analogy of a restaurant. At the restaurant, the waiter interacts with the end-user. It takes their requests and places the order into the kitchen. The food requires some raw ingredients available in storage area. The chef takes the raw materials, prepares the food, and passes it to the waiter.

As you can see in this analogy, the customer is the end-user. The waitstaff is the front-end or client-side. Food is the website. The shelf or storage area of the restaurant is a database. The chef and database are at the back-end/ server-side.

Client-Side

Usually, ab tests are done on the following elements on the client-side.

• copy

• design and layout

• forms

• navigation

• Call to action (CTA)

These are all client-side tests and are support by all popular tools. They are also pretty common and easy-to-do because they don’t require any change to your back-end/ server-side logic.

From our earlier example of a restaurant, it is like ordering readymade biscuits. Easy for the waiter to serve 2 varieties of biscuits (A/B) to the customer to see which one they like better and doesn’t require a chef to do it.

Server-side Testing for Algorithms

You can go also conduct server-side a/b tests where the test requires the server to create different variations before it is delivered to the client.

Suppose you are building a recommendation engine for the eCommerce website. You can also a/b test whether the new recommendation algorithm performs better than the existing one by showing them to different sets of users.

You can do the same over the search results as well. These require deep experimentation capabilities and are useful for dynamic websites.

As these tests got popular, tools like Optimisely which started with only client-side testing have started providing provide full-stack testing.

From our earlier example of a restaurant, it is like ordering Omelets. The chef can prepare different Omelets (A/B) and serve it to different customers to see which one they like.

Split URL Testing

URL is the address of a website. Split URL Testing is testing multiple versions of your webpage hosted on different URLs. The main difference between a Split URL test and an A/B test is that in the case of a Split test, the variations are hosted on different URLs.

Split URL testing is preferred when significant design changes are necessary, and you don’t want to touch the existing website design.

It’s like building a variation of the restaurant at a different location and testing if customers prefer one versus the other. You don’t do it often as it requires significant investment.

MultiVariate Testing (MVT)

It is a method in which changes are made to multiple sections of a webpage, and variations are created for all the possible combinations of changes.

For instance, you decide to test two versions of each of the hero images and the CTA button color on a webpage like Obama’s campaign. By using MVT, you can create one variation for the hero image and one for the CTA button color. To test all the versions create combinations of all the variations, as shown here: 

After running a multivariate test on all combinations, you can use the data to see which one performs best and deploy that. In the image below, the winner is Hero Image 2 + Blue CTA with an 18% conversion.

Multipage Testing

Multipage Testing is a form of experimentation where you can test changes to particular elements across multiple pages.

Suppose you want to introduce social proof (3000+ paying customers) on your website. You add it on every page of the funnel in one variation and remove it in the other. You have gotten multi-page testing now 😊

Join 3,000+ smart, curious folks who have subscribed to the growth catalyst newsletter so far by subscribing👇

Subscribe now

This would be all for this post. See you next week,

Sincerely,

Deepak

In this post, I have shared the top resources I have come across while researching on Retention. A lot of readers requested this. Planning to keep it as the last post on the topic.

I wanted to take a couple of minutes to share my thoughts on the Growth Catalyst Newsletter so far and what’s coming next. In the last 6 months, we have covered 6 of 9 topics we started with in mind,

SECTION I: LAUNCHING THE RIGHT PRODUCT [DONE]

1. Product-Market Fit: 5 posts - May’20

2. Positioning: 4 posts - Jun’20

3. GTM: 4 posts -Jul’20

SECTION II: GAIN TRACTION [DONE]

1. Growth Analytics: 3 posts - Aug’20

2. Growth Accounting, Models and Loops: 5 posts- Sept’20

3. Onboarding and Retention: 5 posts - Oct’20

SECTION III: ITERATE AND GROW [NEXT THREE MONTHS]

1. Research and Experimentation: How to generate ideas and improve the metrics - Nov’20

2. User Psychology and Gamification: How applying game mechanics can put you to advantage - Dec’20

3. Mental Models for Growth: Bringing it all together and best models from various disciplines like behavioral psychology, finance, computer science, biology, etc. - Jan’21

So far, I have written 50 k words at Growth Catalyst. It’s a good milestone 😊 Looking forward to the next three months ✌️

3,000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

Off to the update,

In the post on resources around data, I wrote why I have started sharing these resources.

Depending on the kind of person you are, you might prefer one or more of the methods to learn about a topic.

Everyone has a different learning style. And that’s why I am going to share 5x3 resources on topics I write

• top 5 books

• top 5 articles/blogs

• top 5 videos/channels

Please feel free to choose the medium and learn :)

In line with this thought, I have also started making videos around product, growth, and career recently on the YouTube Channel.

Here you go!

top 5 articles/blogs

1. Mastering Retention by Amplitude — Most extensive guide on the topic of retention. This one is my favorite.

2. How to Increase Your Product Retention by Lenny Rachisky — This is a good blog post by Lenny and will supplement whatever you read in the retention masterclass series at Growth Catalyst.

3. Intercom on Onboarding by Intercom— Another playbook by Intercom. As Intercom is a user onboarding product, so you can be assured that they know what they are talking about

4. 6 User Onboarding Flows by CXL — The post starts with

   “There’s a popular user experience quote: “A user interface is like a joke. If you have to explain it, it’s not that good.” While clever, that statement is far from true.”

   A very strong start gives you a sense that this post is going to be good. It also gives you a good sense of how to study the onboarding flows of different products and learn from them.

5. Why Retention Is The Silent Killer by Brian Balfour — Another good post by Brian

top 5 videos/channels

1. Why Customer Onboarding is the Most Crucial Part of Your Growth Strategy by Casey Winters: Casey is the CPO of Eventbrite and has built growth courses at Reforge.

1. Growth for Startups by Gustaf Alstromer: Gustaf has mentored multiple teams around growth at Y Combinator. In this video, he covers product-market fit, retention, and a few more topics.

1. How to Think About Onboarding by Colin Bentley, Senior Director of Product at Inside Intercom

1. How to Improve Conversion Rates by Kevin Hale at Y Combinator: Conversion rate improvement is the ultimate goal of Onboarding and a leading indicator of retention. So added one video around that.

1. Growth Masterclass by Chamath: This is a video I share every time someone asks about how to grow a product. Retention is a function of all the small things that you do on a day-to-day basis. Nothing exemplifies that better than Chamath, who built the growth team at Facebook.

top 5 books

1. Hooked: How to Build Habit-Forming Products by Nir Eyal: Hooked is a good book about how user habits get formed. User Habits = High Retention

2. Hacking Growth: How Today's Fastest-Growing Companies Drive Breakout Success by Sean Ellis: A complete book that covers all the topics of growth comprehensively. Probably the only decent book on product growth.

3. The Design of Everyday Things by Don Norman: To be good at Onboarding, it’s vital to understand the fundamentals of design. This is the definitive book on UX design fundamentals.

4. Subscribed: Why the Subscription Model Will Be Your Company's Future - and What to Do About It by Tien Tzuo: Studying subscription businesses is a masterclass in retention because you keep seeing user behavior and preferences month on month.

5. The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses- The lean startup is one of the cult books around product development. It explains the importance of testing your product with users early on and building based on those insights. The methods discussed in this book can help you build the right product for the right audience.

Hope you liked the suggestions. DM me for any suggestions/feedback :)

See you next week ✌️

Sincerely,

Deepak

A couple of weeks back, I surveyed the newsletter readers to publish video content on Youtube actively. The results are out 🔥

Creating content around concepts written in the newsletter (complex ones) and career advice sounds a good start. For now, I will publish one video every week. The first video drops next Wednesday on this channel. Check the YouTube channel if you haven’t

Check the Youtube Channel

Off to the topic at hand,

It’s hard to write an essay on how to fix retention problems. There is a lot of good-sounding advice on the internet. You can create a plan around it, but as Mike Tyson said about plans.

Just as every fight is different, every product retention problem is different. This is why despite so many good sounding plans around ‘how to fix retention and product-market fit,’ most companies fail because of retention and lack of PM/F.

How do you prepare for a punch in the arena? You do it by learning and practicing the fundamentals of boxing. You look beyond your discipline, do cross-training, and pick on different skills. In the end, be ready to be knocked down, and have the courage to stand up.

Like boxing, the solution for retention also starts with recognizing that it is a hard problem to solve. We should learn the basic tools for solving this problem — positioning, PM/F, analytics, models, etc. All the essays I have written so far at Growth Catalyst need to be applied while solving retention. Retention is The Endgame 🎯

3,000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

The Problem with Bullet Points

You will notice that all the top essays in google results around fixing retention have the same list of bullet points. The problem with many bullet points is that they miss the problem's context and become a massive to-do list. They also don't help you think deeply about the problems.

As David Epstein writes in his book Range,

“Whether chemists, physicists, or political scientists, the most successful problem solvers spend mental energy figuring out what type of problem they are facing before matching a strategy to it, rather than jumping in with memorized procedures.”

Instead, if you start with a few key things from first principles, they are very effective. They are easy to remember, argue, and think. We are going to cover 3 key frameworks to fix retention woes.

Framework #1: Fogg’s Model to Build Behavior

What we are trying to do is change the behavior of the users to improve retention. To change the behavior, we can pick a few theories from behavior science and see how they apply in improving retention.

Let’s start with the basic question — what causes a particular behavior? The model that explains it is developed by BJ Foggs, who built the Behavior Design Lab at Stanford University. He and his team research human behavior and how you can influence it to achieve business goals. 

The Fogg Behavior Model shows that three elements should converge simultaneously for a behavior to happen: Motivation, Ability, and a Prompt. When a behavior does not happen, at least one of those three elements is missing.

With ample motivation and ability, the prompt shapes our behavior.

For example, when you receive a notification about a discount on a product you have been thinking of buying, you always check it out. The motivation, in this case, is high. Checking out the offer doesn’t require much effort, so your ability to do it is high. The notification is the prompt or trigger.

You may have come across apps that send a lot of notifications. An example is the notifications we get on the Facebook app nowadays. The novelty of such notifications is lost for some of us because FB bombarded us with irrelevant notifications about Candy Crush for years 😈 As a result, we have lost the motivation to check it out, so the behavior is different in this case.

Please note that prompts succeed only when either one of the factors is very high, or both are moderate. It fails when both are low. An example to this would be IRCTC website. Even if it’s not simple to book a ticket there, the motivation is super high to book a tatkal ticket and you take all the effort to book it.

Now that you have understood the basic definitions, let’s dive deeper into these factors to see how they can be applied.

Motivation — There are three core motivators for humans

1. Sensation: pleasure or pain

2. Anticipation: hope or fear

3. Social: social acceptance or rejection

Here are some products representing different motivations in the table.

Ability — Ability refers to how simple it is for someone to do something at a particular moment in time. The simpler it is, the higher the ability. Fogg outlines six ways that a task can be made simpler.

1. Time - Reducing the time required

2. Money - Reducing the money required

3. Physical effort - Reducing unnecessary physical effort.

4. Mental effort - The lesser the cognitive load, the easier it is

5. Social deviance - Ties to social motivation. No one wants to be the outcast.

6. Habitual - Habitual things are easier; non-habitual things are harder.

Examples of different ways products improve the ability to do something.

The above is an image, and the Read More links in the table are here — Time, Money, Physical Effort, Mental Effort, Social, Habitual.

Prompts — Prompt is simply a reminder. This reminder can be a push-notification, in-app notifications, an ad while scrolling social media, or watching YouTube videos.

You can add extra motivation or ability through prompts, like adding a discount (money) coupon or telling the user how many people are already using the product (social).

A lot of products give extra coupons to users who have churned out of the system. This is a prompt to come back while solving for the ability+motivation. Check out this email you receive from Fotolia (now Adobe Stock) after you don’t log in for a few months.

Fogg’s Model and Product Retention

Let’s now look at some of the key strategies we can employ to improve product retention. You will notice that a single strategy ties to multiple motivation/ability factors. For example, suppose you show users that 100,000 users have bought your product. In that case, you are making it socially acceptable to use the product and also makes it easier for them to make the decision (reducing user's mental effort).

The idea behind discussing these pointers is to show how we can develop various ideas using the simple B=MAT model.

Retention can be increased by

• Acquire the right kind of users — Acquire the users who have the need and capacity to pay. Look carefully at your acquisition channels and user profiles. Stop acquiring from channels that don’t give you sticky users. Stop acquiring groups (age, gender, geography) that don’t stick with the product.

• Bringing users to their ‘Aha! Moment’ — If the product solves one of their problems/pain points, and the user realizes it, they have a higher motivation to stick with your product. This can happen through the right onboarding experience.

• Build delightful experiences — A good design is soothing to our eyes. We associate good design with trust as most things that look good are also good. Apple is a prime example of this. Airbnb photos case study, as mentioned in the last post, is another example of this. Ties to pleasure.

• Add social proof — Adding how many people are already using your products, ratings, and reviews from end consumers can drastically improve activation and retention.

• Mobile apps versus website — If your products have daily usage, launching a mobile app can increase users' ability to check the app whenever they want as the mobile phone is just at an arm's length. It’s also possible to send prompts on mobile apps, compared to websites.

• Offering product at a price the TG is comfortable with — Sale events increase overall purchase/retention from consumers as there is a fear of missing out. The price also becomes reasonable for an additional cohort. This is why you see sale events across the globe.

• Minimizing effort — If you are a Kirana shop or a website selling physical items, you should provide doorstep delivery to reduce the users' effort. Add to that easy returns, multiple payment options including cash-on-delivery, easy to order through Whatsapp versus calling, etc. and you have a huge improvement in retention coming your way. No wonder people are so bullish on e-Commerce and WhatsApp. This is also why malls work better than standalone shops. You don’t have to travel much to go through different shops in a mall.

• PNs and In-app notifications — Make a cocktail of prompts, ability, and motivation through PNs.

• Make it harder to churn — Though not advisable, many companies try to make it harder for the user to churn. Few give you a special discount when you are trying to churn like Fotolia in the example above. Few others don’t offer cancellation through a single click, but make you go through a tedious process. It may lead to short term retention and growth but hurts your brand and trust in the long run. Try canceling cable :)

• Social sharing — Building social share in an app works wonders because it acts as a prompt and social proof. Just ask PinDuoDuo, a $100 B startup out of China. From YC piece on PinDuoDuo,

  The core of the Pinduoduo experience is team purchase, where buyers form a group in order to receive discounts from suppliers. The user experience, as laid out in the graphic below, is as follows: (1) for each item, merchants decide two prices – one for individual purchase and one for team purchase. If the user opts for a team purchase, he or she may either (2.1) initiate a team purchase, or (2.2) join an existing team purchase. If the user has initiated a team purchase, he or she may use social platforms such as WeChat to proactively encourage friends to join their team (3.1) or more passively wait for other buyers to join the purchase on the Pinduoduo (Pinduoduo) platform (3.2). A team needs to be formed within 24 hours to have the order confirmed. Once a team is formed, the purchase is confirmed and the product is shipped within 48 hours.

  

  Nearly all Pinduoduo transactions are completed using team purchase. In the early days, the size of many groups was large (e.g., 10+), but as Pinduoduo has scaled the group size requirements have declined. Team purchase is beneficial to both buyers and manufacturers; buyers benefit from better prices for goods they want and sellers benefit from increased demand and better visibility of future demand. In addition to driving down prices, team purchase helps solve the “trust deficit” of retail in China’s developing cities, where more than 80%5) of retail is unorganized and consumers rely heavily on social recommendations to initiate transactions.

We have now explored a few methods to increase retention. There are numerous others and what will work for you depends on the user personas and the product you have. Once you have identified retention problems, it becomes easier to see them through the B=MAP lens and solve it.

We will study many more methods in the next 3 months, covering experimentation, mental models, user psychology, and gamification.

Framework #2: Super Apps to Improve Retention

Like the malls, the case for a super app can be built to increase retention using the same analogy. If we build up for more use cases, more people will come and retain. China has shown the world that it works through WeChat. However, it hasn’t been successful anywhere else, especially in the US and India. The effort to build a Super App fails because the US has a diversified internet ecosystem. In India, the same players are operating. When smartphones arrived in the US, there were already giants in search (Google), eCommerce(Amazon), and social space(Facebook). This made it difficult for any one player to succeed. China was a mobile-first country as when smartphones arrived, only 16% of the country's population was on the internet in 2007.

As China had already seen mature companies in the US in different sectors like digital payments, eCommerce, and chat, China's big tech-focused on building a super-app ecosystem by using their dominance. Let’s look at the timeline of WeChat

2011: WeChat launched as Tencent’s mobile-only messenger. 100m users

2012: Official accounts from stores and brands with QR code support and features to connect offline-to-online

2013: Moments (like Facebook feed) launched. 300m users. WeChat Wallet and mobile payments also launched

2014: Partnership with apps like ridesharing app Didi Chuxing, JD.com for the online retail marketplace. WeChat stores launched for any business to open a store within WeChat. More than 100m registered Wallet users.

2015: More than 1 billion in spending on red envelope gifts, subsidies, coupons, and incentives for users to adopt the WeChat Wallet.

2016: Bike-sharing within WeChat.

2017: WeChat Mini Apps/Programs feature released. Search engine features are released to go through all WeChat content. Subway system integration.

3-4 things worked for WeChat and failed to work for others.

1. WeChat didn’t face the regulatory hurdle that Microsoft did back in 2000 when they got an anti-trust case filed against them. Part of the reason is that governance is very different in the US and Chine. Maybe, Microsoft could have done it if not for their anti-trust woes.

2. Unless you take a sizeable market in an industry in the internet era, it’s hard to build a sustainable business. As you can see through the timeline of WeChat, it becomes clear takes money, time, and effort to dominate different industries.

3. If the position is already occupied by equally powerful apps in the user’s mind, it will be even harder to displace that. This is a stark difference between a mature market like the US, where multiple strong apps dominated different industries, and an early market like China back when Smartphones exploded in 2007-12.

But don’t despair; we can learn something from Super Apps when it comes to retention. We can always solve adjacent problems and give users more reasons to come back. For Swiggy, it’s grocery and alcohol delivery. For Zomato, its food delivery. For Microsoft, it was cloud and productivity software.

Framework #3: The Dark Science of Getting Users Hooked

I couldn’t write a better introduction to get users hooked, so loaning this from the 1843 magazine.

In 1930, a psychologist at Harvard University called B.F. Skinner made a box and placed a hungry rat inside it. The box had a lever on one side. As the rat moved about it would accidentally knock the lever and, when it did so, a food pellet would drop into the box. After a rat had been put in the box a few times, it learned to go straight to the lever and press it: the reward reinforced the behaviour. Skinner proposed that the same principle applied to any “operant”, rat or man. He called his device the “operant conditioning chamber”. It became known as the Skinner box.

When B.F. Skinner performed further experiments with his box, he discovered that if the rat got the same reward each time, it pulled the lever only when it was hungry. The way to maximise the number of times the rat pulled the lever was to vary the rewards it received. If it didn’t know whether it was going to get one pellet, or none, or several when it pulled the lever, then it pulled the lever over and over again. It became psychologically hooked. This became known as the principle of variable rewards.

Why does the variable reward gets the rat hooked? Dopamine.

Dopamine is a chemical produced by our brains that plays a big role in motivating behavior. It gets released when we take a bite of delicious food, when we have sex, when we exercise, and when we have good social interactions.

The human brain contains four pathways in which dopamine is produced (not getting into detail). All pathways become active when anticipating or experiencing rewards. Anticipation is the key here, which often happens when you do certain activities that result in rewards. For rats, the action was pulling the lever that built the anticipation. Every time a response to a stimulus results in a reward, these associations become stronger.

The variable rewards are what makes pulling a lever in casinos so addictive. These rewards in the internet world could be a text message from a friend, a “like” on Instagram, or a Facebook notification about a friend request.

Nir Eyal’s book Hooked is based on these nuggets of behavioral psychology. He explained that to build a behavior, we need to think about the variable rewards, just like the Skinner box. More specifically, it works like a loop.

Let’s take the example of Facebook.

• The trigger or prompts there is a push notification.

• Action is clicking on that notifications and checking out the new info/details. Checking out new info gives you a dopamine hit.

• Checking out the info/details sometimes gives you happiness, say when you gained many likes and comments on your posts, and sometimes it doesn’t. This feeling is variable rewards.

• Submitting a post or photo and connecting with friends is the investment on Facebook. This increases your likelihood of returning.

Over time, you become addicted to dopamine, and whenever you are getting bored and need some motivation/dopamine, you open Facebook. It becomes an internal trigger then.

Many apps use this science of hooking users to their smartphones. It is a questionable practice, and now companies like Google have started working towards ethical design and well-being.

You don’t need to go to the extent of making users addicted to the app. You can do a couple of things to improve retention though

• Build opportunities for users to get invested in your product. This will increase retention and also switching costs.

• Use variable rewards to get users back to the apps. eCommerce apps providing different discounts in different weeks/months is a good example.

Few Things to Remember before We Wrap Up

Don’t ‘set it and forget it’ once you have cracked retention.

It would be best if you managed it actively. The first reason is that competition will always kill your retention as people can move to another product, which keeps shipping better features.

The second reason is that a new segment of customers will keep coming unless your product matures. This new segment might need something new. Even the same segment preferences might change in light of a shift in technology.

If you aren’t able to move the retention up

Don’t give up too early; wait at least a year before giving up. But you need to seriously give it a thought if you have been trying to improve retention for > 12 months and haven’t reached even local maxima.

Go back to the whiteboard and analyze the market, its need, and preferences after that.

3,000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

With this, I wrap up this week’s post. We will cover experimentation in the next few weeks.

Thanks,

Deepak

This is Part 2 retention masterclass. In case you haven’t read Part 1, I would suggest you give it a read. For the new ones here, do check out the other posts that I have written if you haven’t

1. Measuring Product-Market Fit

2. Understanding Positioning: Through a Practical Lens

3. How to Engage Users Through Gamification

4. Referrals: The Holy Grail of Growth

5. Decoding Growth: Where is the Growth Coming From?

3,000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

Off to the post,

“It's supposed to be hard. If it were easy, everyone would do it.” ― Tom Hanks in A League of Their Own

The quote refers to baseball in the movie. Though it’s true about anything in general in life. What does it have to do anything with retention? You can say the same about retention when it comes to building and growing products. Retention is worth chasing without a doubt, but also one of the hardest metrics to move.

So when you want to improve retention and go to Google searching for answers, there is a lot of advice around what to do to improve retention. The problem with this advice is that it makes it all look so easy.

Which begs the question — “if it were so easy, why isn’t everyone doing it?” Because it’s not.

Recognizing that retention is a hard-to-move metric is the first step in improving it. I speak it from my personal experiences because I have seen people including myself taking ambitious goals on retention. Setting big goals isn’t problematic in itself, but having wrong expectations that few features in onboarding, notifications, user flows, etc. is going to fix retention in a quarter is a problem.

So where do we start? In the last post, we covered how to measure and benchmark retention for any product. In this one, we will see how to discover the problems with retention.

Are you wondering if you have a retention problem?

Do I have a Retention Problem?

Broadly, we can classify the retention problems into two types.

Type I: Curve flattening

The curve is flattening but lower than benchmarks.

I wrote about how to get the benchmarks in the last post so you can check it out if you are unsure about how to benchmark.

Type II: Curve not flattening

Look at the daily/weekly/monthly retention curve. If it’s not flattening like the red line, we are in a tough spot. Exactly how troublesome is this? We can understand it through scenario analysis.

Scenario Analysis

To do scenario analysis, we will take a few variables and make certain assumptions.

Type I retention — If it’s flattening, let’s assume it flattens at y% in n days

Type II retention— If it’s not flattening, let’s assume it reaches 0% in n days

NU = Avg. new daily users

x% = Avg. daily retention for the first n days

We can write the DAU equation for a year (365 days) as follows:

DAU in a year = NU* [(x*n + y*(365-n)]

Scenario A: You have to get to 50 k DAU in 12 months

y = 10% in Type I and 0% in Type II, n=90 days, x= 25%

For Type I, we need 1156 new users daily. INR 36.5 L in 12 months.

DAU = 1000* [(0.25*90+0.1*(365-90)] = 1000*[50] = 50,000

For Type II, we need 2222 new users daily. This will cost us INR 80.3 L in 12 months.

DAU = 2222* [(0.25*90+0*(365-90)] = 2222*[22.5] = 50,000

To reach 50 k DAU, we need to spend 2.2x money in type II compared to type I.

Scenario B: You have to get to 100 k DAU in 24 months

y = 10% in Type I and 0% in Type II, n=90 days, x= 25%

To reach 100 k DAU, we need to spend 4x money in Type II compared to type I

As you can see, the requirement of money keeps increasing with time to get the same outcome in Type II.

Scenario C: Let’s stretch the time to 5 years and target 1 million DAU

y = 10% in Type I and 0% in Type II, n=90 days, x= 25%

To reach 1 million DAU, we need to spend ~9x money in Type II compared to type I

Retention creates a big gap in long-term sustainable growth. The gap keeps widening over time. This way, retention has a huge cost to the company.

The problems with retention are also not obvious in the beginning, so startups don’t care much about it. Over time, they realize the mistake but since retention takes a long time to fix, sometimes it’s too late for a startup to fix this problem and get sustainable growth.

Now that we have a good sense of the two types of retention problems, we can clearly see that Type II is a difficult problem to have. That said even if it’s a Type I problem, one should remember that competing products with higher y% have a big advantage over you in the long run, and the sooner you fix it the better it is.

Why is my retention low?

It’s clear from the previous section that there are two objectives to chase around retention.

1. Flatten the curve (move y% from 0 to a non-zero value)

2. Shift the retention curve up (increase y%)

We can’t achieve these goals without knowing the root cause of low retention. Knowing the root cause is the stepping stone in fixing retention problems.

There are three reasons why your retention could be actually low

• You are attracting the wrong users to your product. Here acquisition is the issue. You can fix it by figuring out which kind of users are finding value in your product.

• Users aren’t able to see the value in your product. You can fix it by building the right onboarding experience. Here is a detailed post I wrote on this.

• Your product doesn’t have any value to offer for the end-user. This one is hardest to fix because this means that the entire foundation of your product is questionable. This might lead to a pivot, or killing the product.

But how do we figure what’s the reason for the low retention?

Figuring the reason for low retention

Diagnosing the problems in retention requires both Quantitative (Quant) and Qualitative (Qual) methods.

The qualitative method involves collecting and analyzing non-numerical data (e.g., text, video, or audio) from users to understand them. We get the non-numerical data through interviews, focus group studies, forums, etc.

The quantitative method relies on numerical data that is generated through the website/app, surveys, etc.

We start with quant as that is readily available in the analytics tool. Here we need to answer a few questions. You can ask the data to team figure out these from the analytics tool or do it yourself.

You are attracting the wrong users to your product

Bounce rate across segments

To start with, you can just take a look at the bounce rate across different segments of users. The bounce rate is defined as % of users who come to your website/app and do nothing. As a rule of thumb, if the bounce rate of a segment > 40%, you need to look deeper. You can benchmark the bounce rate of similar websites using Similarweb.

Activation and retention across segments

You should also check the activation and retention metrics across segments like Demographics (age, gender, geography), marketing channels (are some channels bringing you the wrong set of users?), etc. Usually, if there is a big gap in bounce rate, activation, and retention across segments, you can eliminate the segments which aren’t performing and improve your overall retention.

Users aren’t able to see the value in your product

Activation rate

What % of users are experiencing the aha! moment? In other words, figure out the activation rate for the overall product and benchmark it with similar apps in the category.

Success patterns

If you are finding it hard to figure the aha! moment of the product, finding correlations between actions and retention (successful behavior) can give you the answer.

Facebook did this by analyzing and figuring out that if people had more than 7 friends in the first 10 days, their retention was high. So they build products which made finding and adding friends easier like contact list based suggestions, people you may know, etc.

It should be noted that a strong correlation between an action and retention doesn’t mean that more people doing it will definitely lead to higher retention. But it’s worth a shot because in many cases, it would.

Your product doesn’t have any value to offer for the end-user

Surveys

To evaluate this, we need to use the SATISFY framework. What % of my current userbase would be disappointed if they would not be able to use the product in the future?

You can answer this by doing a survey. More about this survey here. This % should be higher than 40%. Use the SATISFY framework to also narrow down the TG that loves your product, and understand why. If you don’t find many people who will be disappointed, it is a red flag that people don’t see any value in your product.

You can also use NPS surveys to understand customers.

L30 curve

L30 curve gives you a good sense of usage frequency and stickiness. Look at the L30 curve for the cohort that experienced the aha! moment. If the curve looks don’t look good, it’s a sign of lack of value.

Bonus question for monitoring product health

Retention across cohorts

If you are already working on improving retention, a good thing to monitor the health of the product would be measuring retention across weekly or monthly cohorts. Is it moving up?

Figuring the reason for low retention through Qualitative Methods

The quant analysis helps us figure what’s wrong, Qual will tell us why it’s wrong. Understanding the why is equally important as it helps us take the right measures to course correct.

Though there are many qualitative research methods that you can read about, what I have found most useful are usability testing and one-on-one user interviews. I will be covering both in next month's essays around experimentation.

Usability Testing

Usability testing is assessing whether your product is usable or not. This involves sitting with a user, letting them use the app/website to perform certain tasks. We then observe whether the user faces some issues in getting the task done.

For example, if you are a PM at Make My Trip (MMT), you can ask users to book a ticket using MMT and see the problems different users face. Here is a good diagram explaining usability testing.

User Interviews

User interviews involve talking to users to understand their problems, motivations, aspirations, etc.

User interviews are the holy grail of insight building and like all the best things in life, are often ignored by PMs and marketers. Here is a good primer on user interviews if you want to read more.

More on this next month.

As you can only do limited sessions with users because of time constraints, some of the findings from the qualitative methods need to be validated further. The validation can come through surveys. You can ask specific questions in surveys to a higher number of users (>1000) to validate these findings and be confident of them.

So much Gyan! Let’s get to some real-life examples of how products solve their retention problems.

Retention problems and how companies solve it

Airbnb — From the first round review article,

At the time, Airbnb was part of Y Combinator. One afternoon, the team was poring over their search results for New York City listings with Paul Graham, trying to figure out what wasn’t working, why they weren’t growing. After spending time on the site using the product, Gebbia had a realization. “We noticed a pattern. There's some similarity between all these 40 listings. The similarity is that the photos sucked. The photos were not great photos. People were using their camera phones or using their images from classified sites. It actually wasn't a surprise that people weren't booking rooms because you couldn't even really see what it is that you were paying for.”

Graham tossed out a completely non-scalable and non-technical solution to the problem: travel to New York, rent a camera, spend some time with customers listing properties, and replace the amateur photography with beautiful high-resolution pictures. The three-man team grabbed the next flight to New York and upgraded all the amateur photos to beautiful images. There wasn’t any data to back this decision originally. They just went and did it. A week later, the results were in: improving the pictures doubled the weekly revenue to $400 per week. This was the first financial improvement that the company had seen in over eight months. They knew they were onto something.

Airbnb solved its activation problem by putting beautiful photos of apartments. They spent a lot of time during the early days of Airbnb talking to customers in NY and improved the product on those insights. Some of these insights like better photos, more information about listings make so much logical sense that you don’t even need to validate further. These insights are kind of ‘aha! moments’ of product development for the founders and PMs :)

Intercom.io — Let’s look at how Intercom analyses the success and failure patterns to monitor retention and finding ways to improve retention. From Intercom blog,

Intercom.io’s Des Traynor argues that not all churn is equal, and that cancellation is the final straw in a process that begins much earlier. Thus, Intercom.io’s automated emails focus on re-engaging entire teams whose interaction is lagging.

The Research

Traynor identifies “Activity Churn” as a phenomenon that occurs prior to cancellation, as depicted in the chart below:

Before cancellation ever happens, there are clear signals that a customer is in danger of churning. Engagement flags, and your product or service goes from an everyday occurrence to being used once a week, then once a month. Finally, they decide it’s a waste of money altogether.

How it helps?

The most effective time to address churn is when you see a drop in usage across an entire team. This is where many auto-mailers get it wrong, spitting back “We miss you, please come back” emails to users who simply took a week vacation. For most SaaS apps it’s acceptable for a user from a team to disappear; it’s when the activity of an entire team is dropping that you worry you’re losing a customer.

When you identify previously active customers who are slipping, a well-timed personal email can help to re-engage them. The best case is that you recover a customer; the worst case is you learn why you users leave. Either of these leaves you in a better position than doing nothing.

The key is understanding that churn happens long before customers decide to cancel.

Hopefully, I have convinced you to do a few things (quant/qual) to actively identify and monitor retention. Off to questions,

Q: What if I can’t flatten the curve? Can my business still work?

A: Upfront payment products like online courses at Udacity or upGrad can still work in absence of any meaningful retention. For example, Udacity charges $500 for a course and acquires a customer at $150. It is already operating at an LTV > 3x of CAC.

The caveat here is that if users don’t complete the courses after buying it, it is an indicator that these online courses are creating not much value. This means lesser word of mouth and negative recommendation. This will hurt the company eventually.

So such companies with the upfront payment model should actively monitor the NPS and completion rate of the courses. Or it will come to haunt you just like retention.

Q: What’s next?

A: Next post is a step-by-step guide on what can you do to improve retention. The problem with such guides is they become a list of bullet points and useless. I will try to avoid that actively :)

That’s all for this week. Goodbye, and see you next week. Comment or DM me in case you have any questions/suggestions.

Sincerely,
Deepak

As we covered Onboarding in the last couple of posts, let’s cover retention now.

For the new ones here, do check out the other posts that I have written if you haven’t

1. Measuring Product-Market Fit

2. Understanding Positioning: Through a Practical Lens

3. How to Engage Users Through Gamification

4. Referrals: The Holy Grail of Growth

5. Decoding Growth: Where is the Growth Coming From?

3,000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

Off to the post,

Albert Einstein reportedly said, "Compound interest is the eighth wonder of the world. He who understands it earns it; he who doesn't, pays it"

And a century later, Warren Buffett became one of the greatest investors of all-time understanding the power of compound interest. Buffett understands compounding in a way most people don't. But why?

Because compounding is not intuitive to our brains, so we ignore its potential and focus on growing money in another way. Look at this chart. >99% of Warren Buffet's wealth came after his 50th birthday, and this guy is investing since he was 11. Notice $67 million at the age of 47?

Does the graph look intuitive to you?

While we are talking about wonders, is there one when it comes to product growth? Yes, there is, and you may have guessed already — retention. Retention is one thing that, when done well, will forgive all of our sins in product development.

Like compound interest, retention isn't intuitive to founders, PMs, marketers. We go after performance marketing to get more signups to grow products because our brains can easily understand n million signups.

We also try to make buttons more appealing or redesign the entire app/website hoping that things will improve. But to our surprise, it doesn't help most of the time.

Retention is the open secret of product development. Those who understand it build products the right way, those who don't keep chasing vanity metrics. And it's not our fault entirely; we keep hearing about MAU, registrations, signups, etc everywhere. Retention isn't usually discussed in their quarterly reports.

And that's the reason for writing the three posts around retention. In this post, we will understand retention and its nuances. In post #2, we will cover the ways to diagnose the problems with retention. Then we will talk about ways to improve retention in the last one (#3).

Let's start with defining retention.

Defining Retention

The fundamental definition of retention is retaining the new and existing users of your product.

Different teams measure retention in different ways. The key reason for the difference in measurement is that in order to make retention metrics useful for your product, you have to define it according to your product usage and business model.

The top 3 ways to define retention are

1. N-day retention: % of new users who come back on a specific day (N). For example, 7-Day retention would be % of new users coming back on the 7th day.

2. Rolling retention: % of new users who return on/after a specific day.

3. Range retention: It is a variation of N-Day retention where we can pick a range to calculate retention, says Day1-3 retention.

Let’s try to check our understanding through the table below. What does Day-7 retention look like? What about D3 rolling retention? Day 1-3 range retention?

Hint : We have limited data set so can calculate the numbers for a 7 day range only. D7 = 1%, D3 rolling retention = (D3+D4+D5+D6+D7) = 13%+6%+3%+2%+1% = 25%, Day 1-3 range retention = 88%

N-day retention is the most widely used method and hence called classic retention. Classic retention can also be measured at a week, month, or year level.

N-week classic retention: W1, W2, W3, and so on. On a weekly level, we look at % of new users in Week 0 (1-7 days) who came back in Week 1 (8-14 days), Week 2 (15-21 days), etc.

N-month classic retention: M1, M2, M3, and so on. On a monthly level, we look at % of new users in Month 0 (0-30 days) who came back in Month 1 (31-60 days), Month 2 (61-90 days), etc.

N-year classic retention: Y1, Y2, Y3, and so on. On an annual level, we look at % of new users in Year 0 who came back in Year 1, Year 2, etc.

We mentioned at the beginning that different teams measure retention in different ways. How should you measure your app retention?

How should I measure my app retention?

You need to define 3 things before finalizing the retention method.

1. Natural usage interval of your product

2. The key event of your product — this will map to the Aha! moment

3. Business model (CAC-LTV)

Let’s take two different products and understand how these 3 things matter — Facebook.com and Booking.com

For Facebook.com

1. Natural usage interval is multiple times a day

2. The key event is scrolling through the feed, which leads to user engagement and ad revenue.

3. The business model is revenue through ads

For Booking.com

1. Natural usage interval is once a month or quarter as we don’t travel every day

2. The key event is doing a transaction, i.e. booking a flight or hotel

3. The business model is commission per transaction

For facebook.com, it makes sense to look at daily classic retention (D1, D7, D30, D90, D180) as it will expect users to come back again and again.

For booking.com, a monthly or quarterly classic retention makes sense. You can also look at classic monthly or quarterly retention (M1, M2, or Q1, Q2) as most people travel once every 3-6 months.

It is very important to note that you should merge data across platforms while looking at retention (website, app, m-site).If you don't merge data, the calculation will go off and the insights will probably be wrong. 

Oh, but why do we need a key event? It may happen that a lot of users downloaded your app, but didn’t signup and saw the feed/ made a transaction. These users didn’t reach their aha! moment and hence should be treated separately in retention analysis. This cohort has an activation problem, and not a retention problem.

The retention should be measured for users who performed a key event. They are your activated users. You can keep a tab for all new users’ retention as well, but need to worry if activated users’ retention is bad. This means that even after users experienced your product’s value, they aren’t returning. That’s bad.

What about the business model? For Facebook and booking, the business model aligns with the usage intervals of the product. High usage of Facebook along with a high # of users leads to a high # of ads and significant revenue. Low usage but high ticket size and commission of booking is good enough to recover CAC and other costs.

Why you can still fail with high retention? Usage <> Business Model Fit

Sometimes there is a mismatch between usage and business model, which raises questions around whether the product will be successful or not in the long run. An example includes ad-based OTT platforms. OTT platforms invest significant money in creating content. For example, Netflix invests > $10 B dollars in creating content. Creating content like movies and TV shows requires significant investment. Further, storing and delivering video content is also costly compared to text/images. So despite heavy usage, an OTT platform will fail if it relies on ads for revenue only (AVOD - advertising video on demand). This is the reason why Netflix has a subscription model (SVOD). Disney Hotstar and Hulu offer both AVOD and SVOD but are increasingly focussing on subscription.

Youtube still works fine with the AVOD model as it doesn’t invest anything in creating content and relies on individual creators to make videos. All it needs is to recover storage, delivery, and platform development costs.

Why you can still succeed with low retention? Usage <> Business Model Fit

Platforms with power-law distribution can succeed despite low retention as only a few users can lead to high returns.

Dan Hockenmeir wrote about this case a while back. These platforms need to have

- Low or zero acquisition costs

- Low or zero marginal costs in serving to these users

- Exponential returns to scale for few customers that do retain

An example of this is Shopify. With zero or low acquisition and operational cost, these platforms like Shopify can attract and serve a high # of users at a low spend.

Even if few merchants on Shopify succeed and make big business, it will create a high return for Shopify as for every transaction, Shopify takes a small percentage from the overall cost of the sales. Here again, the usage <> business model fit is there. Ben Thompson spoke around this on @patrick_oshag's podcast: “The more companies that use Shopify and fail is a positive indicator for Shopify because that means they are getting more rolls at the dice at that one thing that hits it big. In a world of abundance, it’s all about increasing the rolls of the dice. It’s not about your failure rate, because failure is cheap - it’s often zero."

So now that we have understood how to define and measure retention, we can move on to why it matters. This will be a recap of what I have already written in earlier posts.

Why Retention Matters

Retention matters because of two reasons. First, it is a true indicator of product-market fit. As I wrote back in the piece Measuring Product-Market Fit

Here is a comparison of two products acquiring 3 million users/month for 12 months. For the sake of simplicity, we have taken monthly retention rate as constant. In reality, it keeps reducing over months.

Two products with same # of registered numbers with different retention rates. As you can see, retention rate can create huge gap between companies’ monthly active user (MAU) base. This gap will only become bigger over time, making the first one a huge success while the other one a cautionary tale. This is why, retention along with healthy acquisition is the leading indicator of product-market fit and one of the most important metric to track for product success.

The second reason why it matters is that it is not a vanity metric. Getting users by spending money through marketing is easy. Retaining them requires a good value in the product. So good retention is also a good signal of value creation. In absence of value creation, you can’t charge money and build a sustainable business over time.

But what do we call good retention?

What’s a Good Retention?

Retention varies across categories of the apps because of multiple reasons.

1. We have already discussed that the natural usage interval of different product categories is different. So products with different usage intervals should not be compared.

2. Further, the business models of products in the same category can be different. For example, AVOD vs SVOD in video streaming will work with different retention measures. Whereas daily retention is important in AVOD, monthly subscriber retention is important in SVOD.

3. Another factor is geography. Users across different geographies (India vs the USA) behave differently. We should keep this in mind while benchmarking.

I will emphasize that don’t compare retention rates unless the products fall in the same usage pattern, category, and business model. What I would emphasize is looking at whether your retention curve flattens over time. As I wrote in Frameworks for Building a Successful Product,

For a product to grow, the retention curve should flatten at some point just like the blue line curve in the picture below. In absence of curve flattening, all the user you gained will get out of your system at some point. As you can see in the red line, you would lose all users you acquired today in next 10 days. The product displaying the red line behavior doesn’t have product-market fit.

We will discuss more about this in the next post. Let’s now look at how to benchmark retention for your app.

How to Benchmark Your Retention

To benchmark your app/website, you should start with local competitors if possible. In order to get the competition numbers, you can use App Annie or Similarweb. App Annie is usually more reliable about app numbers, whereas Similarweb is more reliable for web numbers. The sample graph below is from App Annie.

If you are looking for global benchmarks, Lenny Rachisky along with Casey Winters wrote an excellent post about it taking inputs from growth experts across companies. This is a snapshot from his post. Retention at 6 months means classic Month-6 (M6) retention, i.e. % of new users in M0 who came back in M6.

Try to benchmark both locally and globally. Usually, in order to succeed, you need to beat local benchmarks and aspire for global benchmarks. But don’t let yourself down if you can’t beat, say Facebook.com benchmark and you are a social app.

Reasons for Low Retention

If you have low retention, it’s hard to tell some silver bullets which will work in this case. It is usually hard to figure out the reasons for your low retention.

You should remember that retention is one of the hardest metrics to move and a true test of product management and growth skills. This post would help you define how to look for retention for your product.

In the next post, we will discuss how to figure out the reasons for low retention. DM me on Twitter if you have any question.

Till then, Goodbye and have a good Sunday ✌️😊

Thanks,

Deepak

For the new ones here, do check out the other posts that I have written if you haven’t

1. Measuring Product-Market Fit

2. Understanding Positioning: Through a Practical Lens

3. How to Engage Users Through Gamification

4. Referrals: The Holy Grail of Growth

5. Decoding Growth: Where is the Growth Coming From?

3,000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

Off to the topic,

This week, we will explore how to build the right onboarding experience. But first, what holds us back from building the right onboarding experience?

Definition of Onboarding! A lot of PMs and Designers I have met define onboarding as a tool to educate users about different features of the product. This is the reason they use the Onboarding slider telling users what to do. Here is an example from google translate where it shows the instructions around how to use the app.

Few others have developed their fondness for tooltips

It would be wrong to say the tooltips and sliders don’t help. They are generic tools of Onboarding used by many products out there. But like many generic tools, they make a tiny difference compared to what the right onboarding experience can do.

Let’s extend the definition of Onboarding to “getting users to their Aha! moments”. Before we start, this is a long post and might be truncated in email. Go to the deepaksingh.substack.com to read the whole of it.

Working Backwards

Amazon has this famous ‘working backwards’ approach to building products. Ian McAllister, a longtime Amazonian, wrote about it on Quora

There is an approach called "working backwards" that is widely used at Amazon. We try to work backwards from the customer, rather than starting with an idea for a product and trying to bolt customers onto it. While working backwards can be applied to any specific product decision, using this approach is especially important when developing new products or features.

For new initiatives a product manager typically starts by writing an internal press release announcing the finished product. The target audience for the press release is the new/updated product's customers, which can be retail customers or internal users of a tool or technology. Internal press releases are centered around the customer problem, how current solutions (internal or external) fail, and how the new product will blow away existing solutions.

If the benefits listed don't sound very interesting or exciting to customers, then perhaps they're not (and shouldn't be built). Instead, the product manager should keep iterating on the press release until they've come up with benefits that actually sound like benefits. Iterating on a press release is a lot less expensive than iterating on the product itself (and quicker!).

We can adopt the same approach for building the Onboarding experience. The idea is to start with defining the Aha! moments of the product and build it back from those Aha! moments.

The first step to this is identifying the Aha! moments for your product.

Identifying Aha! Moments

Every product has a different Aha! moment(s). To identify these moments for the product, let’s start with the end-users.

A consumer product can have millions of users. How do we think about what millions of users want? We start by categorizing these users basis a few key characteristics. In the world of product and design, these categories are called user personas. A persona is a collective image of a group of customers who have similar behavior or characteristics.

This is how a user persona looks like

We have been taught that every human being is unique. What’s also true is we are similar in a lot of ways. A user persona exercise tries to find those similarity patterns and groups the users.

How many personas should be there? After all, there are billions of people on this planet and you can come up with hundreds and thousands of personas if you want. We should create 3-8 user personas for the product as they often cover most of the end-users for the product. More than 8 is overkill.

Once we have these personas, it’s easier to identify Aha! moments for each of them. A single ‘Aha! Moment’ can apply to most or all of the personas as well. An example of this is finding 7-10 friends when you join a social network like Facebook. It will be hard to find a person who doesn’t like that. This is why People You May Know was such a huge success for Facebook as we discussed in the last post.

Step 1: Identify 3-8 user personas for your product

Step 2: List down their Aha! moments when they will use your product.

If you can’t come up with Aha! moments in your product, you most likely don’t know the consumers well. You will have to talk to these potential users and figure out whether they need the product you are building or not. Most of the successful products have very clear aha! moments as you can see.

Assuming we have the personas and Aha! moments, how do we design the onboarding experience? If you aren’t a designer, it will bee hard to design one for yourself. What I am going to give you is a good list of elements of Onboarding. Treat it like a mental checklist that you have to think about in discussions around Onboarding.

Tools of Onboarding

There are multiple tools of Onboarding. We will cover 5 key tools of Onboarding. These tools should be used to take users/customers to their Aha! moments.

1/5 — Use the AIDA Model to Explain Benefits

AIDA model is a concept taught in marketing classes. When a user opens your app or website, they follow the AIDA journey as shown by the diagram

Attention — Grab their attention

Interest — Show them something interesting

Desire — Once they engage, show them something they want/need

Action — Give them a call to action and a reason to act on it

Normally, the AIDA model describes the steps a customer goes through in the process of purchasing a product. But it can be applied to all products. Almost all the good landing pages are created following the AIDA model. Though one thing to remember is that it doesn’t happen one by one in Internet products, but all at once. Let’s take the landing page of the Airbnb host page.

A — The picture on the top of the page gets you curious. To grab attention, all the landing pages have pictures.

A/I/D — The top fold of the page gets you interested in a clear callout “Earn money as an Airbnb host”. Earning money is a desire almost everyone in the universe has.

A — The action here is “get started” by telling a little about your place.

You can see AIDA on all the good landing pages. A better landing page tries to give multiple reasons to invoke interest, desire, and action. If you scroll down the Airbnb landing page, you will find a different CTA every fold and benefits listed clearly to increase interest and desire.

Airbnb providing assurance

CTA to learn how they host, multiple benefits

You get the idea!

A bad landing page will look something like this. This was the landing page of Cruise.com a while back.

A lot of info with no clear action. Fortunately, they took note and improved it

So remember to include elements of AIDA next time you design your landing page or promotions page. Please note that the benefits should be explained before the signup/login happens. What do we do once we have convinced the user about the benefits?

2/5 — Progressive

A lot of apps want to explain the function of the app so that users can use it. These apps either fall into a new category or are complex with many features. Most of them solve this problem through sliders.

This approach has disadvantages. One, the users don’t want to read a lot of information, and two, they can’t remember it all.

A good onboarding experience is progressive. In general, people learn best by doing something. Progressive onboarding is presenting information gradually to users as they use the app. It avoids information overload for users and is very effective.

If you have to learn about progressive onboarding, there is nothing better than video games. Video games get complex over time. If someone opens a video game they have been playing for the last few months and shows to a friend who hasn’t played it once, the friend will find it hard to understand the game.

Video games break the complexity and teach them one by one to the users. Let’s take Cut the Rope as an example. The game is pretty popular with 100 mn+ downloads. It gives you an option to swipe and cut the rope just like in the game. They could have opted for screenshots to tell you this, but learning by doing is more delightful and effective.

Over time, they give you an option to toggle the gravity. They will present this option when you have mastered the first few levels. This is an example of progressive onboarding.

Another thing to remember while building progressive onboarding is to explain the benefit of the tasks.

In apps, progressive onboarding is especially useful when pointing out hidden functionalities that only power users would want to use. So if you are building a complex app with too many features, don’t despair. Just do progressive onboarding.

If the app isn’t that complex, say Youtube where the user just has to play a video, you don’t need progressive onboarding.

3/5 — Personalised

Creating personalized onboarding, you can reduce the drop-offs in Onboarding. The personalization can happen by asking for explicit info from the user and customize onboarding.

Suppose you are building an app in India where people speak 22 different languages across India. If your app is available in all languages, it is better to ask the user upfront which language they want to use when they open the app.

You can provide the rest of the experience in the language selected by the user.

4/5 — Managing the psyche of the user

Psych framework written by Darius Contractor is one of the best thing I have read in Onboarding. Every UX interaction on the page increases or decreases Psych. The higher the psych, the more the motivation.

Once you understand which things increase or decrease motivation of the user, you can design your page to optimise for Psych.

Putting pricing between user testmonials and benefits is one way to optimise Psych. Once you start thinking like it, it becomes easy to understand and eliminate friction points.

Always remove friction, as much as possible!

5/5 — Behavioral tools

Humans are social animals. We rely a lot of authority figures and people around us when it comes to making decisions.

It is important to add social proof and authority to build trust with users.

Social proof — consumers are heavily influenced by the actions and recommendations of others. Ratings, reviews, # of paid customers etc. are some of the examples of social proof. Here is the social proof I have used on the home page of this newsletter, mentioning 3000+ subscribers

Authority — When someone in authority like the givernment, celebrities, newspapers, CEOs speak about something, we tend to trust them more. Examples of authority can be seen in testimonials on books, yelp ratings in a restaurant and also on websites. Here is a review section of the book ‘The Psychology of Money’

Scarcity — Scarcity is telling people ‘only few left in stock’ to create a sense of urgency. Amazon does this well

Research

You can design a good onboarding experience through identifying user personas, their Aha! moments and tools.

What you can do beyonds this is understanding UI design patterns of competitor apps by going through them.

Some common UI design patterns for Onboarding are — welcome messages, tooltips, progress bars, checklists, hotspots, action-driven tooltips, videos, slider. Here is a good resource to learn about these elements if you want to.

Researching apps in your domain can often give ideas you can’t imagine sitting and thinking :)

Re-engage

The last step of Onboarding is giving users reasons to re-engage. There is a good chance that even after doing everything right, few users wouldn’t find their Aha! moments in the product.

You can re-engage them by building things of value. Let me give an example. Suppose you went to intercom.com to check for a customer support chat solution. You couldn’t make the decision then, but found good resources for learning product marketing.

You keep coming to it once every month to read the articles. One day, you went ahead and checked few more things about intercom and found something useful in their product you hadn’t noticed the first time.

This is the value of re-engagement. It gives your product multiple chances to re-engage users and introduce aha! moment.

Sustainable ways to re-engage are

1. Content — users keep coming for reading and researching

2. SEO — you keep appearing in google searches of the users

3. Social — they keep seeing your posts on social media

To sum up, you can build an amazing onboarding experience by

1. Identifying 3-8 user personas for your product

2. Mapping Aha! moments for these personas

3. Using the 5 tools of onboarding — AIDA, Progressive, Personalised, Psych, Behavioral

4. Researching apps in your domain

5. Building ways to re-engage users

Now you may have some questions around onboarding, listing down the useful ones. Please feel free to comments any other questions you may have.

Q: Do products succeed despite terrible onboarding?

A: Yes they do. This happens especially when there is lack of competition and the pain of figuring out the product is lesser than the benefit of using the product. A prime example of this is the IRCTC website in India. Despite horrible experience, it is the only way to book train tickets in India. So people use it.

Very simple products with a single functionality also don’t need onboarding as users downloading this app already know the prime utility they offer. Example — Netflix, Youtube, Whatsapp, Google search

Q: Can I bring the ‘Aha! Moment’ through slider onboarding?

A: You can. If you have a very strong value proposition and the app is simple to use, you can bring the ‘Aha! Moment’ through slider onboarding

Q: Can I use the data and machine intelligence (AI/ML) to personalize onboarding

A : Mostly you can’t. You will have a cold start problem as you don’t have any data from the user.

Q: What’s the difference between a landing page and onboarding?

A : A landing page is the advertisement of your product. Onboarding is the user manual

Q: When does the onboarding stop?

A: Never

Q: Can we use ads and emails as an onboarding tool?

A: Ads are an expensive way to educate users about the product value proposition. But it works when it comes to expensive SaaS products as you can afford to show higher frequency of ads to the end users.

With this, I will wrap this already-long post. See you next week.

Happy Sunday ✌️

Haven’t subscribed yet? 😯 Consider subscribing 👇

Subscribe now

Sincerely,

Deepak

Last week, we discussed how to build Flywheels — the unstoppable force of growth. It’s a very important topic on product growth and if you haven’t read yet, I will advise you to read it.

This week’s topic is Onboarding. It is one of the lowest hanging fruit in achieving your growth targets and building a great product.

For the new ones here, do check out the other posts that I have written if you haven’t

1. Measuring Product-Market Fit

2. Understanding Positioning: Through a Practical Lens

3. How to Engage Users Through Gamification

4. Referrals: The Holy Grail of Growth

5. Decoding Growth: Where is the Growth Coming From?

3,000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

Off to the topic,

Imagine a mall with no human assistance. How would you design it such that a person coming for the first time there can find and buy what they want?

Imagine a clothing store with no human assistance and no tryout rooms. How would you convince someone new to buy clothes from there?

Imagine a park with lots of people. How would you design it in such that a person coming for the first time there can find my friends or other interesting people so that they can enjoy their time in the park?

All of these design problems have a commonality — how do you make the first-time users find the value in the situation quickly. In a park, the value lies in finding friends and family. In a mall or store, it’s finding the right item they are looking for quickly.

All the above happens today on the Internet. The 1st problem is solved by Amazon, Flipkart, and other eCommerce companies. The 2nd problem is solved by specialized eCommerce companies like Myntra and Shein. Facebook and other social media companies solve ‘the park problem’.

First Impressions Matter

We have been taught that "First Impressions Matter." In real life, it's often true but we all have one or more of those friends who had their first impressions wrong. But we met them again, understood them, and became friends.

When it comes to Internet products, the quote about first impressions is more accurate than ever. A store with no person to explain or correct first impressions — its a disaster. The user is never coming back in most of the cases.

Users onboarding is introducing your users to your product and making first impressions. It doesn't matter how strong your product is; if the Onboarding isn't great, and users fall off, people will never get to experience it.

Looking at some stats from the Hubspot blog,

1. 55% of people say they've returned a product because they didn't understand how to use it.

2. Acquiring new customers is between 5 and 25 times more expensive than retaining existing ones.

3. 74% of people will revisit your website if it's user-friendly, and 50% of customers will stop visiting it if it's not.

4. Over 90% of customers think that companies "could do better" when Onboarding new customers.

5. 59% of customers value personalization over speed when it comes to customer service.

6. 97% of people believe video is a useful tool to welcome and educate new customers.

7. 74% of people are likely to switch brands if they find the purchasing process too difficult.

8. 80% of people have deleted an app because they didn't know how to use it.

In the AARRR funnel, Onboarding ensures the activation of users. You won't get any retention, revenue, or referral if users fall at the activation stage in the absence of proper Onboarding.

Hopefully, I have convinced you of the merits of Onboarding. But since just knowing the merits won't help you, let's dig deeper.

The Spectrum of Onboarding

When it comes to Onboarding, the whole spectrum can be seen in terms of ARPU, i.e., average revenue per user. The information and assurance needed to buy the product are proportional to the money you ask the users to shell out.

Low ARPU products are often free to consumers, and the money is made through advertising. For the ad-based products to work, they have to maintain very high retention and engagement and address the mass market. They can't afford paid marketing, and which is why you see products with strong network effects and user-generated content like social media, classifieds, etc. here. Onboarding of such products is built within the product with no human assistance to the user.

Medium ARPU is an exciting category. Most products here are need/utility-based. You go to Amazon to buy items you need and go to Netflix to watch some movies available only there and not free of cost anywhere else. Onboarding of such products needs to be built within the product, but they do provide free trials. They also provide remote support through emails or calls, but the support emails or calls can't be considered part of Onboarding as they happen post-purchase.

High ARPU products like Salesforce or Hubspot offer a free demo or free trial. It is to ensure they get enough time to spend with the users and give them the information and the assurance needed to make the purchase. This is also why the products with high ARPU have an inside/ field sales team for Onboarding. Having a demo/free trial and sales team increases the CAC of the product, but they can justify the high CAC through the high ARPU. Most of these products are B2B SaaS as no one person can afford to pay so much.

The diagram above can now be represented as

The Park with Lots of People

We started this post with 3 design problems. Facebook is a park with lots of people. They wanted new users to find friends and start using Facebook. Let me tell you the story of an onboarding feature that Facebook built that helped its users find friends and interesting people. From the book ‘Facebook: The Inside Story’ by Steven Levy,

The masterpiece of Growth at Facebook is a feature that became almost as much a part of News Feed as weddings, vacations, and political outrage. It’s called People You May Know, referred to internally by the acronym PYMK. Officially launched in 2008, People You May Know is a feature that identifies personally selected prospects for one’s friend list. It wasn’t a Facebook invention — LinkedIn did it first — but PYMK proved to be one of FB Growth team’s most effective tools, and also one of its most controversial ones, a symbol of how the dark art of growth hacking can lead to unexpected consequences.

On its face PYMK seems innocuous enough: a carousel of profile pictures on Facebook presumably connected to you, but somehow not your Facebook friends. Its impetus was to address an imperative that the Growth team’s researchers had unearthed: a new Facebook user is likely to abandon the service if he or she doesn’t connect with seven new friends — fast.

Thus PYMK was essential for Facebook. Exposing potential friends is a way to improve a member’s experience; it increases the chances they will share more, and, most of all, it makes people less likely to bail on Facebook.

People You May Know caused a lot of controversies around privacy issues. But it did solve the core deliverable of onboarding at Facebook — finding friends and activating users.

The Core of Onboarding

The core deliverable of Onboarding is also popularly known as the 'aha!' moment. It's pretty popular in growth circles, and I am not sure who invented the term. It's the moment when the user suddenly grasps the true value of your product and has an emotional reaction to it. For Facebook, it was finding friends whom you could chat and show them your social life instantly.

Onboarding is so crucial for growth that a growth team's most critical job anywhere is to make more and more users experience the 'aha!' moment of the product. So if you know your product's 'aha!' moments, they could form the basis of your activation/onboarding flow design.

I have listed below some popular products and their aha! moments here for you to understand the point well

A Mall with No Human Assistance

Amazon is a mall with no human assistance. So are other horizontal e-Commerce companies. If you go to Amazon, you can quickly see what strategies they employ for Onboarding.

A big visible search bar — usually, if you are going to a mall, you know what to buy. For this reason, Amazon gives you a big visible search bar.

Autosuggestions — When you try searching for a product, it offers you auto-suggestions. Auto-suggestion does 3 things — first, they show you a variety of products upfront matching the words you type. Second, they help you avoid typo on the go. Third, they make your life convenient so that you don't have to type the whole thing. Auto-suggestion reduces friction for users while searching.

Deals and best products in categories — If the user decides to scroll down, the best products and deals are displayed. This is like putting the best products on display when the user enters the mall.

Product descriptions, filters, and reviews — Filters help users in narrowing down to what they want. Descriptions help them understand if the product meets their requirement, and reviews offer social proof that they can trust and buy.

There is the replacement, warranty, return, cash on delivery, photos, etc. on the product page. All of these are aimed at proving the user assurance that they can spend their hard-earned money.

So we have answered how you will design a mall without human assistance where people can buy the product.

1. Make it easy to find through search and navigation.

2. Provide all the details about the products

3. Assure the user by adding free-of-cost replacement, warranty, return, etc. Add reviews from other people who have bought and used the product.

These are the part of new user experience (NUX) or Onboarding for a complicated thing like shopping.

It isn't so complicated for other products, but you should give it a thought on how you are taking people to their 'aha!' moments.

A Clothing Store with No Human and No Trial Room

Surprisingly, whatever works for the mall also works for this clothing store as it is a subset of the mall. While designing this, you have the luxury of creating a more convenient NUX/onboarding. Examples include Myntra, Shein, etc.

Off to the questions

Q: You just described the whole Amazon product buying experience in its Onboarding. Isn't that a lot?

A: The jurisdiction of Onboarding depends on where your Aha! Moments lie. For Amazon, it is finding and buying a product you were looking for and getting happy with the experience. I would even go on to include 1st order post-purchase experience in Onboarding at Amazon. Because unless someone gets happy after receiving the first order, the job isn't done.

Q: What's the difference between NUX and Onboarding?

A: NUX and Onboarding are the same things, in my opinion.

Q: How do we design the Onboarding of our product?

A: I will cover it next week

Q: What are some of the good examples of Onboarding?

A: I will cover it next week

Gracias for reading! Please share this post with people who may find it useful and help build a strong growing community.

Adios! Have a good Sunday ✌️

Sincerely,

Deepak

PS: I have started writing and sharing my thoughts more frequently on Twitter. Join me on Twitter, my DMs are open so feel free to drop a message.

Welcome to the 253 new Growth Catalyst subscribers who have joined in the last 1 week! If you’re reading this but haven’t subscribed, join 2,800+ smart, curious folks by subscribing here!

Subscribe now

If you are new here, here are the top 5 posts of this newsletters so far

1. Measuring Product-Market Fit

2. Understanding Positioning: Through a Practical Lens

3. How to Engage Users Through Gamification

4. Referrals: The Holy Grail of Growth

5. Decoding Growth: Where is the Growth Coming From?

Off to the topic,

Last week, we discussed growth loops and where to find them. The post generated good discussions, and I am grateful for folks providing inputs and making it better. Please keep them coming :)

This week I am going to touch a topic often confused with growth loops — Flywheels. The reason for this confusion is simple; they look and feel the same. But as it happens with frameworks, the real value lies in nuances of these frameworks. So if you want to apply the Flywheel concept, it’s better to understand it well.

Jim Collins even wrote a book about this, Turning The Flywheel. Below is an excerpt from his book about Flywheel where he explains the Flywheels concept.

Picture a huge, heavy flywheel—a massive metal disk mounted horizontally on an axle, about 30 feet in diameter, 2 feet thick, and weighing about 5,000 pounds. Now imagine that your task is to get the flywheel rotating on the axle as fast and long as possible. 

Pushing with great effort, you get the flywheel to inch forward, moving almost imperceptibly at first. You keep pushing and, after two or three hours of persistent effort, you get the flywheel to complete one entire turn.

You keep pushing, and the flywheel begins to move a bit faster, and with continued great effort, you move it around a second rotation. You keep pushing in a consistent direction.  Three turns ... four ... five ... six ... the flywheel builds up speed ... seven ... eight ... you keep pushing ... nine ... ten ... it builds momentum ... eleven ... twelve ... moving faster with each turn … twenty … thirty … fifty … a hundred.

Then, at some point—breakthrough!  The momentum of the thing kicks in your favor, hurling the flywheel forward, turn after turn ... whoosh! ... its own heavy weight working for you. You're pushing no harder than during the first rotation, but the flywheel goes faster and faster.  Each turn of the flywheel builds upon work done earlier, compounding your investment of effort. A thousand times faster, then ten thousand, then a hundred thousand. The huge heavy disk flies forward, with almost unstoppable momentum.

Remember the words ‘unstoppable momentum’ when you think of Flywheels.

Limitation of Loops and Birth of Flywheels

‘Yo’ will forever be remembered as the app that was too simple. It lets users do exactly one thing: say "yo" to each other.

Despite its simpler premise, the app had strong growth. The app quietly launched in April of 2014 and grew. In July, the app got $1.5 million in funding from investors. By September, the app had 2.7 million registered users and 1.2 million monthly active users.

Fast forward today, the app still lives. But ‘Yo’ is asking users to donate to its Patreon to keep the app alive. From the Play Store description of the app

Does ‘Yo’ have a growth loop? It does. Here is its growth loop.

A growth loop brings new users to the platform as it did for the ‘Yo’ app. But it does nothing about retaining them. So even if a product has growth loops, it may or may not have sustainable growth.

This is why we needed a new tool to think about sustainable growth — Flywheels.

Flywheels

The term comes from physics — a flywheel is a mechanical device specifically designed to efficiently store rotational (kinetic) energy. The key idea here is that Flywheels resist changes in rotational speed (ω) and momentum by their moment of inertia (I), meaning once they are in motion, they are difficult to slow. Here is a 1- min short video explainer on this.

In the startup world, this implies flywheel-like products gain enough momentum and it becomes their moat over time. It’s hard to beat/slow their flywheel of growth.

Back in 2014, when Uber was valued at tens of billions of dollars, there was a lot of criticism from valuation experts. Famous investor Ashwath Damodaran wrote

Earlier this month, investors poured $1.2 billion into Uber, a tech company whose smartphone app connects taxi drivers to passengers. The share of the business these investors received suggests that Uber is worth $17 billion, a mind-boggling sum for a young company with only a few hundred million dollars in revenue.

An equally compelling response came from one of Uber’s earliest investors, Bill Gurley. This tweet exchange between David Sacks and Bill sums his response.

This tweet explains why Investors got bullish on Uber. I will take a minute to elaborate on the drawing that David Sacks shared.

1. As more users join Uber, more drivers will come looking for them.

2. The geographic density of Uber increases, i.e. more uber drivers and riders in an area. More driver density would mean faster pickup —> even more users will start coming for faster pickups

3. More users will lead to more # of rides for a driver. More rides —> better income. So they can afford to lower the prices.

All of it becomes a virtuous growth cycle. Lower prices and low waiting times will keep users on the platform. As the users are on Uber, drivers will also be on the platform. The lockin will keep Uber’s flywheel in motion over a long period of time. Long enough to weed out any competition, and build a sustainable company.

Today Uber is valued at $65 B in public markets. Whatever one’s feeling is about Uber's valuation, it definitely changed the world. Flywheels often do that as we will see next.

Superimposition of Loops = Flywheel Effect

In the last post about growth loops, we discussed the superimposition of loops. Flywheels often have multiple loops/wheels and can be seen as a superimposition of loops.

Let’s take the best Flywheel built to date to understand this — Amazon. An excerpt from the Amazon website which explains how Amazon grows.

Amazon is built on the concept of a virtuous cycle focused on the customer. The idea was constructed on a napkin by CEO Jeff Bezos, and still remains a living, breathing part of Amazon..

Jeff Wilke explains the meaning of the virtuous cycle in this video.

To break various loops/wheels in motion

1st loop is around customers and seller growth

Customer Experience (CX) —> More Customers —> More Sellers —> More Selection —> CX

2nd loop is around prices and cost structure

More Customers —> Lower Cost Structure —> Lower Prices —> Better CX —> More Customers

3rd important loop not listed above is around delivery time :

More Customers —> Better Logistic Network Utilisation —> Lower Delivery cost +Faster Delivery —> More Customers

The flywheel also has multiple actors like sellers and buyers dependent on each other and creating lockin with each other. And it’s not just Amazon. Best marketplace models are often flywheels with multiple actors. Examples - Uber (riders and drivers), AirBnB (host and guest), AliExpress (buyer and seller), Swiggy (restaurant, buyers), UrbanClap (service providers, buyers), Angellist (investors, startups), etc.

Key to Building Flywheels

Flywheels are all about building advantages. A good list of advantages always comes handy when thinking about Flywheels. Here are the 6 most common advantages -

1. Direct Network Effect examples are telephone networks, the Internet, Facebook, WhatsApp, etc.

2. Economies of Scale can be seen in many software companies. This is because most of their costs are fixed. For example, Google costs come mainly from servers and software development, and these do not increase with the number of users. Every additional search lowers the average cost of a search. Examples of companies profiting extensively from Economies of Scale are Google, Facebook, Salesforce, Amazon, Netflix, etc.

3. 2-sided Network Effect can be seen in marketplaces like Uber, Airbnb, AliExpress, Amazon, Swiggy, UrbanClap, Opentable, Angellist, etc. Recently, it has also happened on social network apps like Instagram with the advent of micro-influencers. Users and micro-influencers create 2-sided network effects.

4. Switching Costs examples include SaaS companies like Salesforce, Intuit, Microsoft, etc. These are mostly unique to products where you can use only one of them at a time. Though B2C companies build switching costs by creating loyalty programs. Amazon has done a fabulous job with Prime, which has an annual churn of 9% only.

5. A brand habit example is Apple. Other companies with strong brand habits are Disney, Tesla, and Nike.

6. Proprietary tech in software companies is mostly in the form of data-based AI/ML algorithms. Google’s search, Netflix’s recommendations, Amazon product search and recommendations, Alexa Skills, etc. are some of the examples. In absence of big data, it’s hard to build this advantage. Though hardware products like Apple and Samsung also rely on hardware or design patents to build an advantage.

You may have noticed a few names appearing multiple times. This is because of the more the advantages the better the flywheel. Let’s see a few examples

1. Uber checks 3 of 6 advantages — 2-sided network, brand habit, proprietary tech like surge pricing

2. Amazon checks 5 of 6 advantages — 2-sided network, economies of scale, switching cost through prime, brand habit, proprietary tech like the recommendation and product search, AWS, etc.

3. Youtube checks 5 of 6 advantages — 2-sided network, economies of scale in CDN costs, switching cost for creators as one has to move subscriber base, brand habit, proprietary tech like a recommendation, and product search.

4. Apple checks all 6 of 6 advantages — 2 sided-network of app developers and iPhone users, economies of scale in manufacturing, direct network effect to some extent, high switching cost due to familiarity of the interface, strong brand habit, proprietary tech on hardware and design.

5. Tripadvisor checks 3 of 6 advantages — 2 sided-network of users and travel commodities, economies of scale high as the cost is limited to software development and the content is user generated increasing the marginal ROI of TripAdvisor manifold, strong brand presence as it keeps appearing on the first page of your search results when it comes to travel.

Differences: Loops and Flywheels

At the start of this chapter, we said that Loops and Flywheels are different. Now we have seen that flywheels are often a superimposition of growth loops. So how are flywheels and loops they different? Why do we need two different terms when thinking about the growth of the product?

To understand that, let’s look at the products which grew due to growth loops but fizzled out in lack of flywheel

1. We have already discussed the Yo app. Yo had 1 of 6 advantages — direct network effects. But a lot of users don’t find Yo useful, so the direct network effect is also limited. Is it a wonder that it failed?

2. HQ Trivia is another example. It was a popular live quiz show where you could earn a life by inviting friends. So it built a viral growth loop and grew to > 2 million DAUs. But because of the lack of any of the advantages (0 of 6), it fell fast and fell hard.

3. Vine, a popular video loop app that offered creators the ability to make and share six-second videos similar to TikTok, had 1 of 6 advantages — direct network effects. For a short period, it also had another advantage, i.e. 2-sided Network Effect. In 2016, Twitter announced that it was shutting down Vine. The app couldn't compete with the rising popularity of Instagram, and users who were once considered as Vine celebrities started posting videos on Instagram. Why did Tiktok succeed, whereas Vine failed?

Vine versus Tiktok

Failure to make money for Vine and Creators, Scale, UX — From TheVerge

Years of executive churn likely contributed to Vine’s failure to make money. For a while, brands were happy to pay Vine stars directly to make ads and share them to their millions of followers. But after Snapchat and Instagram grew into hundreds of millions of daily users, marketers’ interest in Vine dropped significantly. They had once longed for ways to grow their own followings on the app — through paid placement offerings similar to Twitter’s promoted tweets and promoted accounts.But Vine never came through with any options, in part because the founders resisted monetization from the start, sources said. It never took a cut of stars’ deals with brands

TikTok is the overseas version of Douyin by Bytedance.

There were about 400M users of Douyin in China. More importantly, China had a developed business ecosystem formed around KOLs (key opinion leaders) who made their livings online by creating short videos for brands, etc. TikTok allows creators to monetize their own content, and it's got more tools for creators to customize or collaborate on content.

The tools built by Tiktok for creators to make good content and collaborate are pretty good. This means delightful content for the end-users.

The scale allows Tiktok to built a much better content recommendation engine, which is a key strength of Bytedance. The scale of Tiktok also creates a 2-sided network effect.

All-in-all, Tiktok checks 5 of 6 advantages

1. The stronger 2-sided network effect between users and creators

2. Direct network effect built by users sharing these videos with each other

3. Economies of scale due to user-generated content (UGC)

4. Switching costs for creators in the form of losing money if they leave the platform

5. Proprietary technology like recommendation engine which will get better with more users and more usage

Loops vs Flywheels

Looking at these examples and what we discussed before, we can identify a few key differences in Loops and Flywheels:

1. Flywheels are superimposed loops/ multiple wheels interacting with each other. Loops can be seen as a subset of Flywheels.

2. Loops with shorter cycles can make a product viral and it grows very fast like HQ Trivia and Vine. But if it gains none of the advantages of Flywheels over time, it has a high chance of failure. A key difference between loops and flywheels is the cycle time. Flywheels have a longer cycle time because of the completion of a cycle is dependent on multiple actors in the system. As the old adage goes, “Anything worthwhile takes time to build” 😊

3. All Flywheels build sustainable businesses with high momentum. The momentum can serve as the moat for the business as it takes a long time to slow down. Few loops build a sustainable business and the ones that do have long cycles in themselves. UGC loops translating to SEO benefits have a long cycle, same is the case of paid marketing where the LTV exceedeeng CAC will have a long cycle.

4. The feedback cycles are also longer in Flywheels. Even if something breaks, it’s barely noticeable initially because of high momentum. So such products should always define leading indicators of business and growth like NPS, rating, etc. Lagging indicators like revenue and profits start showing signs of worry when it’s too late.

Please note that flywheels and loops do overlap. It’s hard to establish clear distinction at times. But that’s the beauty of frameworks and tools, you should have as many in your toolbox. So get a good grasp over both of these tools while crafting a growth strategy 🚀

So what’s stopping your product from creating a Flywheel? DMs open on both LinkedIn and Twitter for any discussions.

See you next week. We will start the topic of Onboarding and Retention from the next week.

Sincerely,

Deepak

Last week, we discussed how to build growth models and forecast user growth. It’s a very important topic on product growth and if you haven’t read yet, I will advise you to read it.

This week’s topic is growth loops. It is one of my favorite topics because it brings clarity to product and growth discussions.

For the new ones here, do check out the other posts that I have written if you haven’t

1. Referrals: The Holy Grail of Growth

2. Measuring Product-Market Fit

3. Decoding Growth - Where is Growth Coming From

4. How to Engage Users Through Gamification

5. Understanding Positioning Through a Practical Lens

2,500+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

Off to the topic,

When I first heard about growth loops, I thought it was another new jargon thrown around in PM/growth circles. I had wasted my time reading about another buzzword - growth hacking and didn’t want to do it again. So I ignored this buzzword for a while.

It was before Brian Balfour, the founder of Reforge, wrote about it. Brian is one of my favorite people in product and growth. He has brought together some of the best product/growth people in Silicon Valley and crafted a few good courses around product and growth. 

I was intrigued after reading his first post around growth loops. After that, I got to read about it on multiple forums. This post would be a synthesis of what I have learned from growth professionals like Brian and my own experience.

What are growth loops?

Let’s start by defining growth loops. Taking a cue from Kevin Kwok and Casey Winters’ definition of growth loops

A growth loop is a system that generates an output that can be reinvested as an input.

Think about a machine where you can deposit money in your account, and it gives you back a percentage of that money (cashback), say 20%. You can withdraw all the money from the bank later. So when you deposit $ 100 through the machine, it gives you back $ 20 as output. You deposit the $ 20 back into the machine, and it gives you 20% of it again, i.e., $ 4. The cycle continues. The whole thing can be represented as the growth loop, as shown in the diagram below.

Why are the growth loops important?

So with an input of Rs 100, you get a total deposit of Rs (100+20+4+0.8+0.16+0.032..) ~ Rs 125.

Growth loops follow a geometric progression. You can arrive at the total deposit using the formula for the sum of a geometric progression (GP). The sum of all terms of a GP can be calculated if you have ‘a’ (initial number) and ‘r’ (common ratio).

In the example above, a = $100 and r = 0.20, so if we continue the cycle infinite times (n —> infinity), we get the final sum = $100/(1-0.2) = $125

If we do a few scenario analyses on a and r, we will quickly understand the value of growth loops.

a= $100, r = 0.2 : We have already been through this. Final amount = $125

a= $100, r = 0.8 : The total sum through all loops can be $100/0.2 = $500.

a= $100, r > 1.0 : The total sum will keep increasing. It will know no bounds if we keep continuing the cycle.

How do we read this in the context of product growth?

‘a’ is the number of users you get from marketing. ‘r’ is the number of users the user coming from marketing channels gets you.

So if you got 100 users today from paid marketing and these 100 users got you another 10 users, your r = 10/100 = 0.1 and a=100. Most products grow by getting new users from paid marketing every day, and those users bringing in more users.

Different products operate with different r-values. We don’t have to know the exact r-value for popular products, but we can take an intelligent guess. The r-value for WhatsApp would be greater than Reddit as people invite their friends and families when they first sign up for WhatsApp, which isn’t the case of Reddit.

What else are we missing in growth loops? The duration of one loop which determines how fast you can grow in a given period of time.

Say the duration of one loop is 1 month. So with ‘a’=100 and ‘r’=0.5, the first loop will complete in Month 1 and give you 50 extra users. These 50 users will bring another 25 users in Month 1 and so on.

But if the duration of one loop is 1 day, the growth can be much faster.

The duration of one loop is in hours/days in products like WhatsApp. In contrast, it can be months in B2B or SaaS products where users have to use the product themselves before referring to friends. It is crucial to keep the duration of the loop in mind while designing loops and calculating the expected output.

You can see how amazing your product can be if you work on the growth loops through the examples above. Let's see what kind of growth loops exist.

Types of growth loops

Growth loops can be core to your product flow, or you can create them through incentives. We will call the growth loops core to your product flow as core/organic loops, and the loops created through incentives as inorganic loops.

Organic loops

Organic loops are built into the core product flow. The most popular ones are UGC loops.

UGC loops

User-generated content (UGC) brings more traffic and engagement to the products.

Wikipedia, Tripadvisor, Reddit, Pinterest, Zomato, Yelp. etc. constitute the first category of such products.

UGC loops in the platforms listed above are mostly created through google or other search platforms. Here is the growth loop of such products

The second category builds UGC loops by combining status/social proof with UGC.

UGC + Status —> Quora, Medium, Substack. These products get goodness from search engines as well as social share. The social network is absent on these products, and they get their traffic through other social networks.

The third category of UGC products is social networks. These are social products and have strong network effects. It would be unfair to not classify them as a category of their own as it’s a fairly large category. But since social networks ultimately rely on UGC, we are putting them here for the sake of discussion. Facebook, Snapchat, Sharechat, LinkedIn, Twitter, Instagram, WhatsApp fall into this category.

Do note that social networks rely on network effects and their r-value is very high. They usually don’t rely on paid marketing, but high r-value to add new users and get exponential growth.

Curated Content loops

Curated content loops can be created by providing high-shareable content to the end-users. Buzzfeed is an excellent example of this. Some of the content pieces of BuzzFeed have gone viral. End-users share these pieces on their social networks. From their press release

In the U.S., BuzzFeed.com had 64 million unique visitors in February 2020, up 29% YoY, and holding a strong lead over competitive websites like Vox (35M UV), PopSugar (25M UV), and Vice (24M UV).

News websites, especially the ones tracking celebrities and politics, also have curated content loops.

Word of Mouth loops

These are hard to detect as the users coming from it show up as organic or direct users. The only way to create such loops is by having a clear positioning in the mind of the users and build a share-worthy product :)

Inorganic loops

Referral loops

You can create incentivized growth loops using referrals. I wrote a detailed post on how to build referrals. From the post,

The reason referral works so well is because when someone we trust tells us about a product, we are more inclined to try it. It is so alluring because if the virtuous cycle continues, it keeps bringing more and more users to the product, resulting in explosive growth.

Paid Marketing loops

Believe it or not, if everything else fails, paid marketing can create growth loops for many products.

The pre-requisite for this is generating early break-even through the revenue from the user. Once your CAC <= Revenue (CM positive) for the user, you can reinvest the revenue generated into paid marketing.

Advantages of growth loops

Where can growth loops help? The acquisition of new users is a clear one. If we look at the UGC loops, the content created by the users is read by the new users. So UGC loops help in both acquisitions as well as engagement.

What about retention? Whenever the new content appears on a platform, we often go back to check it — thus ensuring the retention. This is relevant to both UGC and curated content platforms.

In the long run, because of accumulated content, SEO benefits, or user data, the growth loops also create a moat. So thinking and designing growth loops can give you benefit in lots of ways, be it acquisition, retention, engagement, or in the long run, a moat for your product.

Let me ask you a question — can a product have more than one growth loops? Superimposing loops can create stronger growth. Any examples?

Superimposing loops

Build multiple loops within a product often requires multiple personas to use the product.

One example is YouTube. Youtube primarily has two personas — superstar YouTubers and users like you and me who consume content.

There are two major loops on Youtube. The first one is the engagement loop created through personalization using data.

The second loop is for content creation by YouTubers as shown below

The content creation loop is superimposed with the engagement loop.

More creators —> more videos —> more users —> more data —> more engagement —> more creators.

Superimposing loops can create a strong growth for the product. In the last section, let’s bring all of our learnings together in designing the loops.

Bringing it all together

These are things to keep in mind while designing growth loops

1. Make your product or part of the product share-worthy — Photo storage apps have been there for long, first in the form of your offline albums and then hard drive folders. Instagram created the photo album app where you could share photos with your friends. This created a strong growth loop for them, giving them a billion-dollar exit.

2. Keep the sharing nudges visible at appropriate points — Triggers are essential for people to share. Look at the Sharechat design of its posts. Share is the primary CTA, and its ratio of like: share would be among the highest across platforms.

   

3. Keep a check on the duration of the loop, r-value — by tracking all the shareable links. Don’t worry about tracking word of mouth early on.

4. It takes some time and effort to reach a minimum threshold when the growth loop starts working. For example, Zomato achieved this threshold by first curating the restaurant menu, photos, and reviews. Over time, the growth loop started working once it hit threshold traffic.

5. Don’t assume it will keep working indefinitely. There is a ceiling to every growth channel. Over time, the user segments change as you move from early adopters to late majority. Because of this, something that was working earlier would stop working. You have to identify the change early on and keep modifying the product to create growth loops for different user segments.

6. If you are working for creators or businesses, keep your brand name upfront for users to notice and join your product. Examples being Drift, Weebly, Substack, Medium, etc.

7. Build long term growth loops that can serve as a moat for your product like personalization, SEO, Content, etc.

8. Try to create multiple growth loops aka superimposing loops in your product that can reinforce each other.

The Story of an Entrepreneur

You may have heard about Zillow, Glassdoor, and Expedia before. All of these companies have two things in common. First, all of these are consumer tech companies, each worth over a billion dollars — Expedia ($12B), Zillow ($19B), and Glassdoor (sold at $1.2B). What’s the other thing in common? They were all founded by Rich Barton.

Some may say that he got lucky. But he has created these companies by following a common playbook around growth loops. From Kevin Kwok’s essay on Rich Barton

The Rich Barton Playbook is building Data Content Loops to disintermediate incumbents and dominate Search. And then using this traction to own demand in their industries.

Or as he puts it “Power to the People”

Much of what Rich Barton pioneered has now become mainstream. SEO/search is well saturated, and the importance of owning demand has been popularized by Ben Thompson’s many essays on (Demand) Aggregation Theory. But the cornerstone of Rich Barton’s playbook, Data Content Loops, are still underappreciated and rarely used.

Owning demand gives companies a compounding advantage, but needs to be bootstrapped. When a company is just starting out, it not only doesn’t own demand, it has all the disadvantages of competing against others that do.

In order to grow their demand high enough to become a beneficial flywheel, Barton’s companies use a Data Content Loop to bootstrap their demand and create unique content and index an industry online (homes for Zillow, hotels and flights for Expedia, companies for Glassdoor).

• Expedia: Prices for flights and hotels that before you’d have to get from travel agent

• Zillow: Zestimate of what your house is likely worth that before you’d have to get from broker

• Glassdoor: Reviews from employees about what a company is like that before you’d have to get from a recruiter or the company itself

These Data Content Loops help the companies reach the scale where other loops like SEO, brand, and network effects can kick in.

Barton’s companies then use this content to own search for their market. This gives them a durable and strong source of free user acquisition, which enables them to own demand

Believe me yet on the power of growth loops?

As we conclude, let’s tackle some of the common questions we get around growth loops.

Q: How are loops different from flywheels?

A: At the first look, growth loops and flywheels look the same. The words are often used interchangeably. They are overlapping but different from each other. We will discuss flywheels in detail in one of the future posts.

Q: Do marketplaces like Uber, Amazon, and Airbnb have growth loops?

A: Marketplace businesses do have growth loops. But marketplaces operate more like flywheels, which we will discuss later.

Q: Which businesses don’t have growth loops?

A: A lot of offline businesses have none/limited growth loops (edited post suggestion by Anshul in the comments 🙏🏻). Brick and mortar shops and offline bookstores to name a few. The growth loops, even the word of mouth for offline businesses are limited by geography, which means a very limited # of cycles. Add to that the friction of accessing it, i.e. going to the place. Offline -> online -> offline builds loops through online channels like WhatsApp, so I wouldn't classify them in this category. Contrary to that, online businesses have high scalability when it comes to growth loops. So all online products try to build growth loops. Even paid newsletters do so by publishing a few free stories that can be shared around. Personal note-taking apps like readwise.io do it by asking users to save threads on Twitter by commenting, which everyone can see, rather than giving the extensions/plugins. Online stores do it through referrals or paid marketing loops. How effective these loops are an entirely different question to answer and depends on the business. Glad that you can evaluate it now.

With this, goodbye for now. I will see you next week.

The newsletter is growing well and now has 2,500+ subscribers. It gives us the confidence that it is useful to lots of people. Please share it on LinkedIn, Twitter, and with your friends and co-workers to help spread knowledge.

Subscribe if you haven’t yet 😊

Subscribe now

Sincerely,
Deepak

Check it out on Amazon if you haven’t yet - Tech Simplified on Amazon. It’s available on amazon.com as well.

Off to the topic,

The Need for a Growth Model

No matter where you work, be it a startup or a big company, people will expect you to set growth targets if you are leading a product or business.

We often take an audacious growth target that looks good to others. A hypothetical sample growth target sheet looks like this.

The company wants to achieve 1.8 million DAU by Mar '21. The argument presented in favor of the target is that "It's just 10% MoM growth, so we should be able to do it. Sounds right? After all, what are we even doing if not achieving this target."

Mar '21 arrives, and we are sitting at 1 or 1.2 million DAU, missing the target by a significant margin. And what do we do then? Take another audacious target. Albert Einstein said,

"The definition of insanity is doing the same thing over and over again, but expecting different results."

What growth teams do while setting top-down goals based on what looks good is nothing short of insanity. So, where are we going wrong?

We are focussing on output metric (DAU) and not inputs. The output or DAU is the final result. What you can influence is inputs, and then only the output will.

Now, if you are reading this, some of you might be feeling, "oh, we do it in the right way." Kudos if you are doing it the right way :)

Majority of teams that I have interacted with, unfortunately, do it the wrong way. Some of you may be feeling, "It's a top-down target. What can we do about it?" You can build a growth model to forecast DAU and MAU. Maybe the management won't listen to you in the next goal-setting exercise. But the one after that, they will.

Let's move on to see how we build this growth model. Where do we start? We start by understanding the inputs of the model.

8,000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

Inputs and Output of a Model

A model requires inputs to provide an output. We already have the output — expected DAU.

There are two inputs we require to forecast DAU growth.

1. Daily new users

2. Day-N retention

Let's be clear on these terms. Daily new users is straight-forward.

Day-N retention measures how many of your users come back to your app on the Nth day. Day 0 is the day user first came to your product. Let's take an example to illustrate Day-N retention.

Suppose 100 users came on 1st September '20 for the first time. Of these 100 users, 50 users came back on 2nd September. The D1 retention here is (50/100) ~ 50%. You can do the same maths for other days. Here is a sample sheet to illustrate this for two different dates. About 100 users came for the first time on 1st September. The other columns represent how many of them came back on respective dates. Similarly, 80 users used the product for the first time on 2nd September. The other columns represent how many of them came back on individual dates.

Let's simplify this table in terms of D0, D1, D2…D7.

Now that we put the table in this form, it becomes easy to calculate Day-N retention.

Once we have understood Day-N retention, we can move forward to see how we can forecast the DAU.

Forecasting DAU

Suppose you launched the app on 1st August 2020, and it has been a week. You have one week's data that looks like this.

As we have 7 days of data, we can get D1 to D6 retention. To do that, we first transform the table into the form below and evaluate average new users and average retention for Day-N.

From the table above, we have the input metrics.

1. Average daily new users ~ 80

2. D1 ~ 42%, D2 ~ 38%, D3 ~34% and so on

As we have data for the last 7 days, we can predict the growth for the next 7 days because we don't know how retention will behave in the long run.

Assuming you don't plan to do any additional marketing exercise, the new users will stay the same for the next 7 days. We already know the retention behavior. Feeding it into the model (spreadsheet with all the calculations here)

As you can see, the model employs this formula to calculate DAUs.

There is one caveat in this model. What about users who came in the period "1st-7th August"? For those users, we need D7-D13 retention data to calculate DAU. As in this hypothetical scenario, we launched the app 7 days ago; there is no way to no D7-D13.

What we can do is make some approximations. You can check how retention for apps with similar retention (D1 ~42% and D6 ~25%) behaves in D7-D13. Usually, the dip in retention in the first week is higher than the second week. Let's assume the D13 retention would be around 18%.

Adding both cohorts' DAU contribution, the DAU for this date range will look like this.

So by 14th August, the app will have a DAU of 359. If we want to increase further (say get to 500 DAU), what should we do? One or more of two things,

1. Increase daily news users

2. Increase retention

If we don't have a way to increase any of the two, it won't be possible to reach 500.

Q&A

There might be some questions around this. I am answering a few on the top of my mind. Please feel free to comment/DM me for any questions you may have.

Q: How can this growth model be applied to get MAU projections?

A: For MAU, the average N-month retention and average monthly new users will be the input metrics.

Q: What do I do if I am just about to launch my product?

A: Focus on acquiring the first 1000 users. It would be best if you have new users, not a growth model :)

Q: What do I do if I have an old product (>1 year) with a DAU of, say 1 million?

A: Good news! You can get D1-D365 retention and create a fantastic growth model

Q: How do we set growth targets now?

A: We can set targets on input metrics (retention and acquisition), and basis that an output metric like revenue or DAU

Q: Can it be applied to the Revenue Growth Model?

A: Yes, we can replace NU by New Revenue and User Retention by Revenue Retention and viola, we have a revenue growth model 😊

Q: What's next week?

A: Next week, we will cover "How to create Growth Loops in your product" The best products have strong growth loops: virtuous cycles that accelerate your growth

See you next week.

Sincerely,

Deepak

In the last week’s post, we saw how usual MoM growth trends fail us. We also covered how to break MAUs into user segments and define useful metrics to decode growth.

This week, we will take decoding growth to the conclusion by discussing two methods — cohort analysis and distribution curves. Let’s start with cohort analysis.

Check out the other posts that I have written if you haven’t

1. Referrals: The Holy Grail of Growth

2. Understanding Positioning Through a Practical Lens

3. How to Craft Your Product Positioning: Framework and Examples

4. 5 Golden Rules of Positioning with Case Studies

2000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

Off to the topic,

Cohort Analysis

A cohort is a group of people having something in common. As we are talking about growth, the cohort refers to people who did <x> in the product in the same period. For example, a July’19 cohort can be people who used the product for the first time in July’19.

Cohort analysis is looking at different cohorts’ behavior over time to see if something has changed. For example, suppose users who joined recently are behaving differently from the users who joined last year. In that case, we should find why it’s changed and do something about it.

We usually measure growth in three different ways — users, time spent, and revenue. We will cover how, including all these three in analysis, it helps us find deeper data patterns.

Let’s start with the first one, users. Here we look at the retention of different user cohorts over time.

User Retention of Cohorts

A retention plot of user cohorts starts with month0, i.e. the month in which the user came to the product for the first time. M0 retention for any cohort is ~100%. So an Apr’19 cohort means that the user came to the product for the first time in Apr’19. For this particular cohort, month1 will be May’19. Month2 will be June’19 and so on…

What about the Jun’19 cohort? Month0 will be Jun’19. Month1 will be Jul’19 and so on…

To create a user cohort retention curve, we pick the x-axis as months after acquisition (i.e. age of the cohort); the y-axis tracks the monthly retention. Each line in this curve represents a different monthly cohort.

Looking at this cohort lines, we can infer Month12 retention of users who joined in Apr’19, May’19, and Jun’19. What we can see is that M12 retention of Jun’19 is higher than Apr’19. So that means the long term retention is increasing.

We also look at whether the retention curve for different cohorts is flattening at one point. If the retention curve doesn’t flatten at any point, it will eventually become 0%. This kind of retention curve that becomes 0% at one point isn’t good as it works as a leaky bucket for growth. As none of the users of one cohort retain in the long run, we have to keep spending money and adding new users to make up for all the users lost.

Suppose the retention curve flattens at, say 20%. In that case, we know that for every 100 users coming to the product, 20 of them will keep using the product for a long period(loyal). These loyal users keep adding up over time and create a huge base.

We will talk more about it in some of the Growth Catalyst's future topics like retention and growth projections.

Customer retention cohorts are similar to user retention cohorts. Month0 for customer retention cohorts is assumed to be the month customer purchased for the first time.

Revenue Retention of Cohorts

The next graph is revenue retention. The revenue retention plot also starts with Month0, i.e., the month in which customers made the first purchase. For that reason, M0 retention of a cohort is ~100%.

Let’s take an example of the Apr’19 cohort. Users in this cohort made their first purchase in Apr’19. Say the total revenue made from the Apr’19 cohort was $1000. We will check the total revenue made from this cohort in next month (May’19). Say the total revenue for the cohort in May’19 is $800. So the M1 revenue ~$800 and M1 revenue retention ~ $800/$1000 = 80%.

A major difference between user and revenue retention is that revenue retention can be > 100%. Revenue retention > 100% happens when a user spends more money in later months, compared to their first month.

As we can see in the chart above, revenue retention has reduced significantly for the Jun’19 cohort.

Just like revenue retention, we can plot engagement (time spent) retention too.

The next chart is a variation of revenue retention. The chart is looking at the life-time value of cohorts. The lifetime value of a cohort is defined as the total revenue generated by a cohort during its lifetime.

Usually, if the revenue retention is falling, we assume that it’s a bad thing. A cohort with lower revenue retention may have a higher lifetime revenue. This is why we need to see the lifetime value of cohorts.

Life Time Value of Cohorts

LTV chart is an interesting one. For this graph, customers have been grouped together in cohorts by the month they first paid revenue to the company.

You can compare this chart and the revenue retention chart. Even if the revenue retention chart of Jun’19 cohort is worse than May’19, the LTV of Jun’19 is better than that of May’19. How does it happen?

As you may observe, the M0 revenue for Jun’19 is pretty high compared to May’19. As revenue retention is calculated with M0 revenue as a denominator, the graph for Jun’19 looks worse.

The key thing to look over in LTV of cohorts is the shape of the cohort lines.

• If the cohort lines curve upwards, that means customers actually spend more as they grow old. This is a pretty good thing for a business and can be used to justify the high upfront cost. Ex - Apr’19

• If the cohort lines are straight, there isn’t much change in spend MoM. Still better than the next one.

• If the cohort lines curve downwards, that means customers are spending lesser as they grow old. This is the case in the Jun’19 curve shown above. It’s worrisome and the growth team should do something about it.

Another Way to Visualize Cohorts

If you have used tools like Mixpanel, you may have come across this kind of retention table.

In the retention table, the size of the cohort for each date is present, and daily retention for 11 days is also present. This table is also color-coded like a heat map, where the color gradient is used to represent how good or bad the retention is.

This table's benefit is that it is easier to read, especially when you are trying to draw retention cohorts for a long period.

You can draw the same table for all the 3 different charts described in the sections above. A sample LTV heat map will look something like this

Distribution Charts

You might be familiar with the frequency distribution chart. It summarizes values and their frequency. For example, if I have to plot users’ engagement by the total number of days they were active in a month, I can use a distribution chart.

A histogram is the most commonly used chart for visualizing distribution, where we display data using bars of different heights.

Let’s try to build a histogram to plot users’ engagement by the total number of days they were active in a month. This is pretty useful in analyzing user engagement and also know as a power user curve or L30 distribution.

Power users are very loyal to your product and use it more frequently than others. L30 is called a power user curve because it can help you figure out what % of your monthly active users are power users.

This is how a typical L30 curve looks like,

There are multiple reasons to love this chart:

• It shows if you have a loyal segment that’s coming back every day. We see from the chart above that around 4-5% of all users come more than 10 days a month.

• It shows the variability among your users: some are slightly engaged, whereas others are power users. Contrast this with other measures of engagement like time spend or DAU or MAU. A single number like these doesn’t give you a lot of information. The L30 chart on the other hand does.

• When we plot the L30 curve for multiple cohorts, we can see how it is changing with time.

Now let’s ask another question, is there a better way to plot the L30 curve? Looks like there is. It is easier to read as a Cumulative chart. In a cumulative chart, we plot the cumulative distribution. Here is how you calculate cumulatively

So when we draw the cumulative chart for the L30 curve, here is how it looks.

In this chart, we can quickly read that around 95% of users are active <10 days a month. Around 70% of users are active < 5 days a month.

What about revenue? Can we apply the same to revenue?

Distribution Charts for Revenue

We can definitely start by plotting a revenue distribution chart. It is better than a single metric like average revenue per user (ARPU).

Please note: Because of the wide range in revenue per customer ($100- $10,000,000), we have used a log scale on the x-axis.

Do we really see such diversity in revenue per customer? In B2C businesses, we don’t. But in B2C we do see such diversity. Take Salesforce as an example, some startups pay <$1000 to Salesforce but some of the fortune 500 companies might be paying millions of dollars.

We can convert the graph above to a cumulative graph like the one here. It is easier to read.

Quick observations:

1. Around 25% of customer pay more than $100,000

2. Around 50% of customer pay more than $1000

Let me in you with a secret, there is always a better way to represent data. And that is true even here ;)

In the next graph, we will plot cumulative revenue as well.

Quick observations from this graph are:

1. Around 55% of customers contribute to less than 1% of revenue. In fact, 75% of customers contribute to less than 5% of revenue

2. Top 10% customers contribute to >80% of revenue

Sounds good?

Now let me ask you a different question. Given that the top 10% of customers contribute to > 80% of revenue, should you be worried?

Well, depends on the scale. Suppose you are in an early-stage SaaS business with 20 customers, and your majority of revenue (80%) comes from 2 customers. In that case, there is a greater risk to your business if those 2 customers go away. If you are late-stage and have 1000s of customers, not so.

Just as we have analyzed revenue, we can analyze engagement in minutes. You can get a lot of useful insights about your power users by doing so.

With this, let me conclude this week’s post. Next week, we will cover growth projection models.

Have a good Sunday!

Sincerely,
Deepak

A reader last week sent a message requesting the newsletter should be sent every Sunday by 9:30 am IST. The posts are long sometimes, and he wanted to start reading this in the morning so that there is enough time to finish it. That makes a lot of sense, so I am going to follow that. Pardon for any exceptions though :)

In the last 4 months, we have covered product-market fit, positioning, go-to-market, and data. These are all foundational pieces of growth. Without them, it will be very hard for anyone to define the growth strategy. The topics I am going to write in the next 4-5 months are very interesting and actionable. I actually use them regularly.

1. Decoding growth, growth projection models, and loops (4-5 posts)

2. Onboarding and Retention (4-5 posts)

3. Experimentation - how to generate ideas and set ab tests (2-3 posts)

4. Gamification (3-4 posts)

5. Mental models for Growth (4-5 posts)

Check out the other posts that I have written if you haven’t

1. Referrals: The Holy Grail of Growth

2. Understanding Positioning Through a Practical Lens

3. How to Craft Your Product Positioning: Framework and Examples

4. 5 Golden Rules of Positioning with Case Studies

2000+ people have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

If you aren’t familiar with the newsletter and online sessions, you can read about it here

Off to the topic,

Let’s start with decoding growth, i.e. understanding where growth is coming from. You can call it growth accounting as well since we are accounting for where growth is coming from.

User Growth: MoM Trends aren’t Useful

Most products, when asked about their growth, will show you this graph MoM

I have seen this graph on pitch decks. Founders/PMs/investors love talking about MoM growth. If you remember our discussion about BranchOut from measuring the product-market fit, investors actually pumped $49 million in that startup on the basis of MoM growth. In the post, I wrote,

As you see in the chart below, it got massive growth between Feb-Mar 2012 which is investors’ dream.

BranchOut MAU in Feb-Mar 2012 before Series C, $25 M round

However, the growth soon started slowing down. The MAU numbers kept decreasing. This is how later part of 2012 looked for BranchOut.

Rise and Fall of BranchOut MAU: A Nightmare for Growth

Despite all efforts by Rick Marini including a pivot, the company had to sell its assets at $5.1 million in August 2014 to 1-page much lower than the total amount they raised.

So we know that relying on MAU growth isn’t enough. What should we do?

First, we need to define the growth rate in a better way

Reading Growth Rates

We all know the definition of growth rate as the change in users/users (t-1).

growth rate = users(t)/users(t-1) -1

The growth rate is often plotted MoM to see if the growth rate is slowing down. Here is how MoM growth will look for the graph above

The problem with this chart is that it’s very hard to read whether the growth rate is good or bad because of variation.

A good method to deal with this problem is CMGR (cumulative monthly growth rate). A CMGR can be calculated over a period of 3-month, 6-month or 12-month.

CMGR for 3 months (CMGR -3) will be calculated like this:

1. Divide the users at the end of the period (say July) by its value at the beginning of that period (April) ~ users (July)/ user (April)

2. Raise the result to an exponent of one divided by the number of months (3) ~ [users (July)/ user (April)]^(1/3)

3. Subtract one from the subsequent result ~ [users (July)/ user (April)]^(1/3)-1

Here is how CMGR-3 looks like

Much better to read and understand? Looks like it’s growth rate is increasing. Happy? Don’t be so sure :)

Breaking MAUs into Segments

To understand MoM growth well, we need to first define a few terms to represent users falling in different segments:

1. New: new users in the current time period (t)

2. Churned: Lost users who were active in the previous time period (t-1), but not active in the current one (t)

3. Resurrected: Users who had churned at some point in the past and were not active in the previous time period (t-1), but resumed in the current one (t).

4. Retained: Users who were active in the previous time period (t-1) and are also active in the current one (t)

From an MAU trend perspective, the time period will be a month. However, the time period could be a day, week, month, or year depending on what trend you are looking at.

Now that we have understood these terms, the next step is to split the MAU into three buckets — new, resurrected, retained.

This is because all users in a month are equal to the sum of new, resurrected, and returned.

users(t) = retained(t) + new(t) + resurrected(t)

This is how the graph looks like now.

Please note that the new users in the graph are increasing at a higher pace than the overall growth. What does that mean? Is it a good thing?

We can answer whether it’s good or bad by defining three metrics.

Useful Metrics for Decoding Growth

There are three useful metrics for decoding growth — gross retention, quick ratio, and net churn.

Gross retention is looking at what % of users got retained from the last period

Gross Retention = retained(t) / users(t-1)

The quick ratio is the sum of gains in users divided by losses in users. QR > 1 for positive growth

Quick ratio = [new(t) + resurrected(t)] / [churned(t)]

Net churn determines if the company can grow without new users. If net churn is positive, the product needs to add users to compensate for the lost growth. If it’s negative, the product will grow even without adding new users.

Net_Churn = [churned(t) – resurrected(t) ] / users(t-1)

Let’s calculate these 3 metrics for the data plotted in the MoM graph above. This is what the calculations look like MoM

As you can see, the QR is fluctuating and you can’t read much from that except that it is >1 and so the product is growing.

Gross retention was stable but is falling steep in Nov and Dec.

Net Churn is interesting, it’s increasing. Now the question is — how come that net churn is increasing, but the growth rate is also increasing in the CMGR-3 curve?

Let’s relook at the data for these months for new, resurrected, churned and retained users (in millions)

As you can see, the makeup for churn is done through an increase in new users. The product is acquiring new users to fill the deficit of users lost. How sustainable is it? Not much. Remember BranchOut? It was doing the same but couldn’t do so in the long run.

To conclude this section, using growth rate and 3 metrics of growth accounting, we can quickly figure out whether there is any problem with current growth and fix it in advance, rather than seeing the dip in growth later when it’s too late.

Okay, so can we use the same model for accounting user engagement or revenue?

Yes, we can.. with some changes.

Growth Decoding for Engagement/ Revenue

There is a fundamental difference between user growth and engagement/revenue growth. Let me take one example,

A user who came last month and came this month is counted as a retained user. The retained user, however, can spend less/more money contracting/expanding the revenue. Same for engagement mins.

To account for this, we have to introduce two more terms

1. Expansion: Gained from users increasing time spent/revenue relative to the previous time period.

2. Contraction: Lost from users decreasing (but not to zero, otherwise they would be churned) time spent/revenue relative to the previous time period.

All revenue in the current time period (t) is now equal to

revenue(t)= retained(t)+ new(t)+ resurrected(t)+ expansion (t)

The definition of churn also changes

churn(t)= churned(t)+ contraction(t)

Defining Useful Metrics for Engagement/ Revenue

Gross retention is looking at what % of revenue got retained from the last period

Gross Retention = retained(t) / revenue(t-1)

The quick ratio is the sum of gains in revenue (new, resurrected, and expansion) divided by the losses in revenue (churned and contraction)

Quick ratio = [new(t) + resurrected(t) + expansion(t)] / [churned(t) + contraction(t)]

Net churn is the sum of losses minus gains in revenue from existing customers only (i.e. excluding new customers) divided by total revenue from the previous time period.

Net_Churn = [churned(t) + contraction(t) – resurrected(t) – expansion(t)] / revenue(t-1)

Sounds good? Now you may have a few questions. Some of the common questions around growth accounting are addressed below.

Q: Can we apply the same framework to DAU growth, WoW growth, or YoY growth?

Yes, you can. The period definition changes, and nothing else.

Q: This looks daunting to calculate. Should we do it on a regular basis, or as a one-off analysis?

You should ideally automate these reports so that you don’t get daunted by them every month or week.

Q: Isn’t it overkill, or data puke to add all these metrics as you mentioned in earlier posts?

These are not vanity metrics as discussed in our earlier posts around data. All I am asking you is to track 3-4 important metrics around users, engagement, and revenue growth. These will ensure you have a check on the pulse of your growth.

Q: If I am already tracking retention, how does it help?

Retention is a very difficult metric to move. You should usually check your retention every month/quarter or whenever you launch a new feature. Growth accounting, on the other hand, helps you keeping track of growth initiatives on a daily or weekly basis.

Wish you a good Sunday. DM me in case of any suggestions. Read here the next part of this analysis through the cohort analysis and distribution curves to see how it helps us decoding growth further.

Sincerely,
Deepak