How to Spot Good Case Studies

Or why you should ignore 90% of case studies you come across!

I am more confident in my ability to make product and business decisions today, than I was 5 years ago. A key factor has been deeper understanding of products, industry, and markets.

So, how did I built this understanding? By learning how to identify good case studies for a particular problem I am solving.

An example could be a particular post talking about how OpenAI became successful by doing X. I searched google around ‘how OpenAI became successful’, and this was among top 3 results.

While reading this paragraph, it’s difficult to not agree with it. But the reason to agree with it isn’t because it’s a terrific case conclusion. The reason we agree with it is because nothing it says is specific we learn from OpenAI, the takeaways are true about every product out there — prioritise usability, harness user feedback, maintain agility.

So how do you build deeper understanding using case studies? Let’s start by understanding the usefulness of case studies.

The Usefulness of Case Studies

Domains like business and product management aren’t structured domains, like mathematics or coding. Case studies are a powerful way to study such domains. That’s why many reputed business schools like HBS or IIMs use case studies to teach business.

As to why case studies are so effective, it’s because they help you understand the context of different problems and how people solved them. Let’s take revenue growth as an example of a problem companies want to solve. By studying how Facebook, Google or Adobe solved the revenue growth problems, you understand how to solve this problem in different contexts.

And the next time you face a problem, you can draw similarities and analogies from previous case studies to solve the problem quickly.

But that also highlights one of the major drawbacks of this method — cases can be misapplied.

The Misapplication of Case Studies

Case studies can misfire if you found some similarities, but missed key differences. Applying the wrong analogies can lead to an ineffective solution. An instance of this was publicly shared in Lenny’s Newsletter by the VP of Product of Duolingo.

The VP joined from Zynga which built the wildly popular game — Gardenscapes. Gardenscapes has progress bar that shows people how close they are to solving a puzzle, and it also shows number of moves left via moves counter.

When applied to Duolingo, this meant asking a question to the user (like translate the text), and providing a progress bar for accuracy and moves counter.

It didn’t work out because a puzzle has anticipation where I don’t know whether I can solve it or not. While translating a text, I know (more or less) the output and there is no anticipation. The progress and moves work well when the outcome is unknown, but becomes useless game dynamics when the outcome is known.

The team corrected this by re-thinking on what mattered to users in learning language and built successful features over time. It then drew right analogies, and built successful products like leaderboards and learning streaks.

You can read the full post written on Lenny’s newsletter here, it’s one of my favourite ones.

---

Quick announcement — we are launching a live cohort for experienced PMs in July. You can find a short video explaining the unique aspects of the program here:

You can check more details on the program page — https://www.pmcurve.com/product-management-interviews

---

How to Identify a Good Case Study

To understand what a good case study looks like, we need to study it’s anatomy. A good case study focuses includes

• User knowledge — the users, their needs and preferences, their cultural and social norms, their fear and motivations, etc.

• Industry knowledge — the competitors, industry structure, how attractiveness the industry is, gaps, etc.

• Product path — what was the path that a particular product took in early days to gain PMF, why was this path successful, what did it do to grow, etc.

• Internal context — internal advantages the team or company had to start with.

With proper context (user, industry, product, internal), your analogies and applications become far more effective.

Here are additional signs to look out for to find good case studies

1. Long-form — A long-form article usually is a good case study, provided it covers all of the above.

2. Reliability of the source — An article written by an operator who worked on the problem or similar problems in the past. For example, Eugene Wei’s blog on Amazon’s growth strategies provides a detailed, insider perspective. That makes it a richer and reliable case study. Even non-operators write good case , but they usually research and include reliable sources while doing so. Stratechery is a good example of cases written by a non-operator.

3. Little counterfactual evidence — It’s hard to find counterfactual evidence to challenge the narrative.

We started this post by saying that business schools like HBS have an effective case study method. When we look at HBS case studies, we can see why they are effective. The first reason is depth. The cases studies consist of several pages, which allows them to cover context clearly. The second reason is operators’ involvement. HBS case studies are often created by founders and operators. Many times, these operators participate in the case discussions as a guest faculty.

How to Avoid a Bad Case Study

Avoid

1. News-like: Sources like TechCrunch, YourStory, etc. piece together information around startups. People read it because it’s entertaining and also keeps them informed. But the flip-side of reading such articles is that they have a lot of narrative bias when they talk about success or failure of a startup.

2. Social media posts: Posts on social media often have simplified narratives to appeal to a broader audience. Such simplification can lead to narrative bias.

---

This would be all for this post,

Deepak