As a part of this exercise, the next logical step is to look at broad areas of product management - Growth, Product Sense, Strategy, and go deeper into it. In this post, let’s look into how Growth PM gets affected by advances in AI.

In case you missed our last email, here is the complete AI Prototyping Course for Free (10+ hours recording of live sessions)

Free AI Prototyping Course

Skills of a Growth PM

Growth is a loosely defined function in product management, in that it could mean various things — getting users to the product, managing churn, monetisation, etc.

All of it requires a set of growth skills. To understand how we can use AI to grow products, we can start by listing down these skills and then see how AI impacts/ enhances these skills.

1. The first skill is getting a product from 0 to 1 — attaining product market fit. This includes defining PMF, measuring PMF, and improving it consistently to get to a strong PMF

2. The next skill is understanding how to get users on the product using product. For example, referral is a good example of getting users on the product through existing set of users.

3. Once the user is on the product, they need to experience the value that the product offers quickly and in the right way. We call this step activation. Activated users (users who have experienced value once) tend to stick to the product longer, and hence this becomes a focus area for the team if the metrics down the funnel like retention, and monetisation, aren’t healthy.

4. Once you get the users activated, you as a Growth PM need to figure out how to get them back to the product again and again. Retaining users, or strong retention is the hallmark of a good product. This is where core product teams working on the product offering also get involved. These core product teams can work with Growth teams to move retention up.

5. So far, we have only talked about the user. Another stakeholder of product teams is the business. We need to monetise the product to create a sustainable business. It’s a common myth that monetisation happens after retention. In many education products like Unacademy, Reforge, etc, the monetisation happens upfront before access to the course is provided.

6. Beyond the five skills we have listed above, there are few skills which are horizontal in nature, i.e. they can be applied across. Designing experiments, or A/B testing is an important skill for any growth PM. It helps the team move faster and gain quick wins.

7. Data is a another super important skill of Growth PMs. Data informs your experiments, help you track the progress, and also helps you predict growth.

8. The last skill is building a coherent growth strategy and roadmap, which is needed when to move to product leadership roles. A subset of this skill is Growth prediction through mathematical models.

AI’s Role in Growth

Now that we have listed these skills, let’s look at how AI impacts these skills. Collating the impact across skills can tell us the role of AI in growing products. Note that I don’t cover the impact exhaustively in this section, but I plan to do it in a live session next week. The link for the live session is available at the end of this post.

1. Product-market fit: AI lets you prototype quickly. This means that iteration cycles can be extremely fast. A fast iteration cycle allows you to test multiple ideas and hypothesis quickly. You can also use AI to analyze user feedback and identify pain points faster.

2. Product-led acquisition: As building products becomes easier with AI, we will see many companies offering sophisticated features for free as an acquisition strategy. We will also see AI-led or assisted content strategies.

3. Activation: Activation is where AI can play a transformative role. Think about an AI assistant that guides users through onboarding, which adapts experience in real-time based on user behavior and preferences. AI can also identify friction points in real-time to resolve them. All of this can dramatically increase activation rates compared to static onboarding flows.

4. Retention: Engagement is a leading indicator of retention. AI can be helpful by increasing velocity of features or content to engages users more. This, in turn, will increase retention. AI can also be useful in conducting and summarizing user research around churn.

5. Monetisation: Monetisation models are many, and what works for a business doesn’t work for another. Sometimes, an innovation in monetisation model can disrupt an entire industry like subscription did with SaaS. While AI can’t take monetisation decisions with conviction, it can be useful in brainstorming novel pricing models. It can also help you quickly gain understanding of similar industries and help you cross-transfer learnings.

6. Experimentation: AI excels at hypothesis generation by analysing user behaviour or user research documents. It can also help prioritize which experiments to run. While Growth PMs need to make the final decisions, those who leverage AI for experiment design may be quick in testing more ideas with higher success rate.

7. Data: Natural language to query translation is available now as a technology. Growth PMs can now ask sophisticated questions about retention cohorts or conversion funnels in plain English and receive instant visualizations and insights. This faster data exploration and analysis means more informed decisions.

8. Growth roadmap and strategy: Domain research in the AI era has changed. AI can synthesise competitive insights, market trends, and user trends. It can help you identify gaps in the market that competitors haven't gone after. AI can also help in sophisticated forecasting. All of this can make you more confident in growth strategy and roadmap.

Live Session: Using AI to Grow Products

In the session, I plan to cover how AI impacts these various skills, and what to do about it. The session assumes that you have broad familiarity with product management and broad familiarity with growth terms. If you didn’t struggle understanding this post, the session will be useful to you.

Here is the registration link for the Maven session

Register for the Session

This would be all for this post!

Here are some apps that learners have built

https://ai.pmcurve.com/

Ankit built a landing page for his new company, and I love it

Sambhav built this bedtime tale generator!

And Archit built this game

You can see all the apps here — Here is What I Built

The best part is that I built this page using AI. You can showcase your own products by submitting the form on the same page.

Next Session Registration

You can register for the next session here: https://lu.ma/shap8ufk. It’s scheduled on 19th Feb (Wednesday, 9 PM IST).

Session Recordings

In the second session, we covered evals and used lovable.dev to build an app with both front-end and backend.

The recordings can be accessed here on our custom learning platform —

Get the Session Recording

Hope to see you in next session,

Deepak

Welcome to another edition of the Growth Catalyst Newsletter. I wanted to cover one of the most important aspect of building AI products this week — defensibility. With thousands of AI products launching every month, defensibility will determine which ones will survive.

To cover this topic, creating a video around it seemed like an apt way to do it. It talks about

• Why defensibility is important to think about in AI products?

• Thin vs Thick Wrappers

• Attributes of thin wrappers and why they are weakly defensible

• Attributes of thick wrappers and why they are strongly defensibly

• What to do if you are building a thin wrappers?

• Where does thin wrapper succeeds?

Go through the video and like/comment if you found this useful.

Free Session on Product Growth

I have another 2-hour session on Product Growth coming up tomorrow, 11 AM IST. Around 300+ folks have registered for it. You can check and register here if the topic interests you — https://lu.ma/rw0aqbp2

Last but now the least,

The 2nd Cohort of Product Sense and Strategy is Here!

Brimming with all the love we have gotten so far, the 2nd cohort of Product Sense and Strategy would commence on 17th August :)

The program is explicitly for experienced PMs and founders. It would be the last live cohort on the topic this year.

We have received highest applications ever for this cohort. The final shortlist goes out this weekend along with details around next steps.

We have 25% spots left, so this would be a good time to apply! If you aren't applying, tell a friend who will appreciate it.

Apply here — https://9y9647f1j0n.typeform.com/to/F7juz3PJ

Check the program here — https://www.pmcurve.com/product-sense-for-pms-and-entrepreneurs

This would be all for the week. Thank you for reading and see you next time,

Deepak

For the new ones here, do check out the popular posts that I have written recently if you haven’t

1. Assessing the effectiveness of PMs

2. Product metrics: to measure or not to measure

3. Necessary Skills to Enter PM Roles/ Crack PM Interviews in India

4. Hidden Layers in Product Management

5. How to Crack the Technical Round of PM Interviews

10,000+ smart, curious folks have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

Dear Reader,

We launched a new program on Product Sense and Strategy yesterday. Here are the slides from the session conducted for the shortlisted candidates for the program. The response has been good and the seats are 80% full in less than 24 hours. If interested, you can apply here.

Over to the topic at hand,

While learning about AI product management, it is important to address practical questions and challenges as we cover the topics. A key question that every PM will face while building AI products is whether Supervised models are still relevant in the era of LLMs?

If we go by the perception of large language models such as GPT, people tend to believe that LLMs can solve all the AI problems. The truth is far from this perception, and we will focus there in this post.

The Relevance of LLMs

LLMs excel in natural language tasks which require understanding the natural language texts and generating them. The prime example for this is ChatGPT and BARD. Both have a text-first interface.

Over time, LLMs will become multi-modal. Multi-modality means that it can handle multiple forms of inputs and outputs like text, image, voice, video, etc.

You can read this series of posts if you are interested in understanding LLMs deeper -

The Growth Catalyst Newsletter by pmcurve

ChatGPT and Large Language Models for Product Managers

9,000+ smart, curious folks have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇…

Read more

3 years ago · 6 likes · Deepak Singh, pmcurve.com

But as you understand them better, you realise that they aren’t useful for every problem out there.

Supervised models continue to work well in scenarios where labeled data is abundant and we need to predict specific outcomes accurately. Let’s delve deeper into it.

The Supervised Models

For PMs, understanding the strengths and limitations of supervised models is crucial for identifying use-cases where they excel.

For example, Customer churn prediction (predicting which customer will churn out of product in a given time period) can be better done by supervised models. LLMs aren't suitable for all predictive tasks in businesses. The reason behind it is that while LLMs are getting better at high school maths, they are designed for words, not numbers. Analysing and predicting numerical data often requires specific mathematical operations, statistical techniques, and algorithms.

Supervised Models > LLMs

Here are few cases where Supervised models can outperform LLMs.

1. Small Datasets: LLMs require vast amounts of data for training, and they may underperform when faced with small datasets. For example, in credit card fraud detection, companies like Feedzai use supervised models trained on historical transaction data to identify fraudulent patterns and behaviours by using anomaly detection at a user level.

2. Content Moderation: LLMs struggle with content moderation of a specific community, where adherence to specific guidelines or regulations is essential. Here is a paper detailing that if you are interested — https://arxiv.org/pdf/2309.14517.pdf.
   
   From the research paper,

   One potential reason for this is that while LLMs are able to reason in a “forward direction,” by interpreting rules and examining if content falls within those rules, CrossMod is a “reverse direction” tool which starts from actual content removals and learns patterns that can be applied to other communities.

On the other hand, LLMs are better than supervised models in NLP tasks such as

• Natural Language Understanding (NLU) such as sentiment analysis, language translation, and question-answering.

• Text Generation tasks like content creation

• Semantic Search: Microsoft's Bing search engine employs LLMs like GPT-4 to improve search relevance and answer user queries more accurately.

• Speech Recognition

• Chatbots

Supervised Models + LLMs

In some cases, leveraging a hybrid approach that combines the strengths of supervised models with the capabilities of LLMs can yield optimal results. For example, we can use LLMs to classify natural language feedback from users as positive, negative, and neutral. This feedback rating can then be fed into supervised models to predict churn better.

Another example is domain-specific tasks. LLMs are trained on general text available on the internet. They may struggle with domain-specific tasks that require specialised knowledge or industry-specific terminology. For example, analysing a legal document well will require both labelled dataset training (supervised) and LLMs.

The supervised models + LLMs will outperform LLMs by leveraging domain-specific knowledge and terminology unique to the legal field.

Cost

The current cost of LLMs is really prohibitive for it to become a viable option for many small-and-medium companies. In fact, it can become unsustainable even for a company like Google. From Arstechnica article,

Exactly how many billions of Google's $60 billion in yearly net income will be sucked up by a chatbot is up for debate. One estimate in the Reuters report is from Morgan Stanley, which tacks on a $6 billion yearly cost increase for Google if a "ChatGPT-like AI were to handle half the queries it receives with 50-word answers." Another estimate from consulting firm SemiAnalysis claims it would cost $3 billion.

Cost should be an important consideration for any business and that is another factor that makes supervised models relevant.

Summary

To summarise, while LLMs are amazing for NLP and NLP-adjacent tasks, they are not very suitable for numerical, predictive tasks. It is particularly true in specific domains, or domains where dataset is small. Add to that, cost is an important consideration to be taken into account while evaluating supervised vs LLM models.

An interesting thing to look out for would be to use both to solve cases that aren’t possible with one of them :)

That would be all for this week.

Check the video on this topic

Thank you for reading :)

If you found it interesting, you will also love my

1. Advanced Growth Program for PMs

2. Product Sense and Strategy Program for PMs

3. Bestseller Book ‘Tech Simplified for PMs and Entrepreneurs’

For the new ones here, do check out the popular posts that I have written recently if you haven’t

1. Assessing the effectiveness of PMs

2. Product metrics: to measure or not to measure

3. Necessary Skills to Enter PM Roles/ Crack PM Interviews in India

4. Hidden Layers in Product Management

5. How to Crack the Technical Round of PM Interviews

10,000+ smart, curious folks have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

Let’s dive in the topic now!

In the realm of product management and entrepreneurship, leveraging machine learning models has become increasingly prevalent. In the last post, we looked into PM’s role in AI models, supervised models, subtypes of supervised models, and how they work. In this post, we will talk about how to measure performance of supervised models!

Note that this post is also supplemented with a video where I talk about everything in this post in more detail wherever required. It gets easier to explain certain concepts that way, hence I am taking a hybrid approach. Here is the video :)

Let’s recap the fundamental components of the machine learning process:

• training data,

• algorithm,

• a trained model,

• and the subsequent results.

As we covered in the last post, AI product managers have an important role in two steps: creating the training data and measuring the performance of the model. Understanding how to measure performance of these models is important for PMs. Once you understand performance well, you can map it back to changes in training data required. We will cover the mapping in a future post.

Let’s get into the performance of supervised models.

Supervised models utilize labeled datasets for training. For instance, in a classification supervised model that distinguishes between images of apples and oranges, the labels are "apple" or "orange." Once trained, these models can predict labels for new data.

We can assessing the efficacy of these models by looking at how accurate the classification is for the set of images we provide. This process of providing new set of images is called Testing the model.

Subsets of Labeled Data

Testing the models requires systematic evaluation. The evaluation starts by dividing the whole labeled data into three parts — training, validation, and test.

• The training subset is used to train the various algorithms DS team feel can be useful

• The validation subset aids in selecting the most effective model

• The test subset evaluates the performance of the best model

It goes without saying that how we measure performance is useful in both validation and test sub-sets. Performance evaluation using validation subset helps in narrowing down to the right algo/model. Performance evaluation using test set helps us evaluate how good is the right model, and whether it can be deployed in production for the end users to experience.

So let’s look into how to quantify the performance of the regression models! It should be noted that the approach is different for regression and classification models because they work in quite different ways.

Performance in Regression Models

Regression models predict continuous values, such as housing prices. Two primary metrics, R-squared and Root Mean Squared Error (RMSE), are commonly employed to check the difference between predicted and actual values.

R-squared provides values between 0 to 1, and closer to 1 indicates better performance. R-squared less than 0.4 is considered weak.

Here is the datasheet and sample calculations.

Conversely, RMSE quantifies the average magnitude of errors, and is a absolute measure of deviation.

You can see the RMSE calculations in the same datasheet.

How good or bad RMSE is, can be defined relative to the magnitude of input and output. For example, RMSE of 2 when the input varies between 1 to 10, can mean up to 20% error. But RMSE of 2 when the input varies between 1 to 100, can mean just 2% error. This is where the judgement of PMs come into picture. They can decide which error rate is okay based on the user experience or in the business context.

Performance in Classification Models

In classification models, we use the confusion matrix.

From this matrix, precision and recall are derived. Precision measures the accuracy of positive predictions.

The recall measure the model's ability to capture all positive instances.

The F1 score, a harmonic mean of precision and recall, offers a balanced assessment of the model's performance.

F1 score closer to 1 is better, and you need both precision and recall high to achieve that.

Performance is Context Dependent

The context of user and the business is quite relevant in assessing the performance. PMs play a pivotal role in contextualizing these metrics within the broader business framework and determining the deployability of models to end-users.

In conclusion, a comprehensive understanding of performance measurement in supervised machine learning models is important for PMs. By mastering these concepts, PMs can effectively collaborate with data scientists, make informed decisions, and contribute to the successful deployment of AI products.

Stay tuned for our exploration of unsupervised models in the next post/ video :)

Resharing the video of me explaining this in-depth 👇‍

Thank you for reading this edition of the newsletter ✨

If you found it interesting, you will also love my

1. Advanced Growth Program for PMs

2. Product Sense and Strategy Program for PMs

3. Bestseller Book ‘Tech Simplified for PMs and Entrepreneurs’

With the advent of LLMs, this is going to change. AI is going to seep into every type of product in near future. So if you are a PM or founder, it’s important to build an understanding and have discussion around these. This way when you get a chance to manage AI/ML products, you know how and where to contribute.

In this article, we discuss the types of data science models, go deeper into supervised models, and PM’s role in it.

Learning AI/ML for PMs

Learning AI/ML for PMs has to be approached differently because their contribution to AI projects is quite different as compared to data scientists.

Broadly, PMs need to

• Understand basic concepts, different types of models and how they work

• Know real-life applications of these models so that they can map feasibility of a real-life problem to a AI project

• Know PM’s value addition and role in AI/ML projects

Let’s get into supervised models to understand these points better:

What are Models?

From a previous post Understanding Large Language Models,

To understand what a large language model is, we first have to understand what a model is.

You may have heard of algorithms if you have done basics of computer science. An algorithm is a set of rules that the computer follows to solve a problem. For example, we can have a set of rules to determine whether a given number n is even or odd, aka algorithm. You can see that algorithm below.

Understanding algorithms is prerequisite to understanding models. Many people confuse models with algorithms. Let’s take a machine learning algorithm.

In machine learning, every enthusiast starts with linear regression algorithm. It defines the relationship between one independent variable (x) and one dependent variable (y) using linear equation (= a straight line). It is well known that equation for a line is y = mx + c, which can also be written as y=w2+w1x. The mathematical rule that we just defined is an algorithm. Let’s talk about models now.

A model is when you determine w1 and w2 in the algorithm based on data you have. Suppose we were trying to find the correlation between the # of hours spent studying and marks obtained on the test. In this case, say we had these data points.

0 hours — 0 marks

3 hours — 33 marks

9 hours — 99 marks

We can put the first datapoint (0 hours, 0 marks) in the equation and see that w2=0

0=w2+w1*0 
w2 = 0

Let’s put another datapoint (3,33)

33=0+w1*3
w1 = 33/3 = 11

Now, let’s see if y=11x holds true for the third datapoint (9,99)

y=11*9 = 99

So now we have gotten a model, y=11x using which we can predict things.

Machine Learning Process

The machine learning process can be explained via the diagram below

We have the following steps:

1. Training data: The training data consists of past data used to train the model

2. Algorithm: The next step is picking an algorithm depending on the problem we want to solve

3. Learning/ training: The training data applied to algorithm creates a custom trained model

4. Results: We test out the trained model to see if it works as per the requirements

Let’s take an example of Supervised Model to understand these 4 steps

Supervised Models

A supervised model is the one where we provide labeled training data to create the model.

Here is a good example below where we provide labeled images of apples and oranges as training data.

After we have trained the model, we can then test it with the test images. Note that test images are similar, but not same as training data. If the model can accurately predict the images, we have got a good model :)

Role of the Product Manager in Supervised Models

There are two steps where PMs play an important role while working with supervised models:

1. Training data: Quality of training data determines the effectiveness of a model, PMs can play a good role here. They can use the systems thinking to point out what all parameters should be covered in the training data.

2. Understanding the effectiveness of the learning model : While measuring how good the model is, PMs can bring external benchmarks. They can also see the critical errors from the user point of view.

Types of Supervised Models

Supervised models are of two kind:

1. Classification: The model tries to classify an input into a category. For example, a particular email classified as spam or not is an example of classification model.

2. Regression: The model tries to predict a number based on past data. For example, predicting housing prices in an area based on input factors is an example of regression model.

This would be all for this article. In the immediate future articles, we would

• Delve deeper into classification and regression

• How to test these models

• How to measure the efficacy of these models

Summary

PMs play an important role in data science projects. Albeit the role isn’t sharply defined in most places. To be effective, PMs need to understand basics of data science, different models and their real-life applications, and their role in these projects.

We cover basics of supervised models in the article, and where PMs can play a role in it.

Here's the video of me explaining this in-depth 👇‍

Thank you so much for reading this edition of the newsletter ✨

If you found it interesting, you will also love my

1. Advanced Growth Program for PMs

2. Product Sense and Strategy Program for PMs

3. Bestseller Book ‘Tech Simplified for PMs and Entrepreneurs’

For the new ones here, do check out the popular posts that I have written about recently if you haven't

1. Assessing the effectiveness of PMs

2. Product metrics: to measure or not to measure

3. Necessary Skills to Enter PM Roles/ Crack PM Interviews in India

4. Hidden Layers in Product Management

5. How to Crack the Technical Round of PM Interviews

10,000+ smart, curious folks have subscribed to the Growth Catalyst newsletter. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

Let’s dive into the topic now!

The book Tech Simplified for PMs and Entrepreneurs was launched around two years back. One of the persistent requests from the readers has been to create an assessment to test their understanding and recall.

Merely going through a technical book isn't enough; you have to recall it in conversations. And so, an assessment would help build that confidence in themselves.

We are launching the first version as the 50-question long assessment. People have loved it so far.

Check the Assessment

I have consciously kept the questions simple. Here is how you should evaluate yourself — If you are able to score 80+, you are in the good league. A score of 60-80 means you don't have a good understanding of certain topics. Anything below 60 means you need to work a little on brushing up on all concepts.

Onto the next topic,

I have been writing a 3-part series on mistakes to avoid in product interviews.

• Part 1 was about the entry-level roles(APM/PM). You can read it here.

• Part 2 is about the lead PM roles (SPM/GPM). You can read it here.

• Part 3 is about executive roles (Director and above). This is what we are going to talk about today.

The interviews in executive roles focus mainly on product and corporate strategy, people hiring and management, setting up a strong culture, and gravitas.

Let's understand them bit by bit! I have been a part of the hiring committee for such roles in startups but not in large companies. I have also made many of these mistakes myself.

So, the blog post is written from that perspective. However, since the skills for such roles are similar in both startups and large companies, the requirements can be assumed to be more or less the same. Also note that rather than focus on mistakes, I have kept the focus on why such a skill is essential, what it means, and then cover one/two key mistakes.

Product and Corp Strategy

We have already discussed the key mistakes people make in the previous post. From the post,

"Corporate strategy is different than product strategy questions. Corporate strategy questions focus on choices that improve the competitive positioning of the corporation as a whole. In contrast, product strategy focuses on choices you must make in a particular product line to meet BU objectives."

At an executive level, you need to be able to not only understand strategy but also define it for the products you are managing. The steepness of requirement changes depending on whether you are managing a product line (as Dir/VP) or a suite of products (as CPO). The higher up you go, the more the focus becomes on corporate strategy.

People hiring

Identifying and hiring strong product leads/managers is a good way of making everything else easier for the organization down the line. It is also the hardest to learn.

That is why the ability to hire and build a strong team is a key consideration in such roles. People want to know how fast you can build a new team, how well-connected you are to bring people on board, etc.

When it comes to people hiring, one of my favorite phrases is — "You're not a "leader" if no one is following you." Remember this when you get to the next interview in a leadership role. Have the stories from the past that build the conviction that you can hire and build a team.

And no matter where you are today on the PM career ladder, start building your tribe. It will come in handy when you get into leadership :)

People management

As you graduate from a lead PM role to an executive role, a steep change happens in the headcount of the team you are managing.

Earlier, you were managing a few PMs in the lead PM role, and suddenly, you started managing lead PMs who are managers themselves. Managing PM managers is a very different skill set than managing IC PMs.

It's really a conundrum. You need to trust your team to execute the plan, but at the same time, you should have the ability to foresee the problems that might affect the plan. You should also be able to spot an issue with team dynamics quickly.

Further, the lead PMs you are managing are better than you in many aspects of product management. I, myself, had a natural struggle with orderliness because I tend to evaluate bigger and better ideas for higher impact continuously. I worked with PMs who were very strong in this suite. Managing people who are mature and better than you in some regard is a different ballgame altogether.

A common mistake that people make when interviewing for such leadership roles is presenting themselves as 'know-it-all' and pitching the story of how their excellence will help them manage excellent people. While it has some truth, it isn't a convincing story. Different things motivate different leaders. You need to show maturity that you can manage people differently and better than yourself. Authority isn't useful here; building meaningful relationships is.

Setting a strong culture

I resonate deeply with Brian Chesky when it comes to culture,

I define culture as a shared way of doing things. That's it. There are not necessarily good and bad cultures. There are weak and strong cultures. What I consider to be a bad culture, others might think it's good. It's just the way they do things. I wanted to have a strong culture where everyone was on a mission.

The job of the leadership is to create a strong culture around whatever helps the company grow. Google has a people-first culture, whereas Airbnb, Apple, and Meta have a product-first culture. There are also business-first cultures like Amazon.

All the companies have done exceedingly well by setting strong cultures.

To be a good product leader, you need to think deeply about what sort of culture needs to be cultivated in a company. When asked in an interview setting, you should be able to put out your beliefs with convincing case studies. Refrain from stating that one culture is superior to others.

Gravitas

Having gravitas is essential in leadership roles. If you do everything right that we mentioned above and fail around this — you will still fail the interview.

From the HBR,

Having gravitas at work means you are taken seriously, your contributions are considered important, and you are trusted and respected. Gravitas increases your ability to persuade and influence and is likely to fuel the extent to which you rise in an organization. The organization also benefits: You're more likely to add value if your voice is taken seriously.

While some learn this behavior based on their background and nature intuitively, others have to learn it as they move to higher roles. If you are in the latter category, start working on it as soon as you become a lead PM.

It will take a lot of practice and introspection to get good at,

• being composed in stressful situations,

• knowing how to respond to a crisis,

• knowing how to respond to a personal criticism,

• understanding where to draw lines..

But start reading around it, and start gauging yourself.

That would be all for this topic!

I would be conducting a Zoom event on Building a Roadmap to Your Next Role

You can check the event here

Besides this, a lot is happening at pmcurve.com.

• We have the Growth Playground launch scheduled on 6th Jan '23. Do attend it since growth is relevant for all PMs, and this would be an interesting learning product.

• We have a Product Sense cohort set to launch in February.

• We have a growth for PM assessment coming soon!

That would be all for this post.

That would be all for this post.

Regards,

Deepak

Subscribe now

I recently conducted a session on ChatGPT and Large Language Model for Product Managers. Around 250+ folks attended it live, and the response was pretty great too. It was a 2-hour long session with ~50 slides covering the following:

Sharing the video and slides, along with some additional resources for you to learn around it.

Before we move forward, I wanted to make a quick announcement. 15th July is the last date to apply for pmcurve Product Growth course, relevant for existing PMs and Founders. Here is the course page from where you can apply. Here are the reviews of the course from 1st/2nd cohort.

Product Growth Course

Recorded Session and Slides

Here is the recorded session from today around product sense.

You can find the slides used in the session here.

Additional Resources

1. To build a basic knowledge of language models, you can read this post I wrote a while back

   The Growth Catalyst Newsletter

   Understanding Large Language Models - The Force Behind chatGPT

   Good Morning, The world is changing rapidly when it comes to AI. And we have to update our understanding. Around a year back, I published the book Tech Simplified for PMs and Entrepreneurs, and loved the response from the community. Writing tech in simple language helps people without a coding background build knowledge to do their jobs better…

   Read more

   3 years ago · 16 likes · 2 comments · Deepak Singh, pmcurve.com

2. To go in detail around plugins, you can read this one

   The Growth Catalyst Newsletter

   ChatGPT for Product Managers (PMs)

   👋 Hey, I am Deepak and welcome to another edition of my newsletter. I deep dive into topics around building products and driving growth. I started writing about ChatGPT and LLMs for Product Managers recently. In the first post on this topic, we discussed…

   Read more

   3 years ago · 24 likes · 1 comment · Deepak Singh, pmcurve.com

3. Detailed post on overview and applications of large language models

   Leigh Marie’s Newsletter

   Overview & Applications of Large Language Models (LLMs)

   Background A few months ago, as I was calling my Uber back to San Francisco after a day of meetings in South Bay, routine small talk led to a lightbulb moment. I was just finishing up a meeting with an early-stage infrastructure startup founder, who is also one of the strongest engineers I’ve ever met. “The team…

   Read more

   4 years ago · 38 likes · 4 comments · Leigh Marie Braswell

4. A good post by Sarah Tavel on moats in the era of AI— https://sarahtavel.medium.com/how-to-escape-competition-building-enduring-application-level-value-with-llms-45fca5fe49f3

5. This one talks about human brain and current AI models

   The Algorithmic Bridge

   Why AI Is Doomed Without Neuroscience

   One question at the core of AI has remained unanswered for 70 years—without giving any signs it’ll be resolved anytime soon: “How much should artificial general intelligence (AGI) resemble the human brain?” We know a lot more about the human brain than we did half a century ago. Yet, its deepest mysteries …

   Read more

   4 years ago · 10 likes · 5 comments · Alberto Romero

6. Risks of AI

   The Road to AI We Can Trust

   AI risk ≠ AGI risk

   Is AI going to kill us all? I don’t know, and you don’t either. But Geoff Hinton has started to worry, and so have I. I’d heard about Hinton’s concerns through the grapevine last week, and he acknowledged them publicly yesterday. Amplifying his concerns, I posed a thought experiment…

   Read more

   3 years ago · 105 likes · 54 comments · Gary Marcus

7. And finally Yann LeCun (Chief AI Scientist at Meta) on why artificial intelligence will not dominate humanity, why no economists believe all jobs will be replaced by AI, why the size of models matters less and less & why open models beat closed models — https://www.thetwentyminutevc.com/yann-lecun/

Hope you find this useful.

Thanks,
Deepak

I started writing about ChatGPT and LLMs for Product Managers recently. In the first post on this topic, we discussed Understanding Large Language Models - The Force Behind chatGPT. In this post, we are going to discuss the what PMs need to do when it comes to building products with LLMs.

For the new ones here, do check out the popular posts that I have written recently if you haven’t

1. Assessing the effectiveness of PMs

2. Product metrics: to measure or not to measure

3. Necessary Skills to Enter PM Roles/ Crack PM Interviews in India

4. Hidden Layers in Product Management

5. How to Crack the Technical Round of PM Interviews

9,000+ smart, curious folks have subscribed to the growth catalyst newsletter so far. To receive the newsletter weekly in your email, consider subscribing 👇

Subscribe now

Let’s dive in the topic now!

If you are a product manager, you might be falling in one of the three categories right now:

1. You are building a product like Jasper.ai, which used large language models (LLMs) - the tech behind ChatGPT. You are aware of the advantages and disadvantages of this new tech on the block.

2. You are working in a product team where leadership wants to use GPT, and has given the mandate for people to come up with ideas. Some have even set up a small task force to figure out where all they can benefit from it.

3. You are working in a product team which doesn’t believe GPT has immediate benefits for them. They have bigger things to worry about. You, on the other hand, still want to learn and possibly surprise the team by showing a good use-case.

If you are in the first category, there is a very good chance the post doesn’t have much to offer you. You may continue reading this post for fun. If you are in second/third category, this post will help.

For the sake of simplicity, we would focus on ChatGPT. Usually, what’s true for ChatGPT for Product Managers would be true for others like Bard as well. Let’s get started.

There is just too much noise around GPT. The goal, then, should be not to cover as much ground as possible. It should be separating signal from the noise.

The best place to start is to list down where all GPT helps/affects PMs. There are two ways in which GPT affects PMs:

1. Improving their productivity by answering questions, writing emails, summarising, etc. We will not spent much time here since there is good literature around it on the Internet. We will do brief intro and share few good resources here.

2. Using ChatGPT and underlying LLM (GPT) to build products. We will spend majority of time here debating which methods are good for building products using this new technology.

Let’s cover productivity in brief first.

Productivity of a PM

Understanding how ChatGPT is helpful for Product Managers when it comes to productivity is quite useful. Productivity improvement would happen by learning how to write good prompts so that you can get relevant information from ChatGPT. It is also known as prompt engineering.

Here is one example of what OpenAI tells us on how to write better prompts.

Prompt engineering is useful to learn for everyone, not just PMs. But you don’t need to spend a lot of time on it. Just like Google search, most smart people would come to write good prompts themselves through trial and error.

PM productivity isn’t what we are focussed on in this post, and maybe we will dedicate a future post to it. For now, if you want a good place to start as a PM, here are some good free resources:

1. https://bootcamp.uxdesign.cc/how-to-use-chatgpt-in-product-management-f96d8ac5ee6f

2. https://twitter.com/lennysan/status/1640385666134704128

3. https://aatir.substack.com/api/v1/file/aac4ca34-1e25-4e41-9b64-5448e447fe5c.pdf

4. https://levelup.gitconnected.com/chatgpt-guide-for-product-managers-top-20-most-important-prompts-a2c48dcc63c6

We are onto the second point. To build products using GPT, you need to build a good thesis/understanding of what’s possible and what’s not possible with this new technology.

There are many ways to go about building this thesis:

• Reading papers behind the LLMs to understand them deeply

• Blog and social media posts to understand its potential and risks

• Going through the trusted sources like what experts have to say

• Looking at plugins/current applications

Let’s get into details of all 4, and see where they help.

Research Papers

Some PMs want to go to the core and start with the research paper by Google in 2017 that started it all — Attention is All You Need. The paper introduced transformers and how they created a way better NLP model as compared to anything that had come before.

Unless you are building a large language model yourself, I would not advice you to go through research paper for a couple of reasons:

1. If you don’t have a data science background or built AI products in the past, it would take a lot of time and energy to understand these highly technical papers. To understand the whole thing, you would also need to learn about the historical evolution of NLP and how we reached here.

2. Even after understanding the papers, you would find it hard to map all applications. The tech behind LLMs got built first, and now we are off to figuring out applications. We keep discovering new advantages and limitations every passing day.

So starting to read research papers isn’t the right way to go about it unless you are building a LLM like GPT yourself. What about reading about what blogs and social media posts have to say what it can and can’t do?

Blogs and Social Media Posts

There are a lot of people who don’t have background in AI/ML and still writing about LLMs heavily on social media and blogs. These people are just repurposing the content for likes and followers. And because they are writing it to garner likes on social media platforms, they end up writing sensational things, most of which may not be true in its entirety.

That makes social media and blogs a very unreliable source of information — the noise is way more than the signal. Be sure to fact check everything that you read there, unless it’s coming from an PM/engineer/researcher in the AI/ML field.

What about going after what top experts have to say?

Expert Take

Experts are focussed on the high level discussions like future risks and rewards of Generative AI. The downsides of generative AI (LLMs) going wrong could be very high, so it’s a legitimate concern.

Further, these experts are divided on what the future of LLMs is. There is great debate happening whether LLMs would lead to human-level intelligence, aka general AI. Some people including OpenAI CEO Sam Altman believe it’s possible. Others don’t believe so. Here is a thread worth reading from the chief data scientist of Meta. This is what he says,

“On the highway towards Human-Level AI, Large Language Model is an off-ramp. To clarify: LLMs that auto-regressively & reactively predict the next word are an off-ramp. They can neither plan nor reason.”

In my opinion, builders should be focussed on what’s possible to start with. You can read about future risks and rewards, but as of now, there is little evidence to end this debate. So start focussing on what’s already happening, which brings us to current plugins/applications.

Starting with current plugins/applications allows you to quickly see how different products are using LLMs. A bunch of smart founders and PMs are already ahead of the curve, and they have built products over GPT. You need to learn about these use-cases and limitations to get upto speed. This is what we should be most excited about.

Let’s start with plugins of ChatGPT that product managers should understand.

Tech knowledge is essential to perform well in a PM job. If you are a PM who doesn’t come from a software background, you can checkout my book ‘Tech Simplified for PMs and Entrepreneurs’ which has been immensely useful (readers’ word, not mine) in getting them to understand tech well :) 250+ people have rated it 4.5+ on Amazon.

Tech Simplified for PMs

How Plugins Work

OpenAI plugins connect ChatGPT to third-party applications like Instacart, Expedia, etc. Understanding how the plugins work will help you decide whether you can use ChatGPT for your product or not.

If you have been following the news, you may have heard about ChatGPT plugins from popular companies like Instacart, Expedia, etc. Here is a list of ChatGPT plugins in Beta.

While using ChatGPT, limited users would be able to see and interact with these plugins. For example, when you ask the integral of x^2cos (2x), the ChatGPT used Wolform plugin, and answers it pretty accurately. You should note that the user is typing the query in ChatGPT and not on Wolform website.

So how does the plugin shown above work? We have to understand the flow to see where all can we apply in our product. Here is an explanation step by step:

1. When asked to perform computations, ChatGPT interprets the user’s question and formulates it as a query.

2. The query is can now hand it off to Wolfram Alpha instead of attempting to generate the answer by itself.

3. Wolfrom computes and passes the response to ChatGPT.

4. ChatGPT structures its response based on the response.

ChatGPT is great at understanding user natural language, and Wolfrom is great a performing computations. Both sides use their strengths in this arrangement to create an amazing user experience.

Current Plugins’ Use-cases

To build a holistic understanding, let’s look at few of the plugins one-by-one:

1. Instacart: Instacart is a grocery shopping app in the US. The Instacart ChatGPT plugin helps people figure out what they would need to make a particular meal, create an instant shopping list based on ingredient needed, and get ingredients delivered to their door so they can start cooking. Like Wolform, user interacts with this plugin on ChatGPT website, and is redirected to Instacart for modifying the cart and payment.

2. Expedia: Expedia is a travel platform. The users can get recommendations on places to go, where to stay, how to get around, and what to see and do. The plugin automatically saves hotels discussed in the conversation, and adds on flights, cars or activities. The payment for the booking still happens on the main Expedia website.

3. Opentable: Opentable does restaurant reservations. The plugin will help with open-ended questions such as — “I have a date I want to impress. What’s a restaurant with oysters and great cocktails in the Upper West Side NY?”. These sort of questions are hard to search on Opentable website.

4. Zapier: You can automate tasks that require data to flow from one place to another using integrations in Zapier. And that is possible now from within ChatGPT's interface. It saves you time and the hassle. Here is a sample use case.

5. Fiscalnotes: Fiscalnotes shares information about global policy and governance to various companies. ChatGPT can quickly find the right information users are looking for, thus saving time.

We can look at more plugins, but we have gotten a broad idea of where ChatGPT is helping users. The products seems to use the plugins for the following use-cases:

1. Processing natural language queries — One thing that GPT does very well is understand the natural language query. When a user provides a query for real-time information, ChatGPT can understand what the user is looking for and retrieve that information pretty quickly. This is happening across apps in beta.

2. Maintaining context for conversations just as humans do — GPT can retain the information from past when the user asks the next question. This wasn’t possible in regular search on websites and apps, and that is where ChatGPT shines. Look at this example below - ‘what’s the salary’ is a pretty generic question. ChatGPT is able to retain context from the previous question like humans.

   

3. Searching and filtering to show the most relevant results from structured database — The plugin isn’t accessing the data itself, it is hitting search endpoint with keywords and filters. The particular app will process the keywords and filters to send a response with relevant results.

   
   Klarna, OpenTable, Kayak, Expedia — all are using the search and filter to produce relevant results.

4. Automating tasks for users — This is where things get interesting. For Instacart, the input provided by the user is a meal name. ChatGPT is fetching the recipes, and ingredients. Further, it is taking into account the portions of ingredients, and creating an automated shopping list.

   For a user, creating a shopping list is a pretty mundane task. It’s much easier to have a starting shopping list of 15 items, and remove/modify it. The experience of grocery shopping actually improves significantly. The same can be said about the Zapier integration.

5. Finding relevant information from unstructured data — This is yet another example where experience becomes significantly better. When applied to Fiscalnotes, which would have thousands of policy documents lying around, GPT can bring the relevant data pretty quickly. Searchability in unstructured data is a big problem in policy research, market research, law, etc. ChatGPT would play a massive role in reducing the effort required to search and summarise in these areas.

   Other applications of this capability for product companies is in searching knowledge base and answering support queries from the customer. Stripe has already started using it to improve the experience of reading support documents. They have heavy documentation for developer APIs. When ChatGPT sits on the top of documentation, it can help developers finding the right piece of code/information pretty quickly.

6. Translation: Speak is a language learning app, and one of the current plugins. Speak plugin provides a tailored language learning experience whenever a user is looking for a translation or explanation across languages.

7. Solving questions and explaining them — Wolfram can be used for computation and finding answers to math/science questions. What’s better, ChatGPT can write simpler explanations because of its NLP capabilities. So if a student isn’t able to understand the answer, they can request ChatGPT to explain it in simpler language.

Your Product Plugins

No matter which product you are working on, one or more of such capabilities can be put to use. Providing support post-sales by simplifying documentation, or building a sales bot are few obvious use-cases for all transactional and productivity apps out there.

Search and discovery experiences are also quite horizontal in nature across apps that ChatGPT seems to improve.

One thing that can stop you from building a ChatGPT plugin at this point is whether your customers are already on ChatGPT or not. If they are already spending time in ChatGPT, building a plugin can give you an advantage over a competitor who is not open to the idea. What’s better, it can also become a source of new user acquisition given that more than 100 million users are active on ChatGPT. You have to evaluate for yourself on whether a plugin makes sense or not. It will come down to how much the experience improves, and whether ChatGPT has your current/future customers on its platform.

This post is already quite long. I plan to write about ‘understanding applications built on the top of LLMs’ in the next post.

Meanwhile, let me know how you found this post by liking or commenting.

Have a good day!

Deepak

The world is changing rapidly when it comes to AI. And we have to update our understanding.

Around a year back, I published the book Tech Simplified for PMs and Entrepreneurs, and loved the response from the community. Writing tech in simple language helps people without a coding background build knowledge to do their jobs better.

So I thought of starting this newsletter series of Tech Simplified, where I write about newer topics in Tech. Most recently, the AI fever has caught the world, specifically chatGPT. So there can’t be a better topic to start with. Before we start, an announcement!

The applications for the 2nd cohort of Product-Led Growth are now open! The first cohort had 60 members (limited seats), and all of them are existing PMs and Founders. It’s highly recommended for those who want to learn advanced skills in Product Growth with a super-smart peer group.

Why is this course different?

1st cohort testimonials sessions got over yesterday and testimonials just started flowing in. Here are few testimonials from the 1st cohort to check out 🤓

Read on for more context.

First, most courses around growth are focussed on marketing. This one is a product-first course. PMs and Founders need to have product-led growth because spending money to grow isn’t always a viable option.

Second, courses usually focus on learning but not creating immediate impact in your current job. The content, tools, and templates provided in this course are such that majority of PMs and founders from the 1st cohort built their own product growth plan while getting assistance from me. Building your own product growth plan a part of curriculum, and that’s proof of value right there :)

Third, every session has 20-30 objective questions around frameworks and case-studies that help you evaluate whether you really understood the concepts and can apply it.

Add to that — capstone project, case studies, interview prep for FAANG, community of super-smart folks are other benefits 🔥

You can know more about the course at https://www.pmcurve.com/

Check the Course

I would be doing a weekly shortlist of applications for this cohort and there are limited seats. The first shortlist will go out today evening, so apply today if you are considering it.

It would hard to cover chatGPT in a single post, and that’s why I have taken the onus of creating a 60-80 page guide around it which I am planning to publish by April. In this post, we will cover the key to the power of chatGPT — Large Language Models (LLMs)

Models

To understand what a large language model is, we first have to understand what a model is.

You may have heard of algorithms if you have done basics of computer science. An algorithm is a set of rules that the computer follows to solve a problem. For example, we can have a set of rules to determine whether a given number n is even or odd, aka algorithm. You can see that algorithm below.

Understanding algorithms is prerequisite to understanding models. Many people confuse models with algorithms. Let’s take a machine learning algorithm.

In machine learning, every enthusiast starts with linear regression algorithm. It defines the relationship between one independent variable (x) and one dependent variable (y) using linear equation (= a straight line). It is well known that equation for a line is y = mx + c, which can also be written as y=w2+w1x. The mathematical rule that we just defined is an algorithm. Let’s talk about models now.

A model is when you determine w1 and w2 in the algorithm based on data you have. Suppose we were trying to find the correlation between the # of hours spent studying and marks obtained on the test. In this case, say we had these data points.

0 hours — 0 marks

3 hours — 33 marks

9 hours — 99 marks

We can put the first datapoint (0 hours, 0 marks) in the equation and see that w2=0

0=w2+w1*0 
w2 = 0

Let’s put another datapoint (3,33)

33=0+w1*3
w1 = 33/3 = 11

Now, let’s see if y=11x holds true for the third datapoint (9,99)

y=11*9 = 99

So now we have gotten a model, y=11x using which we can predict things. Answer me this

Using y=11x, we can predict that a person studying 5 hours would get 11*5 = 55 marks.

For another test that is tougher, the data might be different. Maybe you take 5 hours to secure only 20 marks there. The model for that test would also be different.

Few definitions now that we understand what a model is:

— w1 and w2, whose value is estimated using data, are called parameters of the model

— Using data to determine various parameters of the model is called training the model

— Using new data that model wasn’t trained on to measure how model performs is called testing the model

So to create a model, we need an algorithm as the base and data to train the model. We also test the model before we make it live to ensure it performs well.

Another important point to note here is we can employ a different algorithm on the above data to create a different model. So, we can generate a new model with the same algorithm with different data , or a different model from the same data with a different algorithm.

Now that you have understood models, let’s jump to language models.

Langauge Models

Have you noticed the ‘Smart Compose’ feature in Gmail that gives auto-suggestions to complete sentences while writing an email? How does the ‘Smart Compose’ feature works? In the simplest terms, it looks at how we normally write our emails, and provides suggestions.

Smart Compose is one of the various use-cases of language models. A language model is a model trained for language-specific use cases like writing an email, translation, chat replies, etc. Language models are useful for a variety of language-related problems

1. Speech recognition — ability of a machine to identify words spoken aloud and convert them into readable text

2. Machine translation — translate content between languages without the involvement of human linguists

3. Natural language generation — generating text that appears to be written by a human, without the need for a human to actually write it

4. Part-of-speech tagging — categorising words in a text (corpus) in correspondence with a particular part of speech

5. OCR — converts an image of text into text format

6. Handwriting recognition and many more

Just like the model above is built on an algorithm, language models are also built on different algorithms. Some of the popular algorithms for language models (also relevant to chatGPT) are n-grams and deep neural networks. The data to train these models comes in the form of articles, books, emails, etc. written by humans in the past.

Now that we understand language models, let’s move on to what’s large in large language models.

Large Langauge Models

The goal of the LLMs is to predict the next word based on the words that came before it. To do this, large language models use deep neural network algorithms and are trained on vast amount of data (billions of words) to learn patterns within the language.

We add ‘large’ to LLMs not (just) because they are trained on vast quantities of data, but because of large number of parameters these models have. Look at the number of parameters various LLMs have. Don’t get daunted by model names :)

BERT-large has 340 million parameters whereas GPT-3 has over 175 billion parameters. GPT-4, which launched this week only, didn’t reveal the number of parameters but you can be sure that it will be much higher than GPT-3. And because of these large number of parameters, we need vast quantities of data to train. You may ask how vast are we talking about?

GPT-3 was trained on 499 billion tokens. A token is similar to a word (we won’t go into details of why we are calling them token here). That translates to roughly 100 billion sentences, assuming 5 tokens in a sentence.

In order to train LLMs with billions of parameters, massive amounts of data and computational power are required. OpenAI, for example, trained GPT-3 using a combination of 45 terabytes of text data and 3,175 NVIDIA V100 GPUs running in parallel. The training process took several weeks to complete.

What about the algorithms? Deep neural networks would need another post to get through so I would leave it here by saying that these algorithms need to have billions of parameters like w1, w2, …

Bringing it All Together

Looking at how LLMs are built explains why they are so powerful at predicting the next word in a sentence, which is the foundation of tools like chatGPT. While writing an answer, chatGPT is essentially writing (or better predicting) one word at a time based on the words that it has generated before. The answer generation in chatGPT interface actually mimics the ‘one word at a time’.

So what’s the application of whatever we just covered here in this section? For one thing, you can take LLMs like GPT as a base model and tune it by training it on a small amount of your product specific data. That way, you can create a model specific to your product.

Look at the ways this has been helpful to products out there, and you might get an idea to do so for your own product.

1. Salesforce Announces Einstein GPT, the World’s First Generative AI for CRM

2. Snap rolled out My AI a new chatbot running the latest version of OpenAI's GPT technology that they have customized for Snapchat

3. Instacart Joins ChatGPT Frenzy, Adding Chatbot To Grocery Shopping App

4. Quizlet Introducing Q-Chat, the world’s first AI tutor built with OpenAI’s ChatGPT

In future posts, we would talk about deep learning, transformers, self-attention, etc. which are relevant to building more knowledge around LLMs. But for now, this would be it for this week.

Hope you find a cool and useful application of LLMs in your product 🚀

Another update I wanted to share is I would be doing a free zoom event around ‘Building a Product Sense’ on 25th April. 500+ folks are already registered for it. You can register for the event on LinkedIn.

Check the Product Sense Event

Have a good day!

Regards,

Deepak