A Brief Guide to Knowledge and Tools That AI PMs Should Master
With the LLMs changing the landscape of AI, the role of AI Product Manager has evolved over the past couple of years. The demand for AI product managers is all time high, and if there was a good time for one to enter this arena, now is that time.
Andrew Ng wrote about AI product management in one of his recent essays,
The demand for good AI Product Managers will be huge. In addition to growing AI Product Management as a discipline, perhaps some engineers will also end up doing more product management work.
The Gap
The skills of an AI product manager are quite different from a traditional PM. While a traditional PM role is focussed on user research, roadmap planning, and success metrics, the AI PM roles requires additional skills on top of these traditional PM skills.
These additional skills is what we are going to cover in today’s post. We will also talk about tools that are relevant for AI PMs. These skills and tools are directly borrowed from the course I am building around AI product management starting next month.
Skills and Tools of an AI PM
To understand the skills needed for an AI PM, we need to understand the process of building AI products, and how it’s different from building usual software products. On a broad level, the lifecycle can still look like the usual cycle of plan-build-deploy, but the nuances are pretty different for each task.
Let’s look at various skills and tools.
Technical Proficiency
Before diving into AI-specific skills, you need to understand how complex technical systems work together. Remember that you aren’t just talking to front-end and back-end developers anymore. You are also talking to data scientists, AI engineers, and data engineers, etc. You would be able to effectively collaborate only if you are technically proficient in system design, RCA, etc.
The next thing you require is foundational knowledge of AI/ML. This includes different approaches in machine learning like predictive vs generative, supervised vs unsupervised, the subclasses like deep learning, etc. You also need to have a decent understanding of limitations of these techniques.
The technical proficiency also helps you read papers in the field of AI. The field is evolving fast, and those who can stay updated by reading these papers have a distinct edge.
Discovery and Prioritisation of AI Use-cases
PMs are good at discovering general problems of a user, but determining whether AI is the right solution requires specialised knowledge. As we discussed in the last section, having the foundational knowledge of AI/ML can be useful. Beyond that you would also need to assess data availability, ROI calculation for AI projects, broad implementation details, timelines, capabilities needed within the team to finalise what you need to build.
The prioritisation will also require impact calculation of such projects, and benchmarking of impact from similar projects across industries.
Some of the tools that you could find useful in this stage are tokeniser like this one from OpenAI, pricing calculators for compute and storage, AI-specific industry reports like this one from Stanford, etc.
AI Product Design
AI products have specific challenges when it comes to design. For example, the generative AI products may hallucinate and provide misinformation to the end user. This could be pretty risks in some situations if not handled well. For example, a Texas A&M University-Commerce teacher gave his entire class a grade of "Incomplete" because when he asked ChatGPT if the students' final essays were AI-generated, the tool told him they all were, even though detecting such text is outside.
So we need to design interfaces that that handle hallucinations, uncertainty, etc.
We need to understand how surface these features, how to help people write effective prompt, handle errors gracefully, etc.
This area of AI product design is still evolving. Leading companies such as Google, Microsoft, etc. keep publishing stuff as they learn. Here is design.google sharing few tips/concepts — https://design.google/library/people-ai-research
AI Engineering
AI engineering covers designing the right model and architecture for a use-case and deploying it. While you aren’t expected to do it yourself, you need to be actively involved in the entire process. Just as a PM needs to sit through architecture discussion in usual software development, you need to sit through discussions around model selection, architecture, and cost implications of the choices.
Let’s take model selection for an example. To shortlist few model initially, you need to give few relevant prompts to different models and evaluate their responses. A PM can do this job better than most since they have broader context of customer needs, business needs, etc. If a PM isn’t involved in model selection process, they will be limited by the model that team chose. And since they don’t have an understanding of other options, they will fail to see what could have been an optimal choice today or in future.
Some tools and concepts that PMs needs to explore in this stage are Scale Spellbook, RAG implementation using Langchain, Vector databases, multi-modal UX, etc.
AI Agents
Agentic AI is where you don’t need to think and guide with continuous prompts for AI to work for you. An AI agent can take a broad instruction, create an execution plan, and implement it by itself without much human supervision.
An AI PM needs to understand how agents work, and the process of building them.
Understanding tools like AutoGPT, OpenAI Operator, Semantic Kernel, n8n, etc. can be pretty useful in building agents.
Evaluation and Improvement of AI Products
PM needs to evaluate when a particular AI product or feature is working as intended. In software, it’s done using acceptance tests. For AI products, the evaluation process isn’t so simple and we need to design comprehensive evaluation frameworks for the product we are building.
Besides testing the product, evaluation process helps in identifying improvement areas for AI products. The improvement in AI products can be made in various ways - by managing dataset quality and bias, providing human feedback via rating, or by providing sample answers written by humans. A PM needs to work with the AI engineering team to evaluate cost and benefit of these different options. For example, getting sample answers written by humans can be pretty time and money intensive.
Relevant tools in this stage could be OpenAI Evals, Huggingface, Arize AI, etc.
AI Safety
A major part of evaluation process is making AI safe, unbiased, and ethical. Evaluation process also helps in identifying these safety risks.
Relevant tools in this stage could be AI Fairness Checklist from Microsoft, responsible AI papers from leading companies such as Google, Anthropic, etc.
Pricing and Positioning of AI Products
AI products have unique characteristics that significantly impact how we think about their value and pricing structure. For example, when you use a traditional word processor, it works the same way every time. But an AI writing assistant might improve as it learns from user interactions, making it more valuable over time. So how do you price in this additional value and charge your customers when most b2c traditional word processors are free (most)?
Understanding cost structure, pricing models, pricing research strategies, and consumer willingness to pay is key to build pricing strategy for an AI product.
Positioning is crucial for AI products, as it helps the users understand the value proposition of your product. This requires you to understand the value metrics, and use segmentation + differentiation to craft a clear positioning.
Product Growth and Strategy
Lastly, any product without growth in initial stages will not service. Your AI products and features needs to grow, and the growth strategy need to use special features of these products such as novelty, community, etc.
The AI product also needs a long-term strategy to build moat, keeping in mind that the AI landscape is emergent.
This would be all for the post. As I mentioned earlier, the Advanced Tech and AI course is starting next month. The first cohort will only take 40 students, and I plan to be pretty involved (1:1) for this cohort. The first shortlist goes out tomorrow evening. So please apply if the program feels right for you
Thanks,
Deepak