Building has never been faster and cheaper than it is today. The product management skills that count are very different from the product skills that mattered even two years ago. I'm learning these new skills building RepX, an AI sales agent from Storylane.
RepX is an AI sales agent that answers questions from visitors trying to make software purchases and qualifies them by asking discovery questions like a sales person would. Most chatbots from the pre-AI era handled customer support. They would answer questions and stop there. RepX, though, is for prospects visiting your website - they might be deep in their evaluation process or may have just started to look for something to purchase. RepX needs to offer value when prospects ask questions, be interesting enough for them to continue conversations, and finally persuade them to book meetings.
How we're building RepX with the help of the world's intelligence available on tap - is very different from how we would have built it before the AI era. I believe the skills I've listed below and what AI makes possible are crucial if you're building products for this era. Let's dive in and see everything we're learning and unlearning.
Ben Thompson wrote in his article - AI homework:
Moreover, instead of futilely demanding that students write essays themselves, teachers insist on AI. Here’s the thing, though: the system will frequently give the wrong answers (and not just on accident — wrong answers will be often pushed out on purpose); the real skill in the homework assignment will be in verifying the answers the system churns out — learning how to be a verifier and an editor, instead of a regurgitator.
This has definitely become true about product specs. Product management used to mean taking the time to research, understand competition, understand the problem your team needs to solve and reach a point of clarity about the solution you want your team to build. Before the AI era, this process took quite some time. But now, this part of work is definitely going through a fair bit of compression - you're definitely able to generate a lot of ideas and specs along with prototypes quickly.
If you're spending weeks doing this now, you're wasting time atrociously. PMs should just be doing the work of the editor going forward - trimming, removing irrelevant ideas, etc. and taking a spec and prototype to a point where it's good-to-go for development in days if not hours.
What AI has also made easy is analyzing large amounts of data to test any hypotheses you might have. For instance, given that generating responses takes time and website visitors don't have any patience, we wanted to launch filler phrases (like "got it", "makes sense" etc.) that RepX agents would speak out while the actual answer is being generated. The idea was to reduce perceived latency in voice responses from the agent.
We analyzed a large volume of conversations our own website visitors had with RepX, and got Claude to figure out what kind of phrases sounded the most natural against real sentences in transcripts. Scouring through all this data and figuring out which phrases sounded natural would have taken an inordinately long amount of time earlier, and might have even required me to allocate engineering bandwidth. Something I wouldn't have considered judicious back then.
But now, doing analysis like this on a large amount of data and testing assumptions is way cheaper and faster, with models in the picture. You can test a lot of your assumptions for just a few dollars in token cost, like never before. You can also setup scheduled tasks on Claude to continuously go through large amounts of data to spot trends, anomalies that feed into your product improvements.
Parker Conrad, CEO of Rippling, said this best. All B2B software, at scale, converge towards these things:
This is true. This is all that software used to be:
Before AI, software turned the mess of the real world into structured data we could run operations on. We configured the workflows, software ran them, and the tools we built helped us analyze raw data and resulting insights.
A lot of times, when there was a huge amount of unstructured information in the picture, it required human ingenuity to separate the insights from the chaff. Humans were still needed to do a lot of hard parts.
With AI, it's now possible to solve net new use-cases that were simply unsolvable before. We've already seen emerging examples of solutions being built in legal. Deep work in product management now lies in identifying these use cases, and then deploying models to solve for those usecases.
Domain knowledge alone isn't sufficient enough to capitalize on this fact - you need to be able to discern where the human labour spend goes today, which of those are reasonably repacable with AI already - then, it becomes honing current technology to create outcomes buyers desire.
Code is cheap. Prompts are now your product's intellectual property.
Through nudges, a capability in RepX, we display short messages that get prospects to initiate conversations with the agent, while they're browsing through the website. Nudges that show up are based on the history of pages visited by prospects and their previous conversations.
Someone landing on a case study page might be looking for what other customers got out of the product or a proof point they can share with their team members. Someone on the pricing page is usually comparing plans, or wants to talk to a person before deciding.
The nudges that we want RepX to display have one job - they should feel like we read the mind of a prospect landing on a page, address their concerns or create some sort of a FOMO that gets them to start talking with the RepX agent on the website.
I had Opus write the first version of the prompt that generates nudges, but making these nudges click-worthy required a fair bit of prompt engineering. Our early nudges on pricing pages generated messages about discounts (which we weren't supposed to offer without the customer's approval). There were also several situations where the nudges ended up being based on a niche concept from within the page which a visitor wouldn't care about, making them very uninteresting.
After several iterations on the prompt and evals, we came up with a prompt that
This sounds easy but it requires
I think this is where the hard work sits now. Considering intelligence is available on tap, the job becomes two things:
The latter takes a fair bit of time - time that you need to spend prototyping, writing prompts that offer directives to the model on how you want a problem solved.
One of the key things that we learnt is that fast responses matter in RepX. Website visitors have no patience. They'll move on to the next tab if they don't get an answer in two seconds. RepX needs to pull up the right resources (images and demos), run the answer against inbuilt guardrails for safety checks, and then deliver an answer within this span of time.
The only models we started preferring were models with low latency that gave our users fairly good responses. However, when we used the same models to create prompts for some of our use cases, the prompts led to the agent behaving in weird ways with lots of errors. We use Opus 5.5 to write our prompts now, and run the agent on Gemini flash models. We’re also planning to use the decisions API to power some of our use cases.
The table below is a fairly high-level example of how I think about these two models, but I think having a deeper sense of what models to deploy for use cases and how to leverage their superpowers while working around their limitations is a core AI PM skill right now.
RepX agents run on low-latency reasoning modes because response time matters a lot for us. When a reply takes longer than 3 seconds, people drop off more often than not. So when I use Opus to write a prompt, I tell it which model the agent will actually run on. Opus looks at that model's spec, figures out where it's strong and where it slips, and writes the prompt around both.
By training on these prompts and noticing actual customer use cases where the models are failing, our product managers have also built a mental profile of which models work great in certain situations. Our prompts and harnesses are written to work around their limitations and capitalize on their strengths. I believe this is a true AI PM skill. You don't acquire some of this skill by reading about models, but a lot of it by building solutions for real problems.
At any given time you're going to have hundreds or thousands of customers. Over time, you end up engineering prompts and building AI harnesses that solve different kinds of problems for these customers.
On RepX, for example, we have a set up experience that generates a system prompt for each Rex agent based on how a marketing administrator sets one up. We have multiple versions of templates from which these prompts are generated when an agent is set up, and we’re constantly upgrading these templates. Every time we do this, we run evals based on conversation scenarios to be able to see if template updates we’re making are going to have a positive or an adverse impact on our customers. Without these evals, we'd be clueless if our updates are improving the overall product for our customers or if they're actually making it worse.
Building a working eval system that
is squarely in the PM's ownership territory in the new era. This is the new functional testing - in a world of non-deterministic models, you need to be able to run evals to see how well they're performing against different use cases when you tweak your prompts or harnesses.
Models are smarter than humans. We only exist to direct them. If you point them to a large amount of data and feedback, and ask them how a system needs to be improved, they typically do a fantastic job of figuring out new issues, anomalies and risk patterns - a whole class of improvements that you can make. They’re still bad at coming up with super creative ideas for new product capabilities, but the best reasoning models in the world are great at finding problems and improvement ideas in your existing data.
Every product org should have a dashboard that shows a draft product specs or prototypes, or even better, code with a PR drafted for you or your team to review and push out. The tool should rinse and repeat the same activity every week, and over time you’ll see a lot of gaps getting plugged with very little human intervention. I believe building a self-recursive system like this for product builders to own should be within the product team’s ambit.
Given the nature of how work is changing, I certainly believe product management, engineering and design orgs need to reorganize around two kinds of work: prototyping and scaling.
Prototypers are closer to customers, creative, come up with ideas, know how to build a v1 of the idea quickly leveraging tools in the market, and know how to write evals. Scalers don’t have to be necessarily close to customers, but should be comfortable working with models, stay on top of architecture patterns and own the implementation of quality solutions.
Broadly, this is how you can think about reorganizing into two major divisions:
How are you staying on top of the changing world, tweaking your work patterns and reorganizing your teams for this brave new world? Would love to hear your thoughts out.
.avif)
.avif)
Into the Future
.avif)
.svg){kind=icon}
.png)
.svg)
