Building Products with Generative AI

Strategic Considerations and Tactical Tips for Success

By: Christine Lee, Senior Director of Product, GoDaddy and WIP Board Member

At the end of the year, I include a short letter with the holiday cards we send to our extended family. Usually my daughter and I collaborate to summarize highlights. Last year, my tween procrastinated so much that I had to write it myself. I joked that in the future, a bot would take care of it. Then ChatGPT launched in November, and it became clear that generative AI is going to change just about everything in everyday life.

This is an exciting time to build as a Product Manager. Much has been written about how the release of ChatGPT was a tipping point that will drive widespread consumer adoption of artificial intelligence¹. In fact, the chatbot from OpenAI reached 100 million users in two months after launch.

This growth is remarkable; in comparison, TikTok took 9 months, and Instagram took 2.5 years to reach this level of monthly active users². Even more extraordinary is how quickly subsequent releases have emerged that are less costly and more powerful. GPT-4 and developer APIs were released in March 2023, less than 4 months later. The ChatGPT API (based on gpt-3.5-turbo model) is 10 times cheaper than the original GPT-3.5 models, at only $.002 per 1000 tokens (about 750 words)³. GPT-4 no longer requires text inputs. As the saying goes, a picture is worth a thousand words, and it’s now possible to generate captions, classifications, and analyses from a single photo⁴.

OpenAI is not the only leader providing AI tools. Given the computing power required to train AI models, all the major cloud infrastructure companies have a stake in generative AI. Microsoft (which is a major investor in OpenAI), Google, and Amazon have released several announcements in early 2023 about how they are developing AI models and applications across search, enterprise solutions, and cloud services. The global generative AI market size is forecasted to reach USD $109.37 billion by 2030, with a CAGR of 35.6% from 2023 to 2030⁵. This is attracting countless B2B and B2C startups. According to CBInsights, 34 AI startups reached unicorn status in 2022⁶ including Hugging Face and Stability AI.

So how can you leverage generative AI in the products you build? Here are five things to consider:

1. Focus on the customer problem
2. Ask the right questions
3. Craft the user experience
4. Consider second order effects (and bad actors)
5. Plan how to scale

These are product management fundamentals. You may not consider yourself an “AI expert” (yet), but you have the core skills to build innovative new applications for generative AI.

1. Focus on customer problem

With all the hype around generative AI, companies are racing to showcase how they are innovating with this new technology. ChatGPT’s release was amazing because it showed that AI can not only return information but also create seemingly original content. As industry pundits prognosticate about the wonders of this technology, I’m reminded of an adage I first heard from Scott Cook when I was working at Intuit: “fall in love with the problem, not the solution.” There is a risk that in their eagerness to create AI-driven solutions, product teams will reprioritize roadmaps in haste. Make sure that you are solving the top priority customer problems. Consider whether generative AI is the best solution for the problem, or the optimal investment for your limited resources. For example, while the costs for Large Language Models (LLM) like ChatGPT have dropped substantially with each new version, it may still be cheaper to use other solutions such as Search.

That being said, there certainly are many, many use cases that can be automated or solved more effectively by leveraging generative AI. In particular, content creation focused on sales, marketing, and customer support are basic areas to improve. Writing is hard or slow for many people, and bots like ChatGPT or Google Bard can now create narratives that are hard to distinguish from human-generated copy. Internal use cases can also be addressed, ranging from generating code to planning customer research. Imagery can be created based on detailed specifications that would be difficult or time-consuming to set up in real life. Finally, automated service interactions can feel more personalized and faster through voice-enablement. These are just a few examples of applications. Almost every human interaction can potentially be automated and enhanced.

The key will be to identify how you can leverage AI and LLMs to create a unique value proposition for customers. In the near future, every product is likely to have some level of automation and AI enablement, so differentiation will require more than the basics. Perhaps your organization has proprietary data that would give your product an advantage in developing new AI models. Another opportunity could be to identify what capabilities would give your organization a distinct advantage in the AI application, such as market share, brand awareness, partnerships, user experience, or channels. Your team might identify a novel way to assemble a variety of AI and non-AI solutions to meet your customer needs. There are so many new frontiers beyond AI, such as Augmented and Virtual Reality, Web3, and the Internet of Things (IoT). Imagine what will be created when these are combined in new ways! Ultimately, the role of a Product Manager is perhaps more important than ever. ChatGPT and other AI tools empower PMs to be more effective in many areas of the job⁷, but good PMs are still required to lead the team to deliver the right solutions for customers.

2. Ask the right questions

Product managers are curious and ask a lot of questions. This makes the learning curve to optimize applications of generative AI less steep. The intersection of generative AI and Natural Language Processing (NLP) means that you can “program” simply by asking the right questions (particularly in English, though other languages are quickly being added⁸). “Prompt Engineering” is a relatively new field that involves “…prompting the AI, training the AI, developing and maintaining a prompt library, and testing, evaluation, and categorization.”⁹ It’s easy to generate content, but the art is in developing the right questions to efficiently produce what your product needs. Some considerations as you develop questions include: tone, brand personality, number of words, and ways to reuse prompts. Where possible, encourage sharing across product teams to broaden the collective understanding of what is and isn’t working.

Iteration is required to develop and refine the questions or prompts used to generate content. You will need to set up a scalable way to review how relevant, consistent, and accurate the content produced will be for your customers. There are industry benchmarks like Pile, LAMBADA, and Big Bench for evaluating the performance of different Large Language Models (LLMs)¹⁰, but for now, you will likely need to recruit humans to examine content outputs for your specific use cases. These reviewers will provide valuable feedback to inform how the prompts should be modified.

For example, if you are using generative AI to develop marketing copy, set up comparisons of the machine- vs. human-created content. Content creation time will be exponentially faster, and the volume of output will require efficient processes for review. Think about ways to create automated feedback loops with content experts and customers. A basic tactic is to create a review system that allows your team to evaluate content in bulk. This can be as simple as generating all the content in a single file so that testers can quickly compare results. Obviously you can also apply automation and machine learning to set up self-sustaining reviews. Crawl, walk, run to optimize the quality of content produced.

3. Craft the User Experience

No matter the quality of the content that AI models can produce, it will only be as good as customers perceive it. How will your product deliver the user experience? The same fundamentals apply – ease of use depends on how well your users can complete key tasks successfully, quickly, and confidently. Experiment to optimize how much should be automated to reduce user friction. Learn what your customers need in the experience to feel successful. Beyond getting the job done, how much control does the user expect to have in the process? This is not always straightforward to determine.

Make sure you are reviewing the end to end experience when evaluating usability. It may be simple and even fun for your customer to enter their own prompts, but this may not hold true after the tenth interaction. Time and skill are required to optimize questions, so it might be a better experience to add guardrails on the type of inputs users provide. You may also be able to predict and default the content desired through pre-determined prompts for specific use cases. This has the additional benefit of caching data so that the product has reliable and fast times for data to be returned (slow response time has been an issue for some ChatGPT users¹¹).

Another UX area to evaluate is Trust and Safety. It is often hard to detect when AI-generated content is wrong. The news is filled with stories about how algorithms and AI can create fake information¹², whether intentionally by bad actors or due to the limitations of the AI model. For example, fraudsters have leveraged AI to enhance phishing attacks or create malicious social media posts that have gone viral. Open-AI’s GPT-4 model was trained on data through September 2021, and they caution that it “…is not fully reliable (it ‘hallucinates’ facts and makes reasoning errors).¹³” Trust can also be eroded if the content is incomplete. The product team needs to understand what word, token, or other limits exist to avoid situations where text or imagery is cut-off or might otherwise not meet user expectations.

Finally, evaluate how your customers feel about the quality of the content. Some of this depends on what the product has promised. If customers were expecting human interaction, then they may have a low tolerance for AI-augmented responses. They may become skeptical about all the conversations with your organization, which can lead to broader distrust and brand decay. These are manageable risks that simply require anticipation.

4. Consider second order effects (and bad actors)

I want to emphasize the importance of planning for not only risks but also unintended use cases. Leading technologists, academics, and concerned consumers have raised concerns about potentially dystopian outcomes from AI systems. As of April this year, over twenty-six thousand signatures have been added to an open letter calling for a six month minimum pause on “giant AI experiments¹⁴.” Seven countries including Italy have banned ChatGPT due to privacy and misinformation concerns¹⁵.

If you expect the unexpected to happen, then you can manage it like any other product use case. One way to start is by setting up a brainstorm with a cross-functional team to identify who the “wrong” users would be. This obviously includes bad actors, but also think about the target age, demographic, and market for the product. Your product could be so successful that it attracts widespread usage. What is the worst or best-case scenario if customers outside of the intended segment use your product? How prepared is your organization to anticipate edge cases before they escalate into major issues? Check that you have the right instrumentation to anticipate changes. This is particularly important where it may impact data privacy, accessibility, or copyright concerns.

Also, be aware of the biases inherent in AI-generated content. The U.S. National Institute of Standards and Technology (NIST) argues for a “socio-technical” approach that considers three types of biases: human, systemic, and computational.¹⁶ Human biases refer to how people make inferences about the data that may be incorrect. Systemic biases occur when institutions operate in ways that are harmful to specific groups of people, such as racial discrimination. Computational biases are introduced when the data used to train AI has limited representation, lacking a diversity of gender, economic, or global inputs. Consider, for example, that while there are over 7000 spoken languages in the world, the Internet (on which many AI models are trained) uses 10 main languages, with English, Chinese, and Spanish being the top 3.¹⁷

Another thought exercise is to argue against automation even where there would be clear efficiency gains. What might be lost from human interactions or creativity that might impact factors beyond the task at hand? AI only knows what it has been trained on. Human relationships are more than data. Connections happen through authenticity. AI content is often remarkably specific, but deep personalization requires a level of understanding that may be more emotional than factual. The creative process is hard for humans at times, but perhaps the pride in the outcome is commensurate with the degree of struggle.

Moreover, humans are social beings. If your product has created a vibrant community of users who enjoy helping each other and sharing their unique contributions, how satisfying will that experience be if more or all of that content is AI-generated? It may not matter at all, but it’s worthwhile to anticipate how this might impact user engagement, retention, and brand value.

Your team may feel strongly that the product should not use AI-generated content, or at least not right now. In that case, evaluate what would happen if your competitors or new start-ups in your space offer it? How will this impact your product’s market position? For now, there are no signs that consumer fascination with generative AI is waning anytime soon. What happens if previously open forums become closed, with rich information about questions and answers siloed in proprietary data centers? Make sure your product’s value proposition will endure and thrive with changing market trends. AI can enhance or replace so many processes that you will need to lead your team to consider its impact on customers and internal stakeholders, whether or not your organization plans to build with AI in the near term.

5. Plan how to scale

Product teams are still discovering ways to apply generative AI to solve user problems. As experiments validate the hypotheses, the next step will be to scale. This may be to expand globally, and you will need to consider localization costs and quality. The sheer volume of content that AI models produce may make it cost-prohibitive to rely on your current translation vendors, especially if they provide human-reviewed content. Soon the quality of machine-generated translations will improve, but today coverage is still limited across geographies and content areas.

A bigger global consideration is that regulatory and compliance controls could be introduced at any time. Many would argue that governments have lagged in this area, but conversations are underway. In March, the U.S. Chamber of Commerce issued a report arguing that “It is time for action.”¹⁸ The European Union has proposed the AI Act¹⁹, a law that has the potential to become a global standard. Unlike its western counterparts, China has been actively establishing rules for AI in 3 areas, according to the Carnegie Endowment for International Peace: online algorithms, testing/certification, and ethics review boards²⁰. In April, the Cyberspace Administration of China issued new rules for generative AI applications. Make sure you are monitoring policy changes so that you are prepared to adjust. If you are operating in industries that are already highly regulated, such as health care or finance, you will also need to understand how existing rules may be amended.

Your product may also scale by offering more integrations to enable AI. You will need to develop a framework for partnerships or even acquisitions, ideally with a dedicated business development team. Given the large datasets and computing power needed to train AI models, many organizations will need to leverage APIs or tools from a few big AI providers. This creates dependencies that need to be managed. Beyond factors like cost, response time, and performance, a major consideration is control over ownership and content quality. Should you allow your data to be used to train the open models? You may want to train your own AI models, but the trade-offs involve time and money. On the other hand, your data may be the key to differentiation.

Another difficult problem is accountability for inherent AI biases, misuse by bad actors, and general inaccuracies in the content generated. Who is responsible for poor results from AI technologies, particularly if they result in harm to humans, other forms of life, or the planet? What role can or should your organization play to prevent, manage and mitigate these risks? In most markets, there are no clear policy guardrails. These are issues that every organization will need to address. Note that there are resources your team can leverage. OpenAI has a moderation endpoint that will score the content automatically, and product teams can determine what threshold to use for polarizing areas like hate, sex, and violence²¹. A simple Google search on “AI Content Moderation” returns numerous other options, given how critical this area is. Attorneys will be busy and in unfamiliar territory as they navigate everything from freedom of speech to intellectual property claims. Partner with them as you are planning your roadmap to avoid delays in product releases and to keep your customers safe.

Conclusion

This year, the Women In Product conference theme is Build What’s Next. To learn more about how to leverage AI, look for the keynote by Natalia Castillejo, Group Product Manager at Duolingo. Natalia will share the challenges and opportunities from her team’s experience building with generative AI.

I hope this article helps you consider ways to leverage generative AI in your product roadmap. We are lucky to be active contributors to this next wave of technology innovation. As organizations look for new applications of automation and machine learning, we as Product Managers play a leading role in ensuring that we celebrate the connections, diversity, and creativity that make us human. I’m excited for the scale at which AI, deployed responsibly, can truly enhance everyday lives.

For more articles like this, sign up to receive our Senior and Executive Leadership Newsletter once a month. 

Footnotes

^{1 Mollick, Ethan. “ChatGPT Is a Tipping Point for AI.” Harvard Business Review. December 14, 2022. https://hbr.org/2022/12/chatgpt-is-a-tipping-point-for-ai}

^{2 Hu, Krystal. “ChatGPT Sets Record for Fastest-growing User Base – Analyst Note.” Reuters. December 14, 2022. https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/.}

^{3 Shanklin, Will. “OpenAI Will Let Developers Build ChatGPT into Their Apps.” Engadget. March 1, 2023. https://www.engadget.com/openai-will-let-developers-build-chatgpt-into-their-apps-204737530.html.}

^{4 https://openai.com/research/gpt-4}

^{5 https://www.grandviewresearch.com/press-release/global-generative-ai-market}

^{6 “State of AI 2022 Report.” CB Insights. March 16, 2023. https://www.cbinsights.com/research/report/artificial-intelligence-trends-2022/.}

^{7 Nika, Marily , and Deb Liu. “The Promises and Pitfalls of ChatGPT.” Perspectives. March 22, 2023. https://debliu.substack.com/p/the-promises-and-pitfalls-of-chatgpt.}

^{8 Siavoshi, Mehrnaz. “The Importance of Natural Language Processing for Non-English Languages.” Towards Science, Medium. March 22, 2023. https://towardsdatascience.com/the-importance-of-natural-language-processing-for-non-english-languages-ada463697b9d.}

^{9 Ramlochan, Sunil. “The Importance of Natural Language Processing for Non-English Languages.” Prompt Engineering Institute. March 6, 2023. https://www.promptengineering.org/what-is-prompt-engineering/.}

^{10 Karhade, Dr. Mandar. “Evaluating Language Models by OpenAI, DeepMind, Google, Microsoft.” Level Up Coding, Medium. December 22, 2022. https://levelup.gitconnected.com/how-to-benchmark-language-models-by-openai-deepmind-google-microsoft-783d4307ec50.}

^{11 https://community.openai.com/t/is-it-possible-to-reduce-chatgpt-api-response-time/96069}

^{12 Fischer, Sara. “Exclusive: GPT-4 Readily Spouts Misinformation, Study Finds.” Axios. December 22, 2022. https://www.axios.com/2023/03/21/gpt4-misinformation-newsguard-study.}

^{13 Fischer, Sara. “Exclusive: GPT-4 Readily Spouts Misinformation, Study Finds.” Axios. December 22, 2022. https://www.axios.com/2023/03/21/gpt4-misinformation-newsguard-study.}

^{14 “Pause Giant AI Experiments: An Open Letter.” Future of Life Institute. March 22, 2023. https://futureoflife.org/open-letter/pause-giant-ai-experiments/.}

^{15 Martindale, Jon. “These Are the Countries where ChatGPT Is Currently Banned.” Digitaltrends. April 23, 2023. https://www.digitaltrends.com/computing/these-countries-chatgpt-banned/.}

^{16 “There’s More to AI Bias Than Biased Data, NIST Report Highlights.” Nist. March 16, 2022. https://www.nist.gov/news-events/news/2022/03/theres-more-ai-bias-biased-data-nist-report-highlights.}

^{17 https://www.internetworldstats.com/stats7.htm}

^{18 U.S. Chamber of Commerce Technology Engagement Center. Commission on Artificial Intelligence Competitiveness, Inclusion, and Innovation. 2023. https://www.uschamber.com/assets/documents/CTEC_AICommission2023_Exec-Summary.pdf.}

^{19 https://artificialintelligenceact.eu/}

^{20 Sheehan, Matt. “China’S New AI Governance Initiatives Shouldn’T Be Ignored.” Carnegie Endowment for International Peace. January 4, 2022. https://carnegieendowment.org/2022/01/04/china-s-new-ai-governance-initiatives-shouldn-t-be-ignored-pub-86127.}

^{21 https://artificialintelligenceact.eu/}

---