Arrange Act Assert

Jag Reehals thinking on things, mostly product development

Lessons from Building AI Products with Stakeholders

27 Jan 2025

Building AI products with stakeholders requires a fundamentally different approach than traditional software development.

From my experience working with AI, success depends not only on technical implementation but also on bridging the gap between AI capabilities and stakeholder expectations.

In this post, I’ll share lessons from guiding stakeholders through AI’s possibilities and limitations.

Teaching Stakeholders About the Inner Workings of AI

One of the first things I realised when working with AI is how many misconceptions stakeholders have about how it works, what's possible, and how to maintain AI features.

Many expect AI models will behave like deterministic tools they've used in the past. These misunderstandings often stem from what stakeholders have seen rather than experienced. Industry hype and "demoware" contribute to unrealistic expectations about how easily AI can deliver the results they want.

For example, stakeholders often ask why OpenAI API results differ from their ChatGPT experience. Addressing these questions early is critical for building trust and setting realistic expectations.

Explaining how AI generates results helps stakeholders understand its capabilities and limitations. This lays the foundation for collaboration and ensures stakeholders feel involved in shaping the outcomes.

Starting with the OpenAI Playground

Stakeholders need an interactive way to experiment with AI.

The OpenAI Playground is an excellent starting point. It allows users to test prompts, adjust parameters, and see results immediately.

This hands-on experience helps stakeholders understand how generative AI works and how changing inputs can influence outputs. For many, it's their first experience seeing beyond simply entering text into a chat interface.

The OpenAI Playground provides an interactive environment where stakeholders can experiment with prompts and settings to understand how generative AI works.
The OpenAI Playground is an intuitive tool for stakeholders to test prompts, tweak settings, and explore how generative AI works.

The OpenAI Playground also offers built-in collaboration features, making it easy for teams to tweak settings like temperature and learn through trial and error. Its clean, intuitive interface ensures users can get results quickly without being overwhelmed by complexity.

Educating Stakeholders Using Their Data

While the OpenAI Playground is a fantastic starting point, I needed to retrieve data from our database and support experimentation with non-OpenAI models. Using personalised data provides a sharper focus on the goals stakeholders want to achieve.

With my full-stack experience, I built a secure web app that integrated:

This customisation streamlined collaboration with stakeholders, allowing them to experiment directly, compare results, and explore what worked best. It turned abstract technical concepts into practical, measurable outcomes, improving alignment and creating a shared learning experience.

After stakeholders understand how their data can integrate with AI systems, the next step is determining which AI approaches best align with their goals.

Helping Stakeholders Choose the Right AI Approach

There's no one-size-fits-all approach when building AI products.

Stakeholders often assume advanced techniques, like fine-tuning, are essential for success. As a consultant, I recommend beginning with straightforward methods that deliver results quickly using fewer resources and at less expense.

For example, few-shot prompting can be a practical alternative or a precursor to fine-tuning. By providing a few well-crafted examples, the AI can adapt its outputs to meet specific requirements, often achieving the desired results without the need for the added complexity of fine-tuning.

That's not to say fine-tuning is never needed, but taking an incremental approach ensures that solutions are evaluated step by step, with short feedback loops to assess their value, keeping progress aligned with stakeholder goals.

Once stakeholders understand how their data integrates with AI tools and the right approaches to achieve their goals, the next step is to explore trade-offs and implications through experimentation.

Understanding Trade-offs Through Experimentation

Educating stakeholders about the practical implications of AI is essential. This includes:

Through experimentation, stakeholders can observe these trade-offs firsthand. For instance, they can see how increasing temperature settings results in more creative but less consistent outputs, or how longer prompts affect processing time and cost.

These insights enable stakeholders to make data-driven decisions that align with their goals and constraints, ensuring AI solutions are practical, cost-effective, and aligned with their needs.

How to Work with Stakeholders When Building AI Products

Effective collaboration with stakeholders requires more than just technical expertise. It's about creating an environment where stakeholders feel empowered to explore, learn, and contribute meaningfully.

Here's what I've found works best:

  1. Start with Accessible Tools: Use tools like the OpenAI Playground to introduce generative AI concepts in an approachable way.
  2. Focus on Education: Explain key concepts, such as temperature, token limits, and costs, in simple terms. Use examples to make abstract ideas concrete.
  3. Collaborate Transparently: Provide tools that make the decision-making process measurable and clear. Transparency builds trust and fosters alignment.
  4. Adapt to Their Needs: Tailor your approach to stakeholder priorities, whether they focus on cost, speed, or quality.
  5. Experiment: Start with small, low-risk tests to gather insights. Scale up once you've identified what works best.
  6. Use the Best Tools for Their Context: Help stakeholders identify the tools and techniques (e.g., fine-tuning, few-shot prompting) that align with their specific goals and constraints, avoiding unnecessary complexity.

By following these steps and fostering collaboration, transparency, and experimentation, you can empower stakeholders to navigate AI's opportunities and challenges with confidence.

This approach leads to more informed decisions, and with stakeholder engagement, you can deliver outcomes that align with their goals.

ai