The Determinism Dilemma: Why Traditional Software Engineers Struggle with AI Development

The goal of creating something "predictable," reliable, and consistent is a shared principle across all the teams I've worked with throughout my career.

Knowing that the same code would always return the same output when given the same inputs was the foundation of everything we built.

We aimed for no surprises, no matter how complex a workflow might be. Whether implicitly or explicitly using finite state machines, this determinism enabled us to build testable, monitorable, maintainable, and, most importantly, predictable workflows.

We read and shared ideas at conferences, promoting patterns and principles like SOLID and DRY to create functional, composable, and extensible software.

Why Forcing AI to be Deterministic is the Wrong Approach

AI applications challenge this principle in a fundamental way. While AI systems can still be structured and maintainable, their inherent unpredictability requires us to rethink our approach.

For instance, asking an AI model to summarise a piece of text multiple times often produces different responses, especially when using zero-shot prompting. Many developers, uncomfortable with this unpredictability, try to force consistency by tweaking parameters like temperature or top-k.

However, forcing an AI model to behave more predictably risks losing its greatest strength. Generative models excel when producing varied and creative outputs. Overfitting an AI model to achieve consistency may make it behave more like traditional software but, in doing so, diminishes its versatility and potential.

Embracing AI's Creativity

The key is to tailor the approach to the use case. In payment applications, predictability is paramount. These systems must behave in a consistent and reliable manner.

In contrast, a chatbot thrives on variety and personalisation. Its responses should feel unique and engaging because its purpose is to interact with users whose goals or needs may not be immediately clear. This shift in thinking requires developers to embrace AI's non-deterministic nature when appropriate.

Rather than suppressing it, we must design systems that take advantage of this quality while incorporating safeguards to manage its limitations.

"Let Go of Determinism, Luke"

To borrow from Obi-Wan Kenobi's advice: "Use the Force, Luke. Let go."

Knowing when to rely on determinism and when to let go of it is crucial.

For traditional applications, the goal is consistent and predictable behaviour.

For AI, the focus is often on adaptability and creativity.

In future posts, I will explore how to balance these approaches.

For now, remember that the key to success in AI development is not forcing it to follow old rules. The real challenge is learning to use its unique strengths effectively.