In this episode of Developers Who Test, host Chris Harbert sits down with Nikolai Grabner, a Senior Software Engineer and Technical Lead at Enigma Solutions, to talk about what it actually takes to build and test AI systems that are ready for production. Nikolai opens by demystifying retrieval augmented generation (RAG), using the analogy of a knowledgeable judge who consults a specialist library (a vector database) when a question falls outside their general expertise. He explains why companies are increasingly building private, in-network RAG systems: to keep proprietary information out of third-party models like OpenAI and Anthropic while still giving employees a single, instant point of access for things like HR policy questions and onboarding knowledge.

Nikolai shares the origin story behind his own product, SAP Bot, which grew out of market research he did when founding Enigma Solutions. After hearing that many internal RAG systems were, in his words, not working properly, his QA instincts kicked in and he set out to prove a thoroughly tested private RAG could get close to the quality of the big public models. A central theme of the conversation is how testing AI is fundamentally different from traditional pass or fail test cases. Because the same prompt can return different answers each time, Nikolai built a scoring mechanism rooted in statistics, precision, and coverage to detect hallucination (making things up) and drift (staying on topic but giving wrong answers). Chris draws a parallel to Six Sigma and the idea of variability as the enemy of quality.

The two get into the practical realities of building with AI, including using tools like Prompt Foo to fire the same set of prompts at OpenAI, Anthropic, and Gemini and compare results, tuning the temperature for creativity, and learning hard lessons about performance. Nikolai recounts how discovering CUDA and offloading the LLM to the GPU cut his response times from five minutes down to about fifteen seconds with streamed output. They also swap cautionary tales about AI getting things subtly wrong: a coding tool that inverted passes and fails in test results, and an MCP server with start and end dates reversed that the LLM quietly worked around, leaving a hidden landmine in the system.

Much of the discussion centers on discipline. Nikolai argues that vibe coding can produce production-grade software, but only with clear requirements, a design spec, a roadmap, phased delivery, and regression testing after every change. He compares vibe coding to managing a junior dev team that still needs its work tested. Chris highlights how far modern tooling has come, pointing to Playwright MCP and the Testery MCP for running reliable end to end tests at scale and feeding results back to the LLM, and Nikolai contrasts that with the weeks it once took to script a single test in WinRunner back in 1999.

The episode closes on continuous quality and staying current. Nikolai makes the case for always-on testing of AI systems (since a single faulty document can skew an entire RAG), dedicated research and development teams, and giving testers room to run proofs of concept on test infrastructure. Both reflect on how quickly organizations can become dinosaurs in the AI era, the value of conferences like TestCon for learning what is genuinely cutting edge, and how that spirit of learning by doing is exactly why the podcast exists.

Key Topics:

• What retrieval augmented generation (RAG) is and how it extends an LLM with specialized knowledge
• Why companies build private, in-network RAGs with vector databases to protect proprietary data
• The origin of SAP Bot and applying a QA mindset to building AI products
• Testing LLMs statistically: scoring for precision, coverage, hallucination, and drift
• Using Prompt Foo to run prompts across OpenAI, Anthropic, and Gemini for multi-model comparison
• Performance gains from CUDA and GPU offloading, plus response streaming
• Real-world AI failure stories: inverted pass/fail results and reversed start/end dates in an MCP server
• Turning vibe-coded AI into production-grade systems through discipline, specs, and regression testing
• Modern testing tooling (Playwright MCP, Testery MCP) versus legacy tools like WinRunner
• Continuous AI testing, dedicated R&D teams, proofs of concept, and learning from conferences

In this episode of Developers Who Test, host Chris Harbert sits down with Nathen Harvey, who leads the DORA team at Google Cloud. Nathen has co-authored multiple reports on software delivery performance and was a contributor and editor for 97 Things Every Cloud Engineer Should Know.

The conversation starts with what people get wrong about DORA. Nathen explains that the famous four (now five) software delivery metrics are just the surface. Treating them as the whole picture is like stepping on a scale every day and expecting the number to change: the metrics tell you how you are doing, but it is the underlying capabilities and practices that actually move them. He walks through how Accelerate introduced DORA to most of the industry, why so many readers stop at the four keys on page 19, and how the capabilities model in the appendix is where the real value lives.

Chris and Nathen dig into a decade of findings: throughput and stability move together rather than in opposition, smaller batches lead to better outcomes, and trunk-based development is both one of the most effective and most controversial practices, including its surprising link to burnout on teams new to it. They talk about why alignment across practices matters, since you cannot adopt trunk-based development without also addressing test automation, test data management, and CI/CD, and why the goal should be to become an elite improver rather than an elite performer.

The second half focuses on DORA's new AI Capabilities Model, published in December 2025. With roughly 90 percent of respondents now using AI professionally, the differentiator is no longer whether you use AI but how. Nathen lays out the seven capabilities that amplify AI's benefits: a clear and communicated AI stance, a healthy data ecosystem, AI-accessible data, working in small batches, strong version control, user centricity, and a high-quality internal platform. The core 2025 finding is that AI is an amplifier across the SDLC: high-performing teams get faster, while teams with bottlenecks feel that pain even more acutely when they push ten times more change into an unscaled review or testing process.

They close on what this means in practice: AI is democratizing who can build software, so investing in platform guardrails, fast feedback, ephemeral environments, and high parallelism testing becomes more important than ever. Nathen points listeners to dora.dev and the dora.community to assess their own capabilities and start improving.

Key Topics:

• Common misconceptions and anti-patterns around the DORA metrics
• Why the four (now five) keys are only the surface of software delivery performance
• How Accelerate and the capabilities model fit together
• Throughput and stability improving together, not in tension
• Trunk-based development, smaller batches, and the burnout finding
• Alignment across test automation, test data, and CI/CD
• Becoming an elite improver instead of an elite performer
• Contextualizing findings and user centricity
• The new AI Capabilities Model and its seven capabilities
• AI as an amplifier of both strengths and bottlenecks across the SDLC
• Democratized building, platform guardrails, and the renewed importance of fast feedback

Developers Who Test Episode 2

0:00

/2809.208163

1×

Don Campbell on ⁠LinkedIn⁠

Chris Harbert on ⁠LinkedIn⁠

Sponsored by ⁠Testery