In a compelling presentation at QCon, Wes Reisz shared insights from an ambitious experiment: delivering a QCon certification program powered by artificial intelligence. His core message resonated deeply: Lean thinking combined with AI is a superpower, underscoring that human expertise and domain knowledge are more critical than ever in this new era.
Reisz drew an analogy to Nick Swinmurn, the founder of Zappos, who validated his idea of selling shoes online before building any infrastructure. This Lean approach—building the right thing first—was central to the QCon experiment. The certification program was initially sold to a limited cohort of 30 people, creating a "go/no-go" line that forced rapid, focused development. The entire system, from concept to deployment, was built in just four weeks, primarily during evenings, proving that validating an idea and achieving product-market fit should precede extensive technological investment.
The AI-Powered Certification Engine
The challenge was significant: process 75 conference talks, extract key information, and make it queryable by an LLM to provide actionable takeaways. The solution involved a robust, serverless architecture built on AWS, utilizing a monorepo, Python, Terraform, and standard CI/CD practices. This familiar software development stack was adapted to create an AI-driven system.
Retrieval-Augmented Generation (RAG) in Action
At the heart of the system was a Retrieval-Augmented Generation (RAG) architecture. Reisz explained RAG as a mechanism that injects a retriever ahead of an LLM. When a user asks a question, the retriever fetches relevant data from structured or unstructured sources and feeds it into the LLM's context window. This approach offers several critical benefits:
- Reduces hallucinations: By providing specific, verified information.
- Offers real-time data: Overcoming the LLM's knowledge cutoff.
- Provides domain-specific insights: Tailoring responses to particular fields.
- Enhances explainability: Tracing LLM outputs back to their source.
The system employed a dense retriever to convert queries into semantic meaning, going beyond simple keyword matching to understand the true intent behind a question. Reisz also touched upon advanced RAG variations like retrieve-and-re-rank, multimodal RAG (for video, sound, images), graph RAG (using knowledge graphs), hybrid RAG ( blending keyword and contextual searches), and agentic RAG (where agents select the best retriever). A key lesson here was the critical importance of chunking—breaking down content into meaningful segments for effective retrieval.
The Video Transcription Pipeline
To feed the RAG system, a sophisticated video transcription pipeline was developed. This serverless workflow involved:
- S3 Upload: An admin uploads video files to S3.
- Step Functions: Triggers a state machine.
- Transcription: AWS transcription services convert audio to text.
- Chunking: Content is broken into segments (initially based on speaker pauses).
- Embeddings: OpenAI services convert chunks into 1531-dimension vectors, capturing semantic meaning.
- Pinecone: A vector database stores these embeddings for rapid retrieval.
This pipeline processed 75 videos, generating approximately 15,000 embeddings at a cost of around $130, demonstrating that familiar serverless technologies can be effectively leveraged for AI-driven solutions.
Supervised Coding Agents: A Practical Assessment
Reisz made a conscious decision to use supervised coding agents (Claude Sonnet 3.7 via Cursor) for 95% of the code generation. His process was highly structured:
- Idea Iteration: Using a low-cost LLM to shape initial concepts.
- Requirements & Spec: Developing clear requirements and a detailed specification.
- Prompt Plan: Breaking down tasks into small, single-responsibility units.
- Guardrails: Setting up strict constraints using Cursor Rules, Python rules, testing frameworks, Terraform, and one-shot prompts.
While the agents generated readable and functional code, Reisz noted several observations:
- Human Intervention: 5% of the code required manual writing, often due to "doom loops" where the LLM struggled to meet specific demands.
- Code Bloat & Reuse: The generated code was often verbose (20,000 lines for the project) and exhibited poor code reuse, particularly with Claude Sonnet 3.7.
- "As Bad As It's Ever Going To Be": Despite current limitations, Reisz emphasized that AI tools are constantly improving, making it imperative for developers to engage with them now.
The key takeaway for coding agents: they are incredible for removing undifferentiated heavy lifting, but they require human guidance, expertise, and a structured approach to control outputs and avoid pitfalls like "doom loops."
Surfacing the RAG: ChatGPT Custom GPTs
To make the RAG accessible, the team leveraged ChatGPT Custom GPTs. Users could ask questions, and the custom GPT would use its instructions to call the retriever, query Pinecone, and present the results. However, this choice came with a lesson: the proprietary nature of GPT-4 Turbo meant users without a ChatGPT Plus license couldn't access it, highlighting the need to consider open alternatives or custom frontends for broader accessibility.
Workshop and Retrospective: The Human Element
The experiment culminated in a post-conference workshop where attendees used the AI tool to explore key trends and develop action plans. The workshop was a qualified success, receiving high participation and positive feedback. However, Reisz admitted a personal failing: he focused too much on the technology and not enough on the user experience, leading to some negative feedback regarding organization and communication.
A profound lesson emerged: despite having a powerful AI tool at their fingertips, participants still preferred talking to each other. This reinforced the Agile Manifesto's principle of "individual interactions over process and tools," even in an AI-driven world. Other key learnings included the continued importance of Lean thinking, the enduring relevance of RAG, the necessity of human expertise when using coding agents, and the often-overlooked human toil and burnout associated with rapidly adopting new AI tools.
Final Takeaways
Wes Reisz concluded with five enduring principles for the age of AI:
- Build the right thing: Lean practices are essential for validating ideas.
- There are no silver bullets: Technology alone won't solve all problems.
- Coding agents are powerful but need guidance: Human expertise remains paramount.
- Embrace change: AI tools are evolving rapidly; continuous engagement is crucial.
- Individual interactions over processing tools: Never forget the human element in software development.
The QCon experiment demonstrated that combining Lean experimentation with AI can lead to incredible innovations, but it also served as a powerful reminder that people, expertise, and domain knowledge are the true superpowers.