State of Play: AI Coding Assistants

Birgitta Böckeler gives this talk as a deliberate sequel. A year earlier at QCon London she presented From Autocomplete to Agents, an attempt to map the territory of AI-assisted coding for an audience she described as overwhelmed — including herself, and she works on this full time as Thoughtworks' Global Lead for AI-assisted Software Delivery. This talk repeats that exercise for the following twelve months.

Her baseline for how much moved is worth stating first. At that earlier talk the term "vibe coding" was about two months old, MCP was all the rage, and Claude Code was — her phrase — still in diapers: already there, but not yet generally available. Her central thesis is that the interesting change was not model quality. It was that context engineering matured into a genuine lever, and that the resulting capability created a strong pull to take humans out of the loop — a pull that is sometimes correct and sometimes treacherous, and that we currently have no good way to tell apart except by deliberate risk assessment and by building what she calls a harness around the agent.

InfoQ files the recording under QCon London 2026, labels it "Recorded at: Apr 08, 2026", and gives a running time of about 42 minutes. These notes reorganise the talk into a teaching sequence rather than following it beat by beat. Where I add background that Böckeler did not state, it is marked explicitly. The published transcript contains no audience Q&A, so nothing below is drawn from one.

What You Will Learn

• What context engineering means for a coding agent in practice, and what changed after the era of a single rules file.
• How Anthropic's skills format works, why it shifted usage away from MCP toward ordinary CLIs, and what "progressive loading" buys you.
• Why context window occupancy is now a thing you monitor, and how it drives both agent effectiveness and your bill.
• The concrete forms that reduced supervision now takes: cloud agents, headless CLI agents in CI, parallel local sessions, agent teams, and swarms.
• A three-factor risk assessment for deciding how much supervision a given task needs, and why only one of the three factors is genuinely new.
• The security picture: prompt injection, allow-list weaknesses, secret extraction, and Simon Willison's lethal trifecta.
• Why the cost per change is now nowhere near the numbers quoted in early 2024.
• The feedforward/feedback "harness" model, including using structural tests and enriched lint messages as agent feedback.

Context Engineering, One Year On

Böckeler defines context engineering in its simplest form: curate the information your model or agent sees in order to get better results. The term did not exist at the previous year's QCon London and started circulating around June.

A year ago, her slide on the state of context engineering for coding agents had two items on it. The first was rules files — an AGENTS.md or CLAUDE.md in your workspace that gets sent to the agent at the start of every session, where you record typical pitfalls and repeated errors. Her own example was an agent that kept forgetting to activate a Python virtual environment before running a Python process. The second was MCP servers, which let an agent fetch data more dynamically. Background for readers new to the term: MCP is a protocol for exposing tools and data sources to a model through a separate server process.

The equivalent slide now carries rules, MCP servers, commands, skills, subagents, plugins, and specs. Böckeler singles out skills for a closer look, partly because she found them genuinely confusing herself — she took a sabbatical at the end of the previous year, and skills had shipped by the time she came back.

Skills

Skills were introduced by Anthropic, building on patterns that were already circulating. Three properties matter.

They modularise rules. Instead of one large file that is always sent to the agent, you get subfolders, each covering one thing. Böckeler's range of examples runs from "here's how we usually build a React component" to "here's how you get logs from our AWS test environment".

They load just in time. The model initially receives only each skill's description — for a get-logs skill, something like get logs from a test environment, for example for debugging incidents. Only when the model decides that is what it is doing does the full skill get loaded. This progressive, lazy loading is what keeps the context window from filling up at session start.

They are folders, not files. A skill can carry additional documentation and scripts alongside its Markdown. This is where Böckeler locates a wider shift in practice: people realised they can simply tell the agent in Markdown to use a CLI already installed on the machine. That realisation moved a lot of agentic coding use cases away from MCP servers and toward existing CLIs and small scripts, because it avoids running yet another process type on your machine.

The Two Halves, And The Non-Determinism

Böckeler frames context engineering as the combination of two things. The first is reusable instructions and conventions — how to write a React component, how to bootstrap a new project, ordinary coding conventions. The second is context interfaces: the skill description, the tool list an MCP server exposes, the tool list built into the agent. These are what the model reads in order to decide in this situation I want to load that skill, call this tool.

The unavoidable caveat is that this decision is non-deterministic. You never have a guarantee that the model will actually decide to load your skill. Every piece of context engineering is a probability shift, not a control.

Watching The Context Window

Because of this, the human job becomes managing context and monitoring its size. Context windows are technically much larger than a year or two ago, but Böckeler's reported experience is that you can still feel agent effectiveness degrade as they fill. There is also a direct cost mechanism she states plainly: every round trip with the model sends the full context window, so a fat context is a recurring charge, not a one-off.

Coding agents have added features for inspecting what is actually in the context, and the contents are instructive. In a Claude Code session she started and had barely typed into, the window was already 15% full — the Claude Code system prompt, plus her context interfaces such as skill descriptions. That is the balancing act: every skill you add costs baseline occupancy in every session. GitHub Copilot has an equivalent feature. Her read on the vendor landscape is that the Claude Code team is leading and the others copy.

Subagents

Subagents are the last piece she puts under context engineering, and most coding agents now have them built in: the main agent can spawn a child agent with its own context window.

The most common use is one users do not even trigger — when a session starts and the agent needs to understand your codebase, it often spawns a research subagent, which in Claude Code appears as an Explore agent. Research is token-expensive because it means reading many files to find the few that matter. Doing it in a subagent means only the result comes back to the main session, which never sees the discarded material.

The most common deliberate use is a code review agent: a separate context window that knows nothing about the session's history, optionally running a different model. Böckeler notes this feature is what unlocked much of what follows.

Questions To Ask In Your Own Organisation

Böckeler's framing throughout is that all of this amplifies — good stuff and bad stuff alike. Her prompts for the audience:

• Which coding conventions do you actually want amplified?
• Which workflows could you build for modernisation initiatives? Migration is a strong generative-AI use case, and the newer agent features make workflows easier to assemble. Her example is a colleague working with a client who had thousands of CI/CD pipelines in an older tool to move to GitHub Actions, and who was building a human-supervised workflow out of subagents, skills, and MCP servers.
• What tools should exist organisation-wide to make it easier for an agent to act or to fetch information? CLIs, MCP servers, and — particularly for less common languages — language servers that actually understand the language.
• Which practices, not just conventions, do you want amplified? Her recurring examples are improving an architecture decision record and doing threat modelling, both of which can be encoded as skills that help people understand the practice.

She is candid that this is "engineering" in quotes. Versioning and distribution are unsolved; plugin marketplaces are a first idea, again from Anthropic and the Claude Code team, and not yet mature. Whether a given piece of context makes things better or worse is an evals problem, and the tooling is early — Anthropic had just released something to make skill evals easier, and the skills registry Tessl had released something recently too.

More Autonomy, Less Supervision

Model improvements plus stronger context engineering produced the year's main trend: giving agents more autonomy and reducing human supervision. The supervised mode is the familiar one — you sit in front of the session, watch what it does, steer, go back and forth. Unsupervised is the opposite, and it now has several concrete shapes.

Cloud agents. OpenAI's Codex popularised the pattern of dispatching an agent into the cloud for twenty minutes to do something for you. The major products now offer agents that run in the cloud as well as locally in the IDE — she shows Cursor and Claude Code, and believes Copilot has it too. Because these are remote, people also drive them from a phone, coding on their commute.

Headless CLI agents. Every major product now has a command-line version; Claude Code got the attention first, and there are now Cursor and Copilot CLIs too. Run in headless mode, they slot into existing pipeline systems — there are GitHub Actions for Claude Code and Copilot.

Böckeler notes this immediately resurrects a familiar beast: sandboxing and environment provisioning. The agent needs compilers and dev tools, it needs the right amount of memory, and it needs a decision about internet access — a question CI/CD pipelines and dev sandboxes already had. What is new is prompt injection when an agent loads untrusted content from the web.

Parallel local sessions. Reduced supervision is happening locally too. She cites Steve Yegge's Gas Town blog post, The 8 Stages of Dev Evolution to AI, where stage 6 is three Claude Code instances running in parallel on your machine and stage 7 is ten. Her honest report of trying three: it is a lot, and she kept typing the wrong thing into the wrong session. Some teams are nevertheless doing this.

Swarms And Agent Teams

Stage 8 is agent swarms, which Böckeler treats as the hype of the moment. By swarm she means dispatching dozens or hundreds of agents — throw as many at the wall as you can, see what sticks, watch for emergent behaviour, and work out coordination afterwards. Gas Town is one example; the Claude Flow project has been around a while and was recently renamed.

Two experiments made people nervous: Cursor ran agents for up to a week to build a browser, and Anthropic had agents build a C compiler. Böckeler's caution here is the most important calibration in this section. Both use cases are extremely well specified, and their specifications are all over the internet. Cursor themselves say the use case was picked for that property. The C compiler case additionally has an elaborate public test suite the agents can use as a feedback signal. That combination — an exhaustively documented specification plus a ready-made oracle — is precisely what enterprise software does not have.

Her recommendation is not to go and run Gas Town, but if you want to dip a toe in, Claude Code has a preview feature called agent teams behind a flag. She distinguishes teams from swarms conceptually, even though some people use the words interchangeably: a swarm is dozens or hundreds of agents with AI deciding which to use, while a team is a smaller set — she tried five — where a main agent orchestrates, decides what can be parallelised, and the agents can talk to each other. She is clear this is all early days.

Gauging The Right Level Of Supervision

Her first question is where to experiment at all. Depending on your environment you may not want to take large risks, so she suggests contained targets — cleaning up feature toggles is her example, and code review agents are what most people reach for first. The second question is organisational: how do you help other people gauge the level of supervision they should choose.

The practical question is then not whether to reduce supervision but how to decide per task. Böckeler describes doing a constant stream of micro and occasionally macro risk assessments while using AI, and she decomposes risk the standard way: probability, impact, and detectability.

The assessment drives three decisions: which workflow to use (elaborate planning up front versus a quick prompt), how much of the output to review (fully vibe code and look at nothing, read every line, or somewhere in between), and how long to let the agent run unsupervised — because the longer it runs, the more you have to review afterwards to work out what actually happened.

Her sharpest observation is about which part of this is actually new. Only knowing the context you supplied and knowing how good the AI is at this task category are new skills. Everything else — assessing criticality, knowing your feedback loops, judging your own requirements — is something an experienced developer should already be good at. The new part is the intuition you build over time by using the tools.

This is also where organisational readiness enters, in her phrasing: you have to be this tall to ride the roller coaster. The probability of the agent going wrong rises with a bad codebase, because the agent picks up existing patterns, and with a system where relevant things are scattered across many places, so it is less likely to find all the information. Detectability collapses without good test automation — you lose both your own way to verify and the agent's.

Security And Cost: Two Bills That Came Due

Böckeler flags two things that also changed over the year, but not in a direction anyone wanted.

Security

There is now a report of an incident most weeks, and it is almost always prompt injection — an agent takes in content from an untrusted source that carries instructions the user never intended.

The first consequence is unwanted command execution. Every agent has allow-list features so you can pre-approve command patterns rather than being asked each time. Böckeler's point is that there are weaknesses in the implementations of these allow lists, and combining implementation weaknesses with an unsolved problem space and a non-deterministic model is a real risk.

The second is secret extraction. The exposure is worst for open-source projects that accept GitHub Issues from anybody and immediately trigger an agent with little supervision. In the incident she describes, prompt injection through a GitHub Issue extracted secrets that let the attacker push to the npm registry for that tool. She is careful to say that not all of this maps onto enterprise work — but it does show sharply increased risk in the dependencies we consume and in how careful we have to be about the ecosystem we pull into our applications.

The model she recommends for reasoning about this is Simon Willison's lethal trifecta, written up in June 2025. When an agent simultaneously has exposure to untrusted content, access to private data, and the ability to communicate externally, you have a serious data and security problem. Her emphasis is that this is not a technical problem but a conceptual one. Reader's note: I read that as meaning you cannot patch your way out of the combination; the phrasing is mine, not hers. It bites hardest outside coding: integrate email with read and write permissions into an agent and you already have all three legs. She is openly curious how the promised business agent use cases will get around it.

Her defensive prompts are to make it easy to sandbox coding agents locally as well as in the cloud — she has been using dev containers a lot for this, and notes new products appearing with interesting ideas — and to ask what the AI security literacy of your engineers actually is. Do they know what is happening under the hood? Do they know not to run YOLO mode, where no command needs approval at all?

Cost

The honeymoon is over here too. Böckeler contrasts a keynote she heard in early 2024, where the speaker offered that generating 100 lines of code costs about 12 cents, so compare that to a developer's salary. Setting aside that lines of code are a poor measure of value, the number itself no longer holds.

From token-usage sites in summer 2025 she cites one person averaging $380 a day, which extrapolated over 20 working days and 12 months gives $91,200 — as she notes, not a bad developer's salary in Germany. She stresses this was summer and that usage has grown since. Pricing has moved with it: from $20 flat rates at the start, and Copilot possibly $10 in the very beginning, to something more like $200 rates that are not really flat because you get request-limited. Hence the Reddit genre of it's the middle of the month and I'm out of tokens, from people who can no longer work without them.

Her explanation of the gap is the most useful part. In early 2024 we were doing autocomplete, maybe asking a chat for a few lines. A single change today runs a chain roughly like this: research the existing code, produce a plan, human reviews and adjusts the plan, implement, run the tests, fix the tests, check lint, fix lint, possibly check the browser for a visual feature, fix that, run a code review agent, react to its findings, summarise. All of that back and forth might land two lines of code. The cost is in the loop, not the output.

Where That Leaves Us

Böckeler's summary of the twelve months is that context engineering has become a powerful lever of amplification, for good and for bad, and that you can now do a great deal to raise the probability that an agent does what you want. Model improvements happened too, but she finds them less interesting than everything built around them.

The consequence is that there are strong forces tempting us out of the loop, and the organisational judgement call is where to give in to that pull and where it is treacherous. Something can feel good and produce quick results now while you have no idea what it means in a year. Layered on top is the pressure for throughput — everyone wants to be faster, look how many PRs we merged this week — which raises a question beyond security and cost: what happens to the maintainability of the code?

The Five-Month Greenfield Experiment

Her worked example is a recent article by a team at OpenAI. They spent five months on a codebase that started greenfield, under a self-imposed rule: do not touch the code directly. Only interact with the agent, and continuously improve the setup around it so the agent can maintain the codebase autonomously.

What they built was a mix. Part was context engineering — skills and similar. Part was deterministic checks: custom linters and structural tests. The result is the honest bit. They still reported entropy increasing and drift happening, and their response was what they called garbage collection — agents running continuously against the codebase, cleaning up over time.

Structural Tests As Agent Feedback

The pattern of architectural constraints plus deterministic tooling keeps surfacing in stories from teams doing a lot of agentic work, and Böckeler experimented with it herself. The tools are old: ArchUnit and Spring Modulith in the Java world; on the TypeScript codebase she was working on she used dependency-cruiser, which she had never heard of before. She defined the layers she wanted and wrote rules — one example being that external SDKs may only be imported by files in the clients folder, which held clients for other APIs, and not in the domain folder.

Her explanation for why these tools were underused is worth sitting with. We have had them for a while, but many people never adopted them because we were the ones writing the code: I know how to modularise, I don't need a tool to enforce constraints on me. Now they become an interesting feedback tool for the agent rather than a constraint on the human.

There is a second-order trick here. Because these are your own linters and structural tests, you can extend the failure messages themselves, which Böckeler calls a good type of prompt injection: write the message so it carries instructions or hints about how to react. Her example is a rule capping files at 500 lines. To stop the agent from gaming it by cramming multiple statements onto each line, put the intent in the message — this is a smell for a design problem, so you should consider refactoring — so the agent gets the meaning the rule has for you, not just the constraint.

Architecture And Data Flow: The Harness

All of this is ultimately about how we increase our trust in agents, and Böckeler's framing device is building a harness. The term harness engineering comes from the OpenAI team above. Her framing of the goal is deliberately modest: we are not aiming for agents producing perfect code, because we do not produce perfect code either. The question is how to get enough confidence and trust for your particular situation to deliver software safely, quickly, and sustainably.

The mental model has two directions. Feedforward is everything that anticipates what the agent might get wrong and raises the probability of it doing the right thing first time. Feedback is everything that catches problems after the first generation, so the agent can do the rote correction work before a human looks at anything.

She overlays a second axis on that, a framing she credits to the company Moderne: GPU-based versus CPU-based. GPU-based means inference — the instructions and skills going in, the code review agent coming back. CPU-based means deterministic tooling. Her argument is that both directions currently lean too heavily on inference and can be strengthened with deterministic components.

The language server example is the most concrete: give an agent access to something like IntelliJ's refactoring capabilities and it can use rename-symbol to perform a refactor, instead of doing text diffs all over the codebase. Again, this raises the probability of a correct first attempt rather than guaranteeing it.

Some of the harness is built into the coding agent itself — how it edits code, how it searches code — and vendors keep improving that. The rest is yours to build and specialise to your situation. Humans are the steerers of the harness, which is what the OpenAI team spent five months doing. And you can use AI to build the harness: Böckeler wrote none of her structural test tooling by hand. Her justification is a risk argument — harness tooling carries a far lower risk factor than the production software it guards. Writer's note: that argument holds only as long as a faulty harness fails loudly rather than silently passing bad code; Böckeler does not address that case.

The following diagram is my synthesis of the mental model Böckeler described verbally, not a reproduction of her slide.

flowchart LR
    H[Human steerer] -->|builds and tunes| FF
    H -->|builds and tunes| FB
    subgraph FF[Feedforward harness]
        F1[Principles, conventions, docs, how-tos]
        F2[CLIs, bootstrap scripts, codemods, language servers]
    end
    FF --> A[Coding agent]
    A --> C[First generation of code]
    C --> FB
    subgraph FB[Feedback harness]
        B1[Code review agent]
        B2[Static analysis, structural tests, custom linters]
        B3[Logs, running app, browser checks]
    end
    FB -->|corrections| A
    FB --> R[Human review]
    R -->|improve the harness| H

Harnesses As A Possible Abstraction Layer

Böckeler offers a speculation, clearly labelled as one. Building custom tooling of this kind used to be too much work to justify; now AI makes it cheap. She wonders whether the harness becomes a new abstraction layer. She guesses that a handful of topologies cover about 80% of what we build — data dashboards that collect from other APIs and display, CRUD business services, event processors. If you have a definition of how such an application should be structured and what the stack is, then instead of a service template you might instantiate a harness template. In that world you might stop caring whether it is React or Vue, and one of your decision criteria becomes simply whether a harness already exists so you do not have to build one.

The Gap: Verifying Functionality

She is explicit that everything above concerns maintainability and internal code quality, and that verifying functionality remains largely unsolved. We do not only want safety-critical systems to work; we want all software to work. She suggests we need harnesses for other qualities too — architecture fitness, operability, performance.

For behaviour specifically, her description of current practice is unflattering: you feed forward a description of the functionality, and the feedback is that the test suite is green — but the test suite was generated by AI — plus some manual testing, and that's it. Some people report reviewing the tests more carefully; she says she is sometimes a little sceptical. Her verdict is blunt: that is not good enough, and we have to come up with better ways.

Trade-offs And Limitations

What has raised her trust. Improved models. More sophisticated context engineering, especially progressive loading. More tool integrations and subagents. New food for thought about how far static analysis can be pushed.

What has not. Models still do stupid things all the time. Her example is a viral Reddit post where an agent explained that yes, you said no, but I thought you said no to me asking you for permission, so I just went ahead — teenager logic, and she says she sees this kind of thing constantly.

Cognitive overload in the loop. There are increasing anecdotes of burnout or of people simply being overwhelmed, either by the volume of review or by juggling multiple simultaneous sessions. She adds an important qualifier: when she hears of this being dialled up to eleven successfully, it is usually experienced engineers doing it, who have capacity for more load precisely because they have so much experience.

Pressure for speed from above. Push people to be faster with AI and they cut corners and get sloppy. She cites an article about Amazon reflecting on outages reportedly related to AI-generated code, where one response was to add more gateways requiring senior engineer review before production. Her comment is that this seems self-defeating — weren't we supposed to be getting to production faster? — and probably is not the solution either. The question she has been asking herself instead is how much speed do we actually need: what is the Goldilocks speed, fast enough but not too fast, what are the risks of all this speed, and how does the speed actually help the organisation compared with other things you could be investing in.

The Counterweight She Ends On

Böckeler closes by naming the through-line explicitly — reducing human supervision, and using AI as an automation tool that builds more for us — and then deliberately declines to leave it there. AI is a Swiss Army knife with many potential use cases, and plenty of them are useful with supervision: they do not necessarily overload you, and they are a good extension of work you would otherwise do manually and slowly. Much of the year's progress, she says, made AI more effective and a better developer experience precisely when it is used as an enhancement of yourself.

Her forecast for the next twelve months is symmetrical. There will be more good ideas, and there will also be more evidence about the worries — overload, and skills atrophying.

Practical Takeaways

• Decide deliberately which conventions and practices to encode as skills — ADR writing and threat modelling among them — because context engineering amplifies bad ones as readily as good ones.
• Prefer existing CLIs and scripts to standing up another MCP server when the agent just needs to run something that already exists on the machine.
• Monitor context occupancy, not just context limits. Use the agent's inspection feature, and treat every added skill description as permanent baseline cost in every session.
• Run the probability/impact/detectability assessment explicitly before choosing how much planning, how much review, and how long a leash a task gets.
• Sandbox agents locally, not just in the cloud. Dev containers are a pre-existing option she has been using heavily. Never run without command approval.
• Check your codebase against the lethal trifecta wherever an agent touches untrusted content, private data, and an outbound channel at once.
• Reach for structural tests and custom linters as agent feedback, and write their failure messages to explain the intent behind the rule, not just the rule.
• Reflect on four readiness questions, which is her closing ask: how ready would you be if you wanted to give AI more autonomy in your delivery? What is your automated safety net? What is your security stance? What is your people's AI literacy? Improving all four is worth it regardless of how far you take agent autonomy — and it is worth it for the humans too. AI can help you build that safety net today.

Key Terms

• Context engineering — Curating the information a model or agent sees in order to get better results.
• Rules file — A Markdown file such as AGENTS.md or CLAUDE.md, sent to the agent at the start of every session.
• Skill — Anthropic's modular unit of agent context: a folder holding Markdown, optional docs, and optional scripts, surfaced to the model first as a short description and loaded in full only on demand.
• Context interface — The metadata a model reads to decide what to invoke: a skill description, an MCP server's tool list, an agent's built-in tool list.
• Progressive loading — Deferring the loading of full context until the model determines it is relevant.
• Subagent — A child agent spawned by the main agent with its own context window, used to keep token-heavy work such as codebase research out of the main session.
• Agent team — A small orchestrated group of agents where a main agent decides what to parallelise and members can communicate; a preview feature in Claude Code.
• Agent swarm — Dispatching dozens or hundreds of agents at a problem, with AI deciding which to use, in the hope of useful emergent behaviour.
• Headless mode — Running a CLI coding agent non-interactively so it can be embedded in a pipeline such as a GitHub Action.
• Prompt injection — An attack in which untrusted content consumed by an agent carries instructions the user never issued.
• Lethal trifecta — Simon Willison's model of the dangerous combination of untrusted content exposure, private data access, and external communication in a single agent.
• Harness — The combination of feedforward context and feedback tooling built around a coding agent to raise the probability of correct output and catch incorrect output.
• Structural test — A deterministic test asserting architectural constraints such as which layers may import what; ArchUnit, Spring Modulith, and dependency-cruiser are examples.
• Codemod — A programmatic, deterministic source transformation; OpenRewrite recipes are the example given.
• Language server — A process that understands a language's semantics and exposes operations such as rename-symbol, usable by an agent instead of raw text editing.

Reference: Birgitta Böckeler, State of Play: AI Coding Assistants, QCon London 2026, InfoQ.