Cypher-X

An AI product produces two outputs: the response shown to a user and evidence about what the user did next. Ben O'Mahony's presentation explores how to use that second output. His example is a proactive, AI-powered Language Server Protocol (LSP) that records whether a developer applies, regenerates, or dismisses a suggested code fix.

O'Mahony's central claim is that these ordinary product interactions can form a data flywheel. OpenTelemetry connects a model request and its result to a later user action; an extraction job converts selected traces into training data; and a smaller model is fine-tuned to reproduce useful behavior at lower cost and latency. He presents this as a personal prototype and a candidate platform capability, not as a production-proven autonomous training system.

What You Will Learn

Why an AI LSP complements rather than replaces deterministic language tools.
Which generative-AI operations OpenTelemetry can make observable.
How workflow actions become positive, weak-positive, and negative signals.
How to extract complete traces into an illustrative JSONL dataset.
How distillation differs from supervised fine-tuning, preference learning, and reinforcement learning from human feedback (RLHF).
How to generalize one feedback loop into shared platform infrastructure.
Where evaluation, context, privacy, data quality, and model quality constrain the approach.

Put Semantic Analysis Where Developers Work

The Language Server Protocol separates an editor client from a process that understands a programming language. Editors such as VS Code, Neovim, and Emacs can request definitions, references, documentation, diagnostics, and code actions from the same language server.

Traditional servers such as Pyright and rust-analyzer parse code, construct an abstract syntax tree (AST), infer types, and apply deterministic rules. They are fast, precise, and repeatable. O'Mahony deliberately retains them because an LLM should not displace tools that already check syntax and types reliably.

His AI LSP instead targets judgments that are difficult to express as fixed rules: a function name no longer reflects its behavior, error handling appears fragile, or custom code duplicates a standard-library function. Because the model reads text rather than requiring a language-specific parser, the same prototype can inspect multiple programming languages and Markdown.

On each file save, the LSP proactively analyzes the document and places diagnostics in the editor's normal problem interface. A diagnostic and its fix need not point to the same location. The model can offer multiple fixes, which the developer may apply or potentially combine. O'Mahony considers this placement important: engineers need well-designed context for making decisions, just as agents need well-designed model context. A separate chat or a huge generated pull request can impose unnecessary cognitive and review costs.

The interface exposes three consequential actions:

Apply fix: strong positive. O'Mahony interprets this to mean that the diagnosis and fix were useful enough to accept. It does not prove that the resulting code is correct.
Regenerate fix: weak positive. The diagnosis may be useful even though the proposed fix was not. The action does not reveal why the developer wanted a different answer.
Dismiss: strong negative. The suggestion was not useful in that situation, but the cause could be its quality, timing, or the developer's preference.

The extraction job excludes diagnostics that receive no action. Silence is not a reliable label: the developer may not have seen the diagnostic, may have deferred it, or may simply have ignored it.

In Q&A, O'Mahony said he uses the LSP broadly, like a linter. It is not always useful, but its whole-document analysis can reveal an issue outside the feature currently occupying the developer's attention. This differs from autocomplete, which primarily predicts what comes next.

Observe the Entire Interaction

OpenTelemetry (OTel) is a vendor-neutral framework for generating and exporting traces, metrics, and logs. A span records one unit of work, including its timing, attributes, events, and relationship to other spans. A trace joins those spans into one end-to-end execution, including work across services or agents.

The generative-AI semantic conventions discussed by O'Mahony cover:

Input and output events, including system, user, assistant, and tool messages.
Token counts, operation and request duration, time to first token, and time per output token.
Model operations such as chat completion and multimodal generation.
Agent workflow and tool-execution spans.
Vector operations used in retrieval-augmented generation (RAG).

O'Mahony recommends instrumenting every agent from its first useful version. His reason is broader than training: traces make prompts, context, tool calls, latency, and workflow failures debuggable. He reports that framework auto-instrumentation captures most model activity; the application then needs a small amount of custom instrumentation for meaningful user actions.

His prototype uses Pydantic AI with Logfire instrumentation. An OpenTelemetry Collector writes JSON spans to local disk and also forwards them to Opik. He stresses that these are replaceable choices: Logfire, Honeycomb, Datadog, or another OTel-compatible backend can participate in the same pattern.

Architecture And Data Flow

The useful unit is a complete trace, not an isolated feedback event. The trace must connect the source document and model input to the generated diagnostic, candidate fix, and eventual developer action.

flowchart LR
    A[Developer saves file] --> B[AI LSP]
    B --> C[Frontier model call]
    C --> D[Diagnostic and candidate fixes]
    D --> E{Developer action}
    E -->|Apply| F[Strong-positive span]
    E -->|Regenerate| G[Weak-positive span]
    E -->|Dismiss| H[Strong-negative span]
    C --> I[OTel model spans]
    F --> J[OTel Collector]
    G --> J
    H --> J
    I --> J
    J --> K[Trace backend or JSON files]
    K --> L[Query complete labeled traces]
    L --> M[Filter and build JSONL]
    M --> N[Fine-tune and evaluate SLM]
    N --> O[Guarded deployment]
    O --> D

O'Mahony describes the extraction as deliberately simple:

Export spans to a location the data pipeline can query.
Select complete traces containing an apply, regenerate, or dismiss span.
Exclude diagnostics with no observed action.
Recover the model input, output, and behavioral signal from each trace.
Write one training example per JSONL line.

A Concrete Telemetry-to-JSONL Example

The presentation states that the result is simple JSONL but does not publish an exact record schema. The following is therefore supplementary engineering guidance, not O'Mahony's code or dataset. It illustrates the minimum linkage the described pipeline needs:

{
  "trace_id": "4f1c...",
  "messages": [
    {
      "role": "system",
      "content": "Review the saved file and propose diagnostics."
    },
    { "role": "user", "content": "def load_config(path):\n    ..." },
    {
      "role": "assistant",
      "content": "The broad exception hides parse failures. Catch the parser exception and preserve its cause."
    }
  ],
  "candidate_fix": "except ParseError as exc:\n    raise ConfigError(path) from exc",
  "action": "regenerate_fix",
  "signal": "weak_positive"
}

For supervised fine-tuning, an extractor might emit only applied examples as input-target demonstrations. For preference learning, it could pair a rejected or superseded candidate with the subsequently applied candidate for the same input. That transformation is supplementary advice: the talk does not say that O'Mahony constructed such pairs or identify the loss function used in his MLX run.

Before adopting any schema, remove secrets and unnecessary source content, retain model and prompt versions, and preserve trace-level provenance so a bad example can be audited or deleted. These are also supplementary production recommendations rather than reported features of the prototype.

The presentation uses “distillation” as the broad description of transferring frontier-model behavior into a smaller model, and says behavioral signals could support RLHF or another human-feedback fine-tuning method. It does not establish that O'Mahony's local experiment implemented RLHF. Keeping the following terms separate prevents the data source, training objective, and overall goal from being mistaken for one another.

Model distillation is a teacher-to-student transfer goal: train a smaller model to reproduce selected behavior of a stronger model. Frontier-model diagnostics and fixes provide candidate teacher outputs, while user actions help identify behavior worth transferring.
Supervised fine-tuning (SFT) updates a pre-trained model using input and desired-output examples, usually with next-token prediction. Applied fixes or other curated responses can serve as target demonstrations.
Preference learning trains from comparisons or ranked alternatives rather than one target. For the same input, an applied response could be preferred over a dismissed or regenerated response. Direct preference optimization (DPO) is one possible method.
RLHF is a specific multi-stage family of methods: collect human preferences, train a reward model, and optimize a policy with reinforcement learning, commonly with constraints. Behavioral comparisons could contribute preference labels, but the talk provides no reward-model or reinforcement- learning details.

These categories overlap but are not synonyms. A project can distill a teacher through SFT alone. It can perform preference learning without reinforcement learning, for example with DPO. It can also run RLHF for personalization without the student being smaller than the teacher. “Human feedback” describes where a signal came from; it does not by itself make a training run RLHF.

O'Mahony reports starting from an open-source coding model in approximately the 7-billion to 13-billion parameter range, rather than training code understanding from scratch. He used MLX locally and says fine-tuning took a few hours; later, he estimates that a 7-billion-parameter run might take one or two hours. The transcript does not name the base model, hardware, hyperparameters, objective, dataset split, or exact example count used for that run. His references to hundreds of interactions after a month and “several thousand examples, hopefully” are guidance and aspiration, not a consistent measured dataset size.

He warns against over-training a narrow dataset. The desired result is a coding model adapted to a codebase, workflow, and user's recurring preferences, not a new general-purpose coding model. He reports roughly 80%-85% quality on complex, novel evaluations relative to the frontier model, while emphasizing that the student was not equally capable. This is an anecdotal prototype result, not a benchmark that should be generalized.

The Data Flywheel

O'Mahony describes a gradual loop rather than an immediate training project. On day one, ordinary coding yields only several interactions. After a week, there may be tens or hundreds and visible patterns in accepted and dismissed suggestions. After roughly a month, hundreds of labeled interactions may make a small experiment worthwhile.

The economic argument is that product use performs part of the labeling work. Instead of paying annotators to judge every sample in a separate interface, developers create signals while completing real tasks. A better or cheaper specialized model can increase usage, which creates more data for later model versions.

O'Mahony argues that workflow actions are often stronger signals than detached thumbs-up ratings. Applying a patch or exporting an answer to a document shows progress toward a product goal; copy and paste may be difficult to observe. Product designers can therefore create useful actions that both improve the experience and expose outcomes.

This does not make the labels free or objectively correct. Instrumentation, storage, governance, cleaning, evaluation, and training still cost money. Actions are proxies whose meaning depends on the interface and situation. The flywheel can amplify popular mistakes or one user's eccentricities unless its inputs and model releases are evaluated independently.

From Prototype to Platform Capability

O'Mahony generalizes the experiment into shared capabilities for many agent teams:

Standard OTel instrumentation for model, agent, tool, and feedback spans.
Configuration mapping application actions to a positive-negative scale.
A shared trace extraction and JSONL generation pipeline.
Fine-tuning and model-serving infrastructure.
Reusable evaluation tooling and deployment guardrails.

An internal platform could let a team list its strong-positive, weak-positive, and negative span names in configuration. It could go further by offering a small SDK, wrapper, or decorator that emits a standard feedback span. O'Mahony emphasizes making this interface usable by software engineers, data engineers, and data scientists who are not model-training specialists.

Once the common pipeline exists, a new team instruments its agent, defines the meaning and weight of its user actions, and plugs into the infrastructure. A support agent, code-review assistant, and data-analysis agent will have different actions, but they can reuse the transport, extraction, training, and evaluation machinery.

At organizational scale, O'Mahony imagines traces crossing multi-agent workflows. Product knowledge learned from support interactions might inform a documentation generator, while traces identify the contribution of each sub-agent. This is a direction he proposes, not a demonstrated feature. Sharing training data between agents also introduces purpose, access, and ownership questions that a common telemetry layer does not solve.

Context, Retrieval, and Distillation Work Together

Fine-tuning is not O'Mahony's first recommendation. Instrumentation initially supports debugging and prompt iteration; prompt and context engineering should come before a more expensive training loop. RAG adds changing or codebase-wide facts at inference time. Fine-tuning changes persistent model behavior. These tools solve different problems and can be combined.

The prototype sends the whole current file to the model. In Q&A, O'Mahony said this will fail on a very large file, giving a hypothetical 20,000-line example. He is considering Tree-sitter, a parser framework with language grammars, to select syntax-aware regions more granularly.

He did not use RAG because his experiment involved relatively small codebases and partly existed to practice distillation. A multimillion-line legacy system would require substantially more context: AST structure, execution or call graphs, symbol definitions, information from other language servers, or a codebase retrieval index. He also mentioned possible internet lookup for prior art. Fine-tuning can internalize recurring behavior or instructions, but it cannot reliably provide current facts that never enter training or request context.

Evaluation Before and After Training

The presentation recommends turning collected interactions into evaluations even before distillation. O'Mahony used longer coding evaluations, mostly scored by an LLM judge, to observe some training progression. He found coding agents difficult to assess deterministically without a larger benchmark and described his coverage as small.

An LLM judge scales subjective assessment but can be biased and non-deterministic. The key source-backed conclusion is modest: his local model was worse than the frontier model but faster, cheaper, and local. A production team still needs to define acceptance thresholds, safety rails, and continuous checks before automating deployment.

Supplementary engineering guidance: for generated code, combine semantic judging with compilation, tests, linters, static analysis, patch applicability, and human review. Keep a held-out set that is not sampled into training, segment results by repository and task, compare against both the frontier model and the current production model, and monitor regressions after release.

Trade-offs And Limitations

A Smaller Model Gives Up Capability

O'Mahony does not claim that specialization makes the local model universally better. His reported 80%-85% result implies a quality trade-off on complex, novel work. The benefit is a model that is cheaper, faster, more portable, and less likely to repeat suggestions that its user habitually dismisses.

Labels Are Behavioral Proxies

Apply, regenerate, and dismiss have plausible meanings, not guaranteed ones. The dataset reflects active users and instrumented actions, while ignored outputs and rare tasks are missing. Personalization may encode taste rather than correctness. Negative examples are valuable and often hard to collect, but their cause must not be over-interpreted.

Proactive Analysis Has a Product Cost

Sending a model request on every save consumes tokens, which motivated the smaller model. Too many diagnostics can also interrupt the developer; O'Mahony added dismissal partly to protect flow. Any proactive tool must optimize for useful decisions, not maximum suggestion volume.

Privacy and Intellectual Property Are Unresolved

Traces can contain prompts, model outputs, source code, tool arguments, and internal intellectual property. In Q&A, O'Mahony said his personal solution is not to use the prototype on projects with such restrictions. He identified this as a major issue for a commercial product. He speculated that another LLM might mutate data, but immediately noted that this could reduce accuracy; he did not present it as a solved anonymization method.

Supplementary engineering guidance: collect only fields needed for the declared purpose; redact secrets before export; enforce access, retention, tenant, and deletion controls; obtain appropriate consent; and verify that licenses and organizational policies permit training. Local inference can help on projects that prohibit external AI calls, but it does not retroactively make an improperly collected training dataset safe.

The Prototype Is Narrow and Incomplete

The source code was not available at the time of the talk, although O'Mahony said he expected it to be available shortly. The prototype uses whole-file context on small repositories, has limited evaluation coverage, and leaves the exact training method unspecified. The platform and cross-agent learning ideas are architectural proposals rather than reported production outcomes.

Practical Takeaways

Instrument one useful agent with OpenTelemetry before planning model training; use traces first for debugging, prompt iteration, latency, and cost.
Add product actions whose meaning follows from the user's workflow, then record those actions as spans linked to complete model traces.
Keep apply, regenerate, dismiss, and no-action cases separate. Document what each signal can and cannot establish.
Build evaluations from production examples before using those examples to update a model.
Decide whether the data supports SFT demonstrations, preference pairs, or a more involved RLHF pipeline; do not call every feedback-based run RLHF.
Start from a capable pre-trained model and one bounded workflow. O'Mahony suggests collecting a few hundred interactions over about a month before a small distillation experiment.
Compare the resulting SLM with the frontier and current production models on quality as well as latency, cost, privacy, and portability.
Deploy behind guardrails and keep the feedback loop observable. Continuous data collection is useful only when dataset and model versions remain auditable.
If several teams repeat the pattern, centralize span conventions, extraction, evaluation, training, and serving while leaving each product responsible for the meaning and governance of its actions.

O'Mahony closes with a broader question: tools shape their users, so users and organizations should retain the ability to shape their tools in return. His preferred future is not one undifferentiated intelligence, but many local, portable, specialized models that people can continuously improve for their own workflows.

Key Terms

Language Server Protocol (LSP): A protocol through which an editor asks a separate language-aware process for diagnostics, definitions, references, and code actions.
Abstract syntax tree (AST): A structured representation of source code used by parsers, compilers, and static-analysis tools.
OpenTelemetry (OTel): Vendor-neutral APIs, conventions, and tooling for creating and exporting telemetry.
Span: A recorded unit of work with timing, attributes, events, and parent relationships.
Trace: The connected set of spans describing an end-to-end execution.
Implicit label: A training or evaluation signal inferred from behavior rather than entered in a dedicated annotation task.
Small language model (SLM): A model with fewer parameters and generally lower inference-resource requirements than a frontier model.
Model distillation: Transferring selected behavior or knowledge from a teacher model into a student, commonly a smaller model.
Supervised fine-tuning (SFT): Updating a pre-trained model on input and desired-output demonstrations.
Preference learning: Updating a model from comparisons or rankings among candidate outputs.
Direct preference optimization (DPO): A preference-learning method that directly trains on preferred and rejected responses without a separate reinforcement-learning stage.
RLHF: Reinforcement learning from human feedback, typically involving preference data, a learned reward model, and reinforcement-learning policy optimization.
Data flywheel: A loop in which product use creates data that improves the product and may encourage further use.
RAG: Retrieval-augmented generation, which supplies selected external information to a model at inference time.
LLM as judge: Using a language model to score another model's output.

Reference: From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry