Autonomous Data Products: Rethinking Data Architecture for GenAI

Most GenAI projects do not fail because the model is bad. In Jörg Schad's experience the consistent failure mode is everything around the model: the operations, the economics, and above all the question of how a system actually gets access to real enterprise data. His thesis is that the data layer needs the same kind of standardization that Docker and Kubernetes brought to microservices — encapsulate data into self-contained units, expose them through clean interfaces, and then automate the governance around them.

Schad is head of engineering at Nextdata and previously CTO at ArangoDB, with a background in distributed systems including early work on Apache Mesos and Kubernetes. He presented this 53-minute talk at QCon AI New York 2025; InfoQ published the recording and transcript on July 24, 2026. The talk has no audience Q&A section, and it ends with a demo of his employer's product — a commercial interest worth keeping in mind when reading his claims about the pattern.

What You Will Learn

• Why the hard part of production GenAI is data access rather than model choice.
• What "context rot" is and why exposing more tools to an agent can reduce accuracy.
• What an autonomous data product is, and how its sense/transform/verify/promote lifecycle differs from a scheduled pipeline.
• Why running data quality checks before promotion matters more for agents than for humans.
• How multi-modal output ports keep SQL, file, and embedding representations of the same dataset in sync.
• How policy-as-contract enforcement removes the governance team as a review bottleneck.
• How progressive tool discovery over MCP narrows an agent's tool surface to the task at hand.

The Prototype-to-Production Cliff

Schad describes a pattern he says he has seen repeatedly in large enterprises. A consultancy or internal team builds an impressive notebook on a small, clean sample dataset and demos it to senior management. The project then stalls when someone tries to connect it to the actual data estate: new source systems, unfamiliar access paths, other teams to negotiate with, and data that looks nothing like the sample.

What has changed with GenAI, in his view, is not the failure mode but its speed and scale. Previously a data scientist sat between the data and the model, and that person would often notice when something looked wrong. The stated ambition now — the one he attributes to CTOs — is a fleet of autonomous agents consuming the data estate directly, which removes the human sanity check while multiplying the number of access paths that must be safe. He decomposes the resulting problem into four pillars, worth treating as a design checklist:

Data 2.0: The Hairball and Its Owners

Schad presents the current state from two angles. Organizationally, delivering one GenAI initiative pulls in platform engineers, data product owners, an ingestion team, data architects, data stewards, data engineers, data governance, and finally the consumers. Each persona owns one slice, must coordinate with the others, and data and metadata end up split across those boundaries. Architecturally, the same fragmentation appears as what he calls the "data management hairball": separate tools for cataloging, governance, data quality (he names Monte Carlo, Soda, and Great Expectations), lineage, and one or more ETL systems, all of which some team must weave together. His point is not that these tools are bad — he says so explicitly — but that stitching them into a coherent architecture is a per-enterprise effort with no shared abstraction.

His preferred analogy is the LLVM compiler stack: many source languages on top, many target architectures below, and a standardized intermediate representation in the middle where optimization happens. Kubernetes did the same for microservices by standardizing container images, runtimes, networking, and storage so multiple implementations could exist above and below the interface. Data, he argues, lacks that waist in the hourglass, so complexity keeps compounding across independent dimensions — storage location and jurisdiction, access mode (SQL, RAG, MCP, embeddings, BI tools), processing engine (Spark, Polars, Snowflake, Iceberg lakehouse), consuming application, and the differing skills of each persona.

Context Rot: Why More Access Makes Agents Worse

The natural first move most teams make is to expose the data warehouse behind a single MCP endpoint, giving the agent everything. Schad argues this is precisely wrong, and grounds it in context rot: as input tokens or available tools grow, LLM task performance degrades. He cites LangChain benchmarks showing accuracy declining as progressively more tools are exposed to an agent. The mechanism is intuitive once stated. An agent with access to only task-relevant information sits at a sweet spot; beyond that, selection itself becomes the hard problem, the model cannot tell which of a hundred near-identical tools is right, and hallucination increases. His analogy is onboarding a new employee — you hand them the runbook for their first task, not the entire company handbook.

For context, "context rot" is a term of art in the LLM-engineering community rather than a formal metric, and the LangChain benchmark is a specific experiment rather than a universal law. The directional finding — that tool-selection accuracy degrades with tool count — is widely reported, but the threshold is model- and task-dependent. Treat the sweet spot as something to measure for your own agent, not a fixed number.

Autonomous Data Products

Schad's answer follows the historical recipe for taming complexity in computing: encapsulate into manageable units, abstract behind clean interfaces, then automate. Containers did this for compute; a data product is meant to do it for data.

The concept comes from Zhamak Dehghani's Data Mesh book: a single addressable asset that bundles data and metadata, so that answering "how do I get this dataset?" means handing over one reference rather than explaining which team owns which Snowflake schema. Schad notes the term has been diluted — people now call a dashboard or a table a data product — so he uses autonomous data product for the stricter definition.

The distinguishing feature is that it is a running process, not a static artifact. Each data product has a runtime controller he calls a kernel, by analogy with an operating system kernel, that manages its lifecycle:

1. Sense inputs. The product watches upstream sources and upstream data products for changes, and applies a configured policy for when to act.
2. Transform. It triggers work in the underlying compute layer. The data product does not replace Spark or a Databricks SQL transformation; it encapsulates it as a logical layer on top.
3. Verify. Data contracts and quality checks run against the output.
4. Promote. Only after checks pass does the output become visible to downstream or external consumers.

That fourth step carries most of the weight. In a conventional stack, quality tools alert after data is already readable: the agent has answered questions and taken actions on bad data, and the fix is retroactive. Schad's argument is that this was tolerable when a human might notice something looked wrong, and is not tolerable for autonomous consumers, which he says are very unlikely to detect that data looks wrong. Beyond the lifecycle, a data product exposes an OpenAPI endpoint covering its models, inputs, outputs, current status, and event history, so discovery and observability do not require jumping between systems. Sharing becomes handing over a URL, with access resolved by permissions.

Multi-Modal Output Ports

The other structural idea is separating the semantic model of a dataset from its physical representations. Discovery should be about what data exists; consumption format is a second, independent decision. A data analyst wants SQL, a training pipeline wants files, an agent wants embeddings or MCP.

In the Data 2.0 world each of those needs its own pipeline, and Schad points at a failure he says he has seen often: two pipelines drift, one breaks or runs slower, and the SQL table is current while the embedding index is two days stale. Anything consuming both — increasingly common in RAG systems that combine relational filters with vector search — then produces inconsistent answers. With output ports, a new requirement adds a port to an existing data product rather than a parallel pipeline, and all outputs are updated in lockstep.

Architecture And Data Flow

flowchart TD
    U1[Upstream data product] --> S
    U2[External source: PDFs, APIs] --> S
    S[Sense inputs and run policy] --> T[Transform on Spark / SQL engine]
    T --> Q{Data contracts
and quality checks}
    Q -- fail --> H[Hold: not promoted, owner alerted]
    Q -- pass --> P[Promote outputs]
    P --> O1[SQL port: Snowflake]
    P --> O2[File port: Iceberg on S3]
    P --> O3[Embedding port: vector store]
    P --> O4[MCP port: API functions]
    O4 --> G[Mesh MCP gateway]
    G --> A[Agent / chatbot]
    POL[Central policy repository] -.enforced at deploy and runtime.-> P

His worked example is a customer feedback data product built from two upstream products with very different characteristics: structured online reviews arriving close to continuously, and unstructured feedback from physical stores that arrives as scanned PDFs once a week. Because arrival rates differ by orders of magnitude, the sensing policy is a genuine design decision — updating on every review may add little information, while updating only on the weekly batch may be too slow. He contrasts this with the large hourly Spark jobs he has seen running regardless of whether anyone reads the output, noting drily that this is excellent for the compute vendor's revenue and poor for the enterprise, since the report may only be read monthly.

Policy as Contract Instead of Manual Review

The governance tension is familiar: domain teams want to ship, and the CISO wants a guarantee that no PII reaches an LLM. The usual resolution is human review before release, which makes the governance team a serialized bottleneck.

Schad's alternative is to define policies centrally, up front, in a shared contract repository, and require every data product deployed to the mesh to satisfy them. His pseudocode example: if a data product has an MCP output port or an embedding output port, it must implement the predefined PII check. The check on deployment is directly analogous to a Kubernetes admission controller, with one difference he highlights — enforcement also happens at runtime, so policies added later are applied to already-deployed products rather than only to new ones.

The trade-off is that this only works for policies that can be expressed mechanically. "No PII in embedding ports" is checkable; "this aggregation is commercially sensitive in a way our contracts prohibit" generally is not. The pattern shifts the governance team's work from reviewing outputs to authoring and maintaining executable policies, which is a real change in required skills rather than a pure efficiency gain.

Progressive Tool Discovery

Making it easy to publish MCP output ports creates its own problem: a mesh with hundreds of data products produces hundreds of tools, which is exactly the context rot scenario. Schad's answer is progressive tool discovery, built on a property of the MCP protocol.

MCP exposes a list tools endpoint, and — critically — a client can subscribe to it, so the tool list is dynamic rather than fixed for the session. The flow he describes works like this:

1. The whole mesh sits behind a single MCP gateway endpoint.
2. On connecting, the agent's list tools call returns exactly one tool, along the lines of discover_tools.
3. The agent calls it with the task it is trying to accomplish.
4. The subscribed tool list updates to include only the tools relevant to that task — for example get_customer_feedback.
5. The agent uses those tools, gets its data, and reasons over it.
6. After the loop completes, the exposed tool set is reset or re-filtered.

Two properties fall out of this. The agent's decision space stays small at every step, and because everything routes through one gateway, authorization is enforced in one place rather than reimplemented per team. Schad is careful to say he expects MCP to need several protocol iterations before it is truly useful, and that the pattern applies equally to whatever protocol succeeds it.

The Demo: What This Looks Like Concretely

The demo shows Nextdata's implementation, which Schad frames as one implementation of the pattern — Kubernetes, Mesos, and OpenShift being his analogy for multiple implementations of one orchestration idea. Evaluating a data product as a consumer means seeing, in one place: description and freshness, owner contact, storage and compute services, the transformation code, a trust summary covering access frequency and quality check pass rate, the semantic data models, the API functions the product declares it answers well, the available output ports (Snowflake, Iceberg on S3, and Pinecone in the demo), an access request flow that can route to a Jira approval or an automated process, the data contracts and their recent run status, and which central policies the product satisfies. Lineage between products is rendered directly in the domain view rather than reconstructed by a separate tool.

He then connects Claude Desktop to the mesh gateway and asks two kinds of question. Some are metadata questions — which data products exist, which were accessed most last week — answered by a discovery system data product. Others are data questions, where the router selects the relevant data product's tools. The same interface serves both, and a user only ever sees the data products and tools their permissions allow.

Trade-offs And Limitations

The talk itself raises only two of these caveats explicitly — that MCP needs further iteration, and that the demo is one implementation among possible others. The remainder are my own reading of what the pattern costs, not claims Schad made.

• Vendor context. The pattern is presented by the VP of engineering at a company selling an implementation of it. The demo is not independent evidence that the pattern works at scale elsewhere.
• No production metrics. The talk contains no benchmarks, adoption numbers, or before/after latency or cost figures. Claims like "we've seen this fail often" are stated as experience, not measurement.
• Encapsulation adds a layer. A data product wraps Spark or SQL rather than replacing it. Teams take on a runtime controller, a contract repository, and a gateway in exchange for the abstraction, and Schad does not discuss the operational cost of running that control plane.
• Migration is unaddressed. Nothing in the talk covers how an enterprise with an existing hairball incrementally reaches this model, which is usually the binding constraint.
• Policy expressiveness. Automated contracts handle mechanical rules well; judgement-based governance concerns do not disappear.
• Blocking promotion has a cost. Holding data until checks pass trades freshness for correctness. For latency-sensitive consumers that trade may not be acceptable, and the talk does not discuss what downstream consumers should do while a product is held.
• Progressive discovery depends on the discovery step. If discover_tools returns the wrong subset for an ambiguous task, the agent cannot see the tool it needs. The failure mode shifts from too many tools to a bad retrieval decision, and it is quieter.
• MCP is immature. Schad says so himself. Building a gateway on a protocol he expects to change is a real forward-compatibility risk.

Practical Takeaways

• Before blaming the model for a stalled GenAI project, audit the data access path: how many systems, teams, and credentials stand between the prototype and production data.
• Measure your agent's accuracy as a function of exposed tool count. If accuracy falls as you add tools, tool filtering is a higher-leverage fix than prompt tuning.
• Do not expose the warehouse behind one broad MCP endpoint. Expose narrow, task-shaped tools, and let a discovery step select them.
• Move data quality checks from post-hoc alerting to pre-publication gates for any dataset that autonomous consumers read. Agents will not notice bad data.
• If you maintain a SQL table and a vector index of the same dataset via separate pipelines, treat the drift between them as a known defect and consider driving both from one job.
• Make the update cadence of a dataset an explicit, owned decision tied to consumption, rather than a default hourly schedule.
• Encode the governance rules that can be automated as deployment-time and runtime checks, so review effort concentrates on the rules that genuinely need judgement.
• Route agent data access through a single gateway so authorization is enforced once rather than per team.

Key Terms

• Context rot — degradation of LLM task performance as input tokens or available tools increase, driven by the model's difficulty selecting relevant information.
• Data mesh — a decentralized data architecture, from Zhamak Dehghani's book of the same name, in which domain teams own and publish their data as products rather than routing everything through a central data team.
• Autonomous data product — Schad's stricter definition of a data product: a running process with a runtime controller that senses inputs, transforms, verifies against contracts, and promotes multi-modal outputs.
• Kernel (in this context) — the per-data-product runtime controller managing that lifecycle, named by analogy to an OS kernel.
• Output port — one physical representation of a data product's semantic model, e.g. a Snowflake table, an Iceberg dataset, a vector index, or an MCP endpoint.
• Data contract — a machine-checkable promise about a dataset's shape, quality, or content, verified before promotion.
• Policy (mesh-level) — a centrally defined invariant every data product must satisfy, enforced at deploy time and at runtime, analogous to a Kubernetes admission controller.
• MCP (Model Context Protocol) — a standard protocol for exposing tools and resources to LLM agents, including a subscribable list tools endpoint.
• Progressive tool discovery — starting an agent with a single discovery tool and dynamically revealing only the tools relevant to its stated task.
• Lineage — the recorded provenance of a dataset; Schad's point is that encapsulation makes it intrinsic to the architecture rather than something reconstructed by a separate tool.

Schad's closing framing is deliberately implementation-agnostic: the takeaway is a posture, not a product. Keep GenAI use cases domain-centric, expose the smallest sufficient slice of data and tools, and put quality and governance gates before consumption rather than after it. Whether autonomous data products are the right container for that is an open question the talk asserts rather than demonstrates — but the underlying diagnosis, that the data access layer is where GenAI projects actually break, is well argued and matches what most teams find when the notebook meets the warehouse.

Reference: Jörg Schad, Autonomous Data Products for the Autonomous Era: Rethinking Data Architecture for GenAI, QCon AI New York 2025, published by InfoQ on July 24, 2026.