Bryan Oliver's 50-minute presentation, recorded at QCon AI New York 2025 and published by InfoQ on July 10, 2026, asks what chaos engineering must become when a workload is a distributed training job running on tightly coupled, expensive GPUs. Oliver is an engineer and author who had spent the preceding three years focused on platforms, GPU infrastructure, and cloud-native systems at Thoughtworks. He explicitly frames the field as young, fast-changing, and too complex for anyone to claim deep experience at its current scale.
These notes preserve that caution. Numerical claims below are figures Oliver gave in the presentation, not independently reproduced benchmarks.
The Failure Model Is Different
Traditional chaos experiments often assume stateless workloads, commodity hardware, one conventional network, and CPU-oriented faults such as exhausted memory, disk, or threads. Oliver contrasts that with distributed GPU training:
- Training has state, including model weights and checkpoints.
- A job may require a fixed number and arrangement of GPUs working in sync.
- The cluster has both a conventional network and a GPU communication fabric.
- Relevant faults include ECC memory errors, XID GPU fault codes, thermal throttling, VRAM exhaustion, and broken GPU-to-GPU links.
- Capacity held idle for failover is unusually costly.
He groups the GPU-specific test surface into four categories: simulated hardware faults, resource exhaustion, network faults, and workload disruption. He also cautions that, at presentation time, general-purpose chaos tools did not yet understand CUDA or DCGM, while acknowledging that vendor support was changing.
Inference is closer to the traditional model because requests can be stateless, model weights are read-only, and replicas can be load-balanced. Training is the hard case: one failed participant can stop a collective operation and put the whole job at risk.
Oliver uses current hardware to establish the economic context. He describes an H100 configuration with eight 94 GB GPUs and 752 GB of aggregate VRAM, and a GB200 NVL72 rack acting as a much larger connected GPU system with roughly 13–14 TB of aggregate memory. He describes the latter informally as acting like one GPU and gives approximately $3 million as a variable, rough price for one such rack. These are Oliver's scale-setting descriptions, not a hardware specification or durable price list; the point is that poor placement, long pauses, and unused spares can waste consequential capacity.
Why Placement Is Part of Reliability
Topology-aware scheduling
GPU collectives such as Allreduce involve every participating GPU, so the slowest participant constrains the operation. Placement must account for more than region and availability zone: rack, fabric, switch path, and contiguous blocks of healthy GPUs affect communication performance.
Oliver reports an example in which a topology-aware Allreduce took about 20 ms versus 40 ms for a default placement. That is a 50% latency reduction (or 2x speedup); Oliver calls it a 46% improvement in the talk, but does not explain the different calculation. He also contrasts links in the broad range of 100–600 Gb/s, depending on placement and interconnect. These are presentation examples, not universal expectations; topology, hardware, operation size, and measurement method determine actual results.
This makes health information an input to scheduling rather than merely an alert after deployment. A scheduler may need to avoid degraded hardware while still finding the contiguous GPU shape required by a job.
Performance variability
Nominally identical GPUs do not necessarily deliver identical performance. Oliver attributes variability to factors including cooling, power, and hardware differences. He discusses variability-aware scheduling as an emerging approach: classify observed GPU performance and place jobs according to their sensitivity and priority. A laggard-sensitive collective should receive a uniformly healthy block, while a workload tolerant of uneven performance might use a degraded block.
The proposal requires both infrastructure telemetry and measurements of how each job type performs on that infrastructure, not merely a static health label.
This approach depends on telemetry that links infrastructure health to job throughput. Oliver calls that correlation difficult with current tooling and describes variability-aware scheduling as bleeding edge rather than a turnkey capability.
NUMA alignment
Non-Uniform Memory Access (NUMA) determines whether a GPU reaches CPU cores and system memory through a local path or crosses sockets. Local placement offers a shorter PCIe path; remote placement can add latency and reduce bandwidth for data loading, preprocessing, checkpoint writes, and training.
Oliver says NUMA misalignment can degrade GPU performance by up to 40%, and in
one scheduling illustration describes roughly a 50% hit when a two-GPU request
is split across NUMA nodes. He warns that the default Kubernetes scheduler does
not, by itself, understand the required CPU, memory, and GPU alignment. Those
figures are scenario-dependent claims from the talk, but the engineering lesson
is concrete: a pod being Running does not establish efficient locality.
Checkpointing without stopping the fleet
Synchronous checkpointing pauses workers. Oliver estimates that writing a checkpoint for a 100 GB model can take up to a minute; doing so every ten minutes would spend about 10% of time checkpointing. He presents asynchronous checkpointing as the alternative: workers stage state independently in what he describes as milliseconds, training continues, and a checkpoint is committed only when its writes finish.
Asynchrony removes the global pause but introduces more states to test: partial writes, full volumes, slow storage links, and a job failure while a checkpoint is incomplete.
Why Ordinary Network Chaos Misses the Fabric
Remote Direct Memory Access (RDMA) allows one machine to access another's memory without the normal kernel networking path. GPU communication may use NCCL over technologies such as RoCE or InfiniBand; NVLink and NVSwitch provide additional high-speed GPU interconnects.
That bypass matters operationally. A conventional packet experiment built on
iptables or Linux traffic control cannot automatically observe or disrupt
native RDMA traffic because it is not traversing the expected kernel path.
Oliver identifies lower-level libraries such as libibverbs and UCX as relevant
when building custom observability.
The implication is precise: testing the service network does not prove the training fabric is resilient. Teams must identify which path a collective actually uses before selecting an injection mechanism.
Seven Fault-Injection Strategies
Oliver presents seven experiments. The transcript labels the first two, then groups three network mechanisms before calling checkpointing number six and NUMA number seven. The following numbering makes that implied sequence explicit.
1. Simulate GPU fault telemetry with DCGM
NVIDIA Data Center GPU Manager (DCGM) has an error-injection framework that can publish synthetic field values for a selected GPU, simulating how DCGM's APIs report conditions such as XID errors, ECC errors, thermal or power excursions, and NVLink errors. It does not physically induce those hardware failures. Oliver's examples include a GPU falling off the bus, stopping processing, memory page faults, and ECC page retirement. He credits Rob Skillington of Chronosphere for this experiment in their joint work.
The experiment is only useful if it tests the full response chain:
- Did monitoring detect the injected condition?
- Did the correct alert fire?
- Did the scheduler stop placing work on the affected GPU?
- Did the running job checkpoint and recover?
This single synthetic signal can therefore exercise DCGM-based observability, alerting, and any scheduler response wired to that telemetry. Whether it can exercise a running job's checkpoint and recovery path depends on the injected field and the platform's response; the injection itself does not make the GPU fail.
2. Saturate a GPU with GPU Burn
Oliver proposes running the containerized GPU Burn stress tool on selected GPUs to probe VRAM out-of-memory handling, CUDA-context behavior, memory fragmentation, thermal management, power delivery, and compute saturation. Co-locating it with a training workload creates a noisy-neighbor scenario.
Because this deliberately drives hardware hard, his safety warning is important: scope experiments carefully and do not damage expensive equipment. A successful test should define limits and an abort condition before applying load.
3. Change NCCL behavior
NCCL environment variables can disable transports, add overhead, or force specific algorithms. This is the easiest network-level technique Oliver describes because it modifies the training process without requiring a custom kernel program. It can expose assumptions about collective performance and fallback paths.
4. Apply traffic control to IP over InfiniBand
Where traffic is IP over InfiniBand, Linux traffic control can inject latency, packet loss, or bandwidth limits. This tests that IP path, not arbitrary native RDMA communication, so the transport distinction must remain explicit.
5. Instrument native RDMA with eBPF
For communication that bypasses the normal kernel network path, Oliver proposes eBPF hooks around send and receive events. He characterizes this as the hardest and potentially most powerful option: it requires custom programs and kernel verification, but can target multi-node training and collective resilience at the fabric layer.
6. Break checkpointing
Checkpoint experiments include killing a synchronous checkpoint, filling an asynchronous checkpoint volume, and partitioning or slowing the storage network. The expected result is not merely an error message. The team needs to establish whether incomplete state is rejected, the last valid checkpoint remains usable, and the job can make a bounded recovery.
The last sentence is synthesis from the failure modes, not a result Oliver reports demonstrating in the talk.
7. Disturb NUMA locality
Using numactl or related APIs, a test can force remote memory allocation,
create memory pressure on a NUMA node, introduce cross-socket CPU saturation,
or stop the Kubernetes topology manager. These experiments target silent
degradation: GPU utilization, memory, power, and temperature may all appear
normal while remote memory access, PCIe bandwidth, or the data loader slows the
job.
Oliver recommends adding NUMA statistics and events, PCIe bandwidth, and training throughput when a test reveals this blind spot. The objective is to observe delivered work, not only device activity.
The Observability Feedback Loop
Oliver's central operating loop is:
- Inject a fault.
- Determine whether the platform detected it.
- Add a metric or alert when it did not.
- Verify the scheduler or recovery response when it did.
- Document the result and repeat.
This changes observability from a passive inventory of available metrics into a system validated against known failure modes. It can also produce the health baselines needed for topology- and variability-aware placement before customer jobs encounter weak hardware.
Basic device metrics remain useful: utilization, memory use, power, and thermal state. They are insufficient on their own. Oliver highlights the difficulty of correlating a Kubernetes pod with a particular GPU and connecting GPU health to job performance. Training throughput and collective latency can reveal harm that a healthy-looking device dashboard misses.
For execution at scale, Oliver mentions three paths: Kubernetes cron jobs behind an internal API and scheduled through CI/CD, custom experiments executed by Chaos Mesh or LitmusChaos, and GPUDragon, an open-source project that he said Thoughtworks, Chronosphere, and others were starting. At presentation time he described the GPUDragon repository as empty, so it should be treated as an announced direction, not an available implementation demonstrated by the talk.
Synthesis: A Practical Experiment Contract
The presentation supplies fault mechanisms; the following is my synthesis of how to make them safe and decision-oriented.
Each GPU chaos experiment should state five things before it runs:
- Steady state: a job-level measure such as samples per second, collective latency, or checkpoint age, plus relevant device and fabric metrics.
- Fault boundary: the exact GPU, node, NUMA domain, transport, and workload affected. “Add network latency” is ambiguous when control-plane IP and RDMA fabric traffic follow different paths.
- Expected response: detection deadline, alert, quarantine or scheduling action, checkpoint behavior, and recovery-time target.
- Safety controls: blast radius, maximum thermal or power condition, automatic abort threshold, and confirmation that a synthetic mechanism does not physically damage hardware.
- Placement consequence: whether the result changes a health score, topology label, admission rule, or scheduler decision. A finding that never reaches placement leaves the next job exposed.
This contract also separates degradation from failure. Killing a node proves a visible outage path; forcing remote NUMA memory or slowing one collective participant tests whether the platform notices expensive work continuing below its expected rate. For GPU clusters, that silent middle state may be the more important target.
Key Lessons
- Treat GPU training as a stateful distributed system whose hardware topology is part of application behavior.
- Test the actual communication transport. Conventional network chaos does not automatically cover native RDMA.
- Correlate device health with job throughput and collective latency; utilization alone can make a degraded system look healthy.
- Validate that fault signals influence scheduling, not just dashboards.
- Exercise checkpoint corruption, capacity, and storage-network failures, not only worker termination.
- Start with a single bounded DCGM, network, or checkpoint experiment and build the observability loop incrementally.
- Preserve Oliver's uncertainty: hardware prices, performance figures, tools, and scheduling techniques in this area change quickly.
Canonical source: Bryan Oliver, "Chaos Engineering GPU Clusters", InfoQ (QCon AI New York 2025 presentation and transcript, published July 10, 2026).