Architecting a Centralized Platform for Data Deletion at Netflix

Deleting data sounds like the easy half of storage engineering. The speakers' central thesis is the opposite: in a large distributed system, deletion is a first-class architectural problem that has to balance three competing properties — durability, availability, and correctness — and it must be designed up front rather than bolted on after the first incident.

Vidhya Arvind is a Staff Engineer at Netflix and a founding architect of the Data Abstraction Platform. Shawn Liu is a Senior Software Engineer at Netflix working on consumer data lifecycle systems. They presented this 49-minute, 41-second talk at QCon San Francisco; InfoQ published the recording and transcript on June 4, 2026.

Arvind opens with a scenario most engineers recognize: running a destructive command and immediately typing pwd to find out where it just ran. She describes a late-night fleet deployment where an engineer issued rm -rf, and the days or weeks of cascading cleanup that followed. Her framing is that both sides of deletion carry cost. An unintended delete forces an incident response with real human cost — racing against time, stress, guilt, and fear. But not deleting also costs: storage bills grow, and customer trust erodes when data that should be gone is still present. The goal she sets is not to make deletion faster but to place, in her words, a hundred guardrails in front of the delete button.

What You Will Learn

• How TTL, hard deletes, and soft deletes are implemented differently by Cassandra, DynamoDB, EVCache, Redis, Elasticsearch, and RDS, and what each costs at runtime.
• Why deleting only the source record leaves dangling pointers across caches, search indexes, and object storage, and why Netflix chose asynchronous delete fanout instead.
• How tombstone accumulation and compaction storms turn a bulk delete into a latency and availability incident.
• Concrete mitigations: partition-level deletes, TTL jitter, resource-aware throttling, rate limiting, and exponential backoff.
• Why concurrent writes make deletion a conflict-resolution problem, and how client-generated timestamps, idempotency tokens, and compare-and-set restore correctness.
• The four-phase deletion lifecycle Netflix runs: identify, audit and validate, delete, and continuously monitor.
• How a journal of deleted payloads in S3 provides a 30-day recovery window and becomes the basis for customer trust in a central deletion team.
• The reported scale and outcomes, plus the open questions raised in the Q&A.

Why Deletion Is Hard Before You Even Start

The three pillars Arvind uses to organise the talk are worth defining precisely because the rest of the architecture is a negotiation between them.

Note that durability here is inverted from its usual meaning. Normally durability means a committed write survives; in a deletion context it means a committed delete survives. That inversion is the source of a whole class of bugs, because most storage engines are optimised to preserve data, not to guarantee its absence.

How Datastores Actually Delete

Arvind's first substantive point is that "delete" is not one operation. There are three broad mechanisms, and every engine implements them differently, with different performance, operational risk, and cost profiles.

Time to Live

A TTL lets you set an expiry at write time and delegate removal to the database. The critical detail is that expiry and physical removal are almost never the same event.

• Cassandra natively supports TTLs, but does not remove data immediately. Expired data is retained until the GC grace period passes, after which compaction can compact it away.
• DynamoDB lets you mark an item with an expiry attribute; background tasks then remove it.
• EVCache is an LRU cache — eviction is lazy rather than automatic on expiry.
• Redis supports TTL with periodic background cleanup.
• Elasticsearch does not support native TTLs; you use index lifecycle management instead.
• RDS does not support native TTLs either; you need scheduling or a mark-and-sweep job.

The common thread across all of these is a background process — compaction, vacuuming, or code you wrote yourself — that does the real work later. That deferral is where the cost hides.

The Hidden Cost of Deferred Removal

When compaction, vacuum, or background delete jobs run, they consume real resources. Arvind lists the observable symptoms: CPU spikes when there is a large backlog to delete; increased read latency because reads must scan tombstones alongside live data; read timeouts; and a storage footprint that grows rather than shrinks until cleanup completes.

Supplementary context, not stated in the talk: a tombstone is a marker record written in place of a delete in a log-structured storage engine. Because data files are immutable, you cannot edit a row out; you append a marker saying "this key is dead as of timestamp T". Reads must merge the marker with any older live versions to produce the correct answer, which is why tombstones make reads slower until compaction physically merges the files and drops them.

Hard Deletes

A hard delete issues an explicit delete command. Again the engines diverge:

• Cassandra writes a tombstone marker; the underlying data is immutable, so nothing is physically removed at delete time.
• DynamoDB deletes the item immediately.
• EVCache evaluates the eviction lazily; the slab allocation remains in place until some merging occurs.
• Redis behaves similarly, leaving fragmented memory that needs a merge process.
• Elasticsearch leaves segments unmerged until a background process runs.
• RDS needs vacuuming.

Soft Deletes

A soft delete is application-level: you set a deleted column or flag, then run a background job to physically remove the marked rows later. This gives you control over timing and an easy undo, at the cost of writing and operating the sweeper yourself, and of every read path having to filter on the flag.

Arvind's summary of the hidden cost of deletion is direct: it is high for Cassandra, Elasticsearch, and RDS, and low for DynamoDB and EVCache. The takeaway she draws is that everything has a cost and you have to look at it explicitly for each engine in your stack.

Ghosts: When Automation Falls Short

Automation is tempting — push the first domino and trust it to reach the end. Arvind's warning is that automation sometimes falls short, and the failure mode is resurrection: lingering data that reappears, zombie processes, and data that comes back from the dead.

She gives a concrete incident from the month before the talk. A Cassandra cluster had a misconfiguration that went unnoticed, and processes were not restarted for more than 24 hours. Once a node exceeds the GC grace period while down, the deletes it missed can no longer be safely applied — the tombstones that would have told it about them may already have been compacted away elsewhere. When the node came back up, previously deleted data reappeared. What began as an operator error became a cascading correctness problem: data that was supposed to be gone was live again, and the first task was simply identifying which data that was.

Supplementary context: Cassandra's gc_grace_seconds exists precisely to give every replica a chance to learn about a delete before the tombstone is discarded. A node offline longer than that window is why operational runbooks say to run a full repair, or replace the node, rather than simply restarting it.

Copies Everywhere: The Fanout Problem

Liu takes over to explain what makes this dramatically harder in practice: the same logical record does not live in one place. A primary Cassandra cluster may be the source of truth, but the same data is indexed in Elasticsearch for search, cached in EVCache for fast access, and stored in S3 for backup or analytics. Each copy exists for a legitimate reason — performance, redundancy, or analytics — but each is also a place the data has to be deleted from.

This raises the question Liu says most organisations cannot confidently answer: do you actually know where all your data is stored, and how it is connected and transformed? Tracking that gets harder as the system grows and data flows become more complex.

Option 1: Delete Only the Root Record

You delete the record from the source and leave every copy untouched. The failure mode is dangling pointers: you have removed the authoritative record while references to it survive elsewhere. An LRU cache like EVCache may eventually evict the stale copy on its own, but S3, DynamoDB, and Elasticsearch will keep storing it indefinitely. You pay for storage you should not be paying for, and you retain references to data that was supposed to be gone.

Option 2: Asynchronous Delete Fanout

The better approach, and the one Netflix built, is to propagate the delete asynchronously from the source to every downstream copy — caches, search indexes, and other databases. This avoids dangling pointers and removes the unnecessary storage cost, and it is a more comprehensive way to ensure the data is truly deleted everywhere.

graph TD
    A[Delete request for record X] --> B[Cassandra: source of truth]
    B --> C{Async fanout}
    C --> D[Elasticsearch index]
    C --> E[EVCache entry]
    C --> F[S3 backup / analytics copy]
    C --> G[DynamoDB copy]
    D --> H[No dangling pointers, no orphaned storage]
    E --> H
    F --> H
    G --> H

Availability: Is It Actually Safe to Delete?

Liu's next section addresses a question that becomes urgent once you have a large backlog of data that should already have been removed. Deleting it all is not obviously safe.

Bulk deletes in Cassandra create very large numbers of tombstones. Every subsequent read has to scan those tombstones to determine which records are still valid. That extra work increases latency and slows reads, and in the worst case causes timeouts and missed service level objectives — a direct user-visible impact. Before compaction kicks in, cleanup has not yet been triggered, so tombstones keep accumulating. When compaction does start, it is resource-intensive, consuming significant CPU, memory, and I/O. If that is not controlled, it starves critical production workloads and leads to service degradation or an outage, as the cluster tries to serve regular traffic and the delete workload at the same time.

The specific pathological case is a compaction storm, which Liu notes is especially challenging in LSM-tree based datastores like Cassandra. It happens when a large number of tombstones accumulate and many compaction tasks trigger at once. The resource consumption creates cascading effects that can destabilise the entire cluster. And until compaction completes, the tombstones are still on disk, so you are still paying storage costs for data that is pending removal.

Liu's summary: bulk deletes can exhaust resources, slow down reads, trigger compaction storms, cause cascading destabilisation, and produce storage bloat.

Four Mitigations

Partition-level deletes. Deleting individual rows one at a time creates overhead and leaves behind many tombstones, which inflates storage cost and complicates data management. Deleting an entire partition at once is far more efficient because it minimises the number of tombstones created and makes the storage footprint easier to manage.

TTL jitter. Rather than letting deletes cluster at the same instant, add randomness to each item's expiration time. Clustered deletes spike resource usage simultaneously; spreading them across the day avoids the spike and gives you a steady, controlled, manageable flow instead of one huge resource hit.

Resource-aware throttling. Netflix tracks datastore compute and storage usage and makes that information available to consuming applications. Those applications prioritise requests based on available resources. Low-priority delete requests do not have to be handled immediately — they can be delayed and processed asynchronously, reducing pressure on the system and prioritising live traffic.

Rate limiting and backoff. Deletes start at a low rate and increase gradually as confidence grows that the system can handle the load safely. Compaction and resource utilisation metrics are monitored to adjust the rate dynamically. On failure, exponential backoff prevents overwhelming the system, giving it time to recover and avoiding cascading issues and outages. Liu frames the goal visually: the out-of-control spiky behaviour on the left is what to avoid; the smooth, controllable, managed flow on the right is the target.

In the Q&A, Arvind gave a concrete number for how this throttling is parameterised. The signal comes from the downstream system itself — Cassandra, RDS, and EVCache all emit compute and storage utilisation metrics that the delete pipeline consumes. The threshold is per-system rather than a single global ratio. For Cassandra specifically, Netflix keeps roughly a 30% buffer; when live traffic starts eating into that buffer, deletes slow down. Her justification is that these deletes are simply not urgent — the data has not been accessed, it was supposed to be gone already, and in the case of tester data it might otherwise sit there forever. Deletion can afford to run "under the radar."

Correctness Under Concurrency

Arvind returns to handle the hardest pillar. In a distributed system, concurrent operations are normal, not exceptional. Consider client A issuing a delete of id A while client B concurrently inserts id A. Which one wins? Without a defined conflict-resolution rule, you cannot know.

The industry uses several strategies, and Arvind maps them to engines:

For EVCache, she accepts that the end state is unknown — and argues that is acceptable for a cache, because a short TTL will cause it to expire quickly anyway. That is a deliberate scoping decision: not every store needs the same correctness guarantee.

Deletes With Explicit Timestamps

In a last-write-wins system like Cassandra, deletes should be issued with an explicit timestamp option. Arvind's worked example: client A writes x = 1 at T1, client B writes x = 2 at T2, client A writes x = 3 at T3. All three reach the database, and what persists is x = 3 because it carries the latest timestamp. During concurrent updates it is essential to deduplicate the writes so that the genuinely last write is what remains.

Idempotency Tokens

An idempotency token is a unique, auto-generated or randomly generated value attached to every write. Combined with a timestamp unique to that write, it gives you a safety net: it establishes the order of operations and makes it safe to hedge and retry requests. Arvind calls it critical for correctness, especially in a last-write-wins database.

Why this matters: without a token, a retried write is indistinguishable from a new write. With one, the system can recognise the duplicate and avoid applying the same mutation twice — which in a delete pipeline is the difference between "already deleted" and "deleted, then accidentally resurrected by a retry."

Generate the Timestamp on the Client

Arvind asks explicitly where the timestamp should come from, and answers that database-generated timestamps are not sufficient. By the time an operation reaches the database, the true order of operations is no longer known — only the client knows the order in which it issued them. Her recommendation is therefore to use a client-side timestamp and not depend on arrival time at the database.

A caveat worth flagging, raised implicitly rather than explicitly in the talk: client-generated timestamps make correctness dependent on client clock accuracy. This is a well-known trade-off in last-write-wins systems, and it is why such deployments generally require tightly synchronised clocks.

Compare-and-Set

The compare-and-set (CAS) pattern handles the case where two clients both read a pending status and both issue a delete. Only one operation wins; the second is rejected. The result is a single successfully applied operation, which is the desired outcome. Arvind notes that timestamps can also be used within conditional writes.

The Deletion Lifecycle

Having established the constraints, the speakers walk through Netflix's four-phase lifecycle. Arvind introduces the framing with "trust but verify": you give customers the means to delete their own data, but sometimes they do not, or it accidentally remains, and the platform has to audit and verify regardless.

graph LR
    A[1. Identify
what data is deletable
and where it lives] --> B[2. Audit and validate
against S3 backups]
    B --> C[3. Delete
rate-limited, resource-aware]
    C --> D[4. Continuously monitor
catch failures and resurrection]
    D --> B

Phase 1: Identification

Identification means determining what data needs to be deleted and everywhere it is stored. Netflix takes input from multiple sources. Data owners have the best information about what their systems hold, so there is a self-serve mechanism for them to register it. A schema registry with annotations also lets teams mark fields as critical data requiring deletion — Arvind's example is a URI identifying a dataset containing critical information, with the customer ID in the ID field, declaring that all of that data should be deleted when the customer goes away.

Arvind's lesson later in the talk reinforces this: identification and lineage must come first. You need a system that knows where your data lives before you can delete anything reliably.

Phase 2: Audit and Validation

Liu describes this as offline scheduled workflows, and the offline part is deliberate. First, the identifier table and the data table are backed up to S3. A scheduled audit job then runs against those backups to match, validate, and find data eligible for deletion. The results are written to an audit table, producing a clear and complete list of data to be deleted.

Validation is a separate second step that double-checks the list is accurate and current. It can, for example, use a tester identifier table to confirm whether those IDs are present in the data table. The validated result is written to a final set.

The reason validation is separate matters: because the backups may be several hours old, some data may have been reinserted or restored in the meantime. The validation job accounts for those updates and ensures that only data that is genuinely still safe to delete reaches the final set.

In the Q&A, Arvind explained why the audit runs against backups rather than live systems. Taking Cassandra as the example — the largest fleet at Netflix — the backup copy sits in S3, and the audit reads and processes those files directly. Online auditing is avoided because scanning an entire table to find deletable data drives up resource utilisation and read throughput on the live system. The principle she states is that you do not want to affect the online system in any way while performing these deletes.

Phase 3: Deletion

With the final set in hand, the delete service takes over and removes data from Cassandra, RDS, or through delete endpoints exposed by other backend systems. The engine-specific behaviour described earlier is designed into the service: for example, if a database is experiencing a CPU or storage usage spike, the service backs off and waits until it is safe to proceed.

Every delete operation is recorded in a journal service, capturing what was deleted, when, and by which service. This is done asynchronously via a Kafka topic so it does not slow down the main delete path. The result is deletes that are safe, efficient, and fully traceable.

Phase 4: Continuous Monitoring

Issuing a delete does not mean the job is done. Regular audit cycles keep running to catch failed or incomplete deletions. Anything that did not get deleted in the last cycle reappears in the next audit and is requeued. This self-healing loop is what makes the pipeline tolerant of partial failure.

The tracked metrics include the number of deletable records, how long data has exceeded its retention window, and the count of successful versus failed deletions. These allow issues to be spotted quickly and keep the process reliable over time.

The Pluggable Deletion Interface

Some teams need deletes to happen in a specific order, or want to manage deletion themselves. Netflix supports this with a pluggable interface: a user specifies a custom deletion plugin, a callback URL, and payload templates. The delete service then triggers deletion in that system using whatever protocol or logic suits the situation. Liu presents this as what makes the framework extensible enough to support a wide range of requirements across different teams and datastores.

Architecture And Data Flow

Liu's high-level architecture ties the phases together. A control plane triggers the audit jobs. Audit jobs scan both identifier tables and data tables to determine what needs deleting. A validation job confirms the identification and prepares the final set. The delete service removes the data from the corresponding systems. All deletion operations and results are journalled so there is a complete record and recovery is possible.

graph TD
    CP[Control plane] -->|triggers| AJ[Audit job]
    IT[(Identifier table
S3 backup)] --> AJ
    DT[(Data table
S3 backup)] --> AJ
    AJ --> AT[(Audit table:
deletion candidates)]
    AT --> VJ[Validation job]
    VJ --> FS[(Final set:
confirmed deletable)]
    FS --> DS[Delete service]
    DS -->|rate limited,
resource aware| CS[(Cassandra)]
    DS --> RDS[(RDS)]
    DS --> EP[Custom delete endpoints
via pluggable plugins]
    DS -->|async via Kafka| JS[Journal service
30-day retention in S3]
    JS -->|enables| REC[Recovery]
    DS -->|results| MON[Continuous monitoring
and dashboards]
    MON -->|requeue failures| AJ

Earning Customer Trust in a Central Team

Arvind is blunt about the organisational problem: nobody trusts a central team to delete their data. Every team considers its data important, and centralisation concentrates the blast radius.

Her illustrative example is tester accounts. A test device exercises an end-to-end flow through several microservices, and the resulting tester data lands in the accounts database, the profiles database, playback systems, and gaming systems. The data proliferates across the platform and grows over time — exactly the growth you want to stop. But before deleting it, you have to validate that the delete is genuinely valid.

"Trust but verify" is the operating principle. A customer team may assert that they already deleted their data and instruct the central team not to touch it. Arvind's position is that this is fine, but audit jobs should still validate that the data that was supposed to be deleted is actually gone and stays gone — because a failed process or a bug could have left it behind, and you do not want to discover that later.

Three mechanisms build the trust:

Centralised dashboards. Every step — auditing, finalising the deletable set, and issuing deletes — is monitored from a single place where all dashboards are collected, giving visibility into the process rather than asking teams to take it on faith.

A robust recovery system. The journal enables recovery of deleted data within 30 days. Arvind's reasoning about the retention window is practical: you cannot store the data forever, but journals are just logs, so they are cheap to keep in S3 object storage with a 30-day expiry. When recovery is in place, customer trust increases, because a bug or issue discovered after the fact can be corrected.

Gradual, adaptive propagation. Real-time deletes have their own dilemma: they run while live reads and writes are happening, and legacy deletes can slow the system down. The answer is slow, gradual propagation with rate limiting, orchestration, and per-database best practices.

The Q&A sharpened the details of the journal significantly. Asked whether the journal stores only the keys of deleted items or their full detail, Arvind answered that it logs the whole thing, because a recovery system needs the actual data to restore it. This gives a more targeted recovery path than restoring from a backup. The cost is manageable because S3 is cheap enough that a terabyte or petabyte-scale journal costs on the order of a dollar for the retention period. Her empirical justification for 30 days is that issues usually surface within a week, so a month provides comfortable buffer. Liu added the operational constraint plainly: a strict TTL on the journal tables limits the recovery window to one month, so you have to spot an issue within that window for recovery to be offered.

Arvind also described the recovery mechanics in detail. The journal service behaves like a time series database, holding timestamps and dataset information for each delete. If a customer reports missing data that was present yesterday, the team queries the journal to see whether it appears in the delete log. If it does, a bug is confirmed, and the timestamp lets them scope how far the problem extends and pull the full list of deletes in that window.

Re-inserting the data is where last-write-wins bites again. You cannot simply replay the original insert, because a newer legitimate insert may have arrived after the erroneous delete, and replaying would clobber it. Netflix's approach is to take the original timestamp, add one millisecond, and insert the recovered data at that point — slotting it in before any newer write so that newer inserts remain intact. That is the online path; the more effective offline path is bulk recovery, where you build an SSTable containing all the recovered data and load it directly, which is faster.

The Bulk Delete Optimisation Netflix Is Building

Arvind describes an in-progress future direction rather than a shipped system. The idea is to stop issuing individual deletes for large backlogs and instead generate the tombstones directly in the storage engine's native file format. For Cassandra, that means writing the deleted rows as tombstones into SSTables, uploading those files to S3, using Cassandra's import capability to download and load them, and then immediately compacting the data away. The goal is to reduce ingestion costs and improve performance.

Supplementary context: an SSTable (Sorted String Table) is Cassandra's immutable on-disk data file format. Generating SSTables offline and importing them bypasses the write path entirely — no coordinator, no commit log, no per-row replication overhead — which is why it is dramatically cheaper for large volumes than issuing millions of individual delete statements. The same technique underpins the bulk recovery path described above.

Impact and Reported Scale

Liu reports the outcomes. The team has been able to identify and manage deletion across a large number of datasets, and — most importantly — the journey has been free of data loss incidents. (The published transcript renders this sentence as "we have data loss incidents", which is almost certainly a transcription error; the surrounding claims about running deletion "safely and with confidence" only make sense in the negative reading.) Daily deletion counts have increased over time as adoption grew, and the system has remained stable throughout, with deletion continuing to run smoothly and safely.

The specific numbers presented:

The gap between 1,300 identified and 125 audited is worth noting: coverage is still expanding, and the speakers present this as work in progress rather than a finished rollout.

A participant asked whether the sustained high deletion volume is expected to reach zero. Arvind's answer clarifies the shape of the curve. The volume keeps rising because auditing coverage keeps expanding — each newly audited system adds to the numbers. Once the backlog is cleared via bulk load and only forward deletes remain (deleting data as it becomes deletable from today onward), the line flattens and then goes to zero. Her stated goal is that MRO should be zero at all times once both the backlog and forward deletes are handled.

Note on terminology: the transcript uses "MRO" without expanding it. In context it appears to denote a measure of outstanding deletable data — records that should have been removed but have not yet been. The talk does not define it, so treat the exact definition as uncertain.

The same participant asked what makes data deletable at Netflix. Arvind's categories:

• Tester data. Testing end-to-end in the production system means test data spills over everywhere. This is the most prominent category.
• Data belonging to a former customer.
• Data the system failed to delete when it should have.
• More generally, in the centralised view, anything the identifier table classifies as deletable.

Trade-offs And Limitations

Deferred deletion trades read performance for write performance. LSM-tree engines like Cassandra make deletes cheap at write time by writing a tombstone, but every subsequent read pays to scan it until compaction runs. On engines like DynamoDB where deletion is immediate, this trade-off does not exist — which is why Arvind rates the hidden cost of deletion as high for Cassandra, Elasticsearch, and RDS but low for DynamoDB and EVCache.

Delete fanout eliminates dangling pointers but introduces distributed failure modes. Propagating a delete to every copy means every copy is a potential failure point. Netflix accepts this and compensates with continuous audit cycles rather than trying to make the fanout atomic.

Offline auditing against backups is safe but stale. Auditing S3 backups avoids loading the live cluster, but the backups can be hours old, which is precisely why a separate validation step is needed to account for reinserted or restored data. This is a deliberate exchange of freshness for production safety.

Deletion throughput is intentionally sacrificed for availability. Rate limiting, TTL jitter, resource-based backoff, and the 30% Cassandra buffer all mean deletes run slower than they technically could. Arvind justifies this on the grounds that the data in question is stale and unaccessed, so latency in removing it carries little cost — but this reasoning would not hold for deletes with a hard regulatory or contractual deadline, a case the talk does not address.

The recovery window is hard-bounded at 30 days. Liu is explicit that a strict TTL on the journal tables limits recovery to one month. An erroneous deletion discovered on day 31 is not recoverable from the journal. The justification is empirical — issues typically surface within a week — but it is a real limit, not a soft one.

Client-generated timestamps shift a correctness dependency onto client clocks. Arvind is right that only the client knows the true order of its operations, but the consequence is that a client with a skewed clock can write a delete that either loses to a stale write or wins over a newer one. The talk does not discuss clock synchronisation requirements.

EVCache's conflict resolution leaves the end state genuinely unknown. Arvind states this openly and accepts it, on the grounds that a cache with a short TTL will converge anyway. This is a reasonable scoping decision but means the platform's correctness guarantee is not uniform across all datastores.

Centralisation creates an organisational trust problem, not just a technical one. Arvind's observation that nobody trusts a central team to delete their data is a genuine limitation. The mitigations — dashboards, audits, and recoverability — reduce it but do not eliminate it, and they represent substantial engineering investment that exists purely to make the platform socially acceptable.

Coverage is incomplete. With 125 of 1,300 identified datasets audited, the majority of eligible datasets are not yet under continuous audit. The reported absence of data loss incidents applies to the system as operated so far, not to a fully rolled-out state.

Single-service failure is handled by deferral, not rollback. Asked what happens if the delete service is down mid-fanout, Liu's answer was that there is no rollback and none is needed: deletion of expired member or tester data does not have to happen online or quickly, so the next audit cycle picks up whatever was missed. Blast radius is further contained by running separate fleets — a direct delete service on one set of instances and an async stack that calls customer deletion endpoints — sharded so that a failure in one fleet does not affect the delete service in another shard. This is eventual consistency for deletion, and it works only because the deletes are not time-critical.

Practical Takeaways

1. Inventory before you delete. Build data lineage and identification first. Use a self-serve registration mechanism for data owners plus schema registry annotations that mark which fields are subject to deletion. You cannot delete reliably from systems you cannot enumerate.
2. Know your engine's delete semantics individually. Write down, for each datastore in your stack, whether TTL is native, whether hard deletes are immediate or deferred, and which background process eventually reclaims space. Liu's third lesson is explicit: what works for Cassandra may not work for RDS or S3.
3. Fan deletes out to every copy asynchronously. Deleting only the source record leaves dangling pointers, orphaned storage cost, and references to data that should be gone.
4. Prefer partition-level deletes to row-by-row deletes where your data model allows it, to minimise tombstone count.
5. Add jitter to TTLs so expirations spread across time rather than clustering into a resource spike.
6. Drive delete throughput from downstream utilisation metrics. Consume the compute and storage metrics your datastores already emit, keep a headroom buffer (30% in Netflix's Cassandra case), and back off when live traffic encroaches on it. Use exponential backoff on failure.
7. Audit offline against backups, then validate. Run audit jobs on S3 copies of identifier and data tables rather than scanning live clusters, and add a validation pass that accounts for data reinserted since the backup.
8. Make the loop self-healing. Requeue anything that failed or was missed into the next audit cycle instead of building bespoke retry logic.
9. Journal the full payload, not just the keys. Recovery requires the actual data. Write it asynchronously via a message bus so it does not slow the delete path, and store it cheaply in object storage with a defined TTL.
10. Recover by inserting at original timestamp plus one millisecond in last-write-wins stores, so that legitimate newer writes are not clobbered by the restore.
11. Use client-side timestamps and idempotency tokens on every write so that ordering is preserved and retries and hedged requests are safe.
12. Use compare-and-set where your store supports it so that concurrent duplicate deletes resolve to exactly one applied operation.
13. Provide a pluggable deletion interface — callback URLs and payload templates — for teams whose deletion order matters or who need to own the process.
14. Instrument the whole lifecycle on shared dashboards and track deletable record counts, retention-window overruns, and success versus failure rates.
15. For large backlogs, generate native storage-engine files rather than issuing individual deletes. Netflix's in-progress approach builds Cassandra SSTables of tombstones, uploads them to S3, imports them, and compacts immediately.

Key Terms

• Tombstone — A marker written in place of deleted data in an immutable or log-structured store, indicating the record is dead as of a given timestamp. Reads must scan tombstones to determine which records are still valid.
• GC grace period — In Cassandra, the interval a tombstone is retained before compaction may discard it, giving all replicas time to learn about the delete. A node offline longer than this can resurrect deleted data on restart.
• Compaction — The background process that merges immutable data files, physically removing tombstoned and superseded records and reclaiming disk space.
• Compaction storm — A destabilising condition where many compaction tasks trigger simultaneously after a large tombstone accumulation, consuming enough CPU, memory, and I/O to cascade into cluster-wide degradation.
• LSM tree — Log-structured merge tree, the storage architecture used by Cassandra and similar engines, in which writes append to immutable files that are periodically merged. It makes writes fast and makes deletion a deferred, read-taxing operation.
• SSTable — Sorted String Table, Cassandra's immutable on-disk data file format. Generating SSTables offline and importing them bypasses the normal write path.
• TTL (time to live) — An expiry set at write time after which data becomes eligible for removal. Expiry and physical removal are usually separate events.
• TTL jitter — Deliberate randomisation of expiry times to prevent deletes from clustering and spiking resource usage.
• Soft delete — Application-level deletion that marks a record as deleted and relies on a later background job to physically remove it.
• Hard delete — An explicit delete command issued to the datastore, though what it physically does varies dramatically by engine.
• Dangling pointer — In this context, a surviving reference to or copy of data whose authoritative source record has been deleted.
• Delete fanout — Propagating a deletion from the source of truth to every downstream copy: caches, search indexes, backups, and secondary databases.
• Data resurrection — Deleted data reappearing, typically because a replica or process missed the delete and later rejoined with stale live data.
• Last write wins (LWW) — A conflict-resolution rule in which the write with the highest timestamp is the one that persists, used by Cassandra.
• Compare-and-set (CAS) — A conditional write that applies only if the current value matches an expected value, so that concurrent conflicting operations resolve to exactly one winner.
• Idempotency token — A unique value attached to a write so that duplicates and retries can be recognised and applied only once.
• Journal service — Netflix's record of every delete, including the full payload, written asynchronously via Kafka to object storage with a 30-day TTL to enable targeted recovery.
• Control plane — The component that triggers and orchestrates the audit, validation, and deletion jobs, as distinct from the data path that performs the deletes.
• Forward deletes — Deleting data as it becomes eligible from the present onward, as opposed to clearing the historical backlog.
• Pluggable deletion interface — An extension point where a team supplies a custom plugin, callback URL, and payload template so the platform can trigger deletion through their own protocol or ordering logic.

The overall lesson the speakers leave is that deletion becomes safe only when it is treated as a continuously verified process rather than a single operation. Identification, auditing, throttled execution, journalling, and monitoring are not optional extras around a delete statement — collectively they are the delete. Liu closes with the image the talk opened on: once all of these strategies are in place, you can push the first domino and be confident that everything falls into place safely.

Reference: Vidhya Arvind and Shawn Liu, Architecting a Centralized Platform for Data Deletion at Netflix, QCon San Francisco, published by InfoQ on June 4, 2026.