Week-Long Outage, Lifelong Lessons: What Six Days of Elasticsearch Downtime Teaches

Most outage retrospectives end at the bug. Molly Struve's does not. Her talk walks through a six-day Elasticsearch outage in March 2017 that made her employer's flagship feature unusable for nearly a week, and then argues that the three technical lessons the team learned are the less important half of the story. Her reasoning for that ranking is that even teams doing the technical work well will still have incidents — "it's not a question of if, but when" — so what determines the outcome is the human half: how quickly the team asked strangers for help, who they had hired, and how their VP of engineering behaved while the phone kept ringing.

Struve is a Staff Site Reliability Engineer at Netflix with a degree in aerospace engineering from MIT. At the time of the outage she was an engineer at Kenna Security, a cybersecurity company that helped Fortune 500 customers manage vulnerability risk. She credits this incident with pushing her into SRE work. This 49-minute, 32-second talk was recorded at QCon San Francisco 2025 and published by InfoQ on April 28, 2026.

These notes report what Struve presented and clearly mark the small amount of background context added for readers unfamiliar with Elasticsearch.

What You Will Learn

• The timeline of the Kenna Security Elasticsearch outage and why its rollback option turned out to be as expensive as continuing to debug.
• What a rollback plan has to cover for stateful infrastructure, and how an FMEA or pre-mortem is used to produce one.
• Why widely used, mature software still needs performance testing against your own workload, and the mechanisms Struve uses for it.
• How previous-experience bias forms after a successful upgrade, and lightweight ways to counteract complacency.
• How and when to escalate outward during an incident without burning goodwill.
• What leadership behaviour during a multi-day incident looks like in practice, and how to keep long incidents sustainable for the humans in them.

Background: What Elasticsearch Was Doing at Kenna

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. Data is split into indexes, which are further split into shards spread across a cluster of nodes, so that a query can be answered in parallel across many machines. Because the cluster holds the searchable copy of the data, "rolling back Elasticsearch" is not a code operation — it is a data migration. That distinction is the seed of the whole story. (This paragraph is background added for these notes; Struve assumes the audience knows it.)

At Kenna, Elasticsearch was not a nice-to-have. The company's differentiating feature was that customers could search their entire cybersecurity data set in seconds. Struve is explicit that Elasticsearch was "the cornerstone of Kenna's platform, and what set Kenna apart from its competitors". When the cluster was down, the product was, in her words, "completely unusable" apart from logging in.

One clarification she raises up front: the upgrade went from Elasticsearch 2 to Elasticsearch 5, which looks like a three-version jump but was a single major version bump. Struve explains that Elastic had renumbered its releases to match "the underlying language that powers it"; in practice that means the Lucene version. The team was not attempting anything reckless.

The Outage Timeline

Preparation had been underway for months to ready both the codebase and the 21-node cluster. The team scheduled the upgrade for a Thursday evening, deliberately leaving Friday as a buffer to work out problems. The mechanics — shut down the cluster, update the nodes, update the codebase — went "off without a hitch".

Struve does not name the specific bug or its identifier in the talk, so these notes do not either. What she does report is the effect of the workaround: "it was like night and day", and the team cried with relief.

Two details from that week are worth dwelling on because they are the kind of thing retrospectives usually omit. First, the mitigations were ugly — caching placed for no reason other than shedding load, which is a rational choice under duress but also a source of future bugs. Second, the rollback discovery happened mid-incident. Nobody had priced the rollback before starting, so the option that should have bounded the outage to hours instead cost roughly as long as simply continuing to debug.

Architecture And Data Flow

The reason rollback was expensive is structural rather than accidental. A code-only change can be reverted because the deployed artifact is the only thing that changed. The cluster, by contrast, holds the data, so undoing the upgrade meant standing up a second cluster on the old version and copying everything across — the five-day estimate Struve reports.

flowchart TD
    A["Application code
(ES 2 client)"] -->|revert PR: minutes| B["Application code
(ES 5 client)"]
    C["21-node cluster
Elasticsearch 2"] -->|upgrade in place:
one evening| D["21-node cluster
Elasticsearch 5"]
    D -.->|"rollback: stand up a
new ES 2 cluster and
copy all data over
(est. 5 days)"| E["New cluster
Elasticsearch 2"]
    D -->|"CPU/load spikes
under production traffic"| F["Cluster crash
search unavailable"]

The asymmetry in that diagram is the whole first lesson. The upgrade path was one evening; the reverse path was five days. Nobody measured the reverse path before committing to the forward one.

Lesson 1: Have a Rollback Plan, All the Time

Struve's framing is that before any change you must be able to answer three questions: can this be rolled back with a simple revert PR; if not, what does the rollback actually involve; and how long will it take. If the answer to the third is "a long time", that is itself a planning input — it tells you how much detection and mitigation investment the change deserves.

Her diagnosis of her own team's mistake is precise. The upgrade's code could have been reverted with a single PR. "We never considered if the Elasticsearch software and data itself could be rolled back." The blind spot was scoping the question to the artifact an engineer owns rather than to the system.

The tool she recommends for widening that scope is a pre-mortem, which at Netflix takes the form of an FMEA (Failure Mode and Effects Analysis). The mechanics she describes: sit down with the team, "put your chaos hats on", enumerate every way the change could fail, capture the failure modes in a spreadsheet, then rank them by severity. The ranking drives two different kinds of decision. Some failure modes are worth mitigating in advance. Others are not worth preventing but are worth detecting fast, because they are the signals that say stop and roll back now. Deciding your bail-out signals while calm is far easier than deciding them at 2 a.m.

Asked in Q&A whether she recommends other pre-mortem formats, Struve was deliberately unopinionated: the concept is simply thinking about failure ahead of time, and the format can be whatever suits the team, including asynchronous. She does have a preference, though — teams generate genuinely new failure modes when they talk live in a room. Her own team's variant is to have everyone write ideas down individually first, then meet to discuss, which surfaces additional ideas beyond the union of the individual lists. She also points to a talk at the SEV0 conference describing a lighter-weight pre-mortem process for those wanting alternatives.

The extension she adds from seven subsequent years as an SRE is that rollback planning is not a big-change ritual. Even on routine deploys, asking whether the change is backwards compatible, and whether any part of it cannot be undone, is worthwhile. (Supplementary context not from the talk: backwards compatibility is what makes rollback cheap, and a schema migration, a message-format change, or a cache-key change can quietly make the previous version of your code unable to run against the current state of the world.)

Her sharpest practical point is about untested rollback mechanisms. She polled the audience on who had exercised their rollback mechanisms in the last three or six months, and told the rest: "are you feeling lucky?" The supporting story is from Netflix. During an incident she asked service owners whether data could be rolled back. Yes — there was a runbook. When had it last been used? Never; it had been written two years earlier. The rollback took over an hour, hitting "multiple friction points, unexpected behavior". Her phrase for an unexercised runbook is Schrödinger's cat: you do not know whether it works, and you have chosen to find out during an incident. Pick a cadence — quarterly, semi-annually — and actually run the thing.

Lesson 2: Do Performance Testing, Regularly

The second lesson is that maturity and popularity of a dependency are not evidence about your workload. Struve is candid that the team skipped performance testing because they did not believe their usage was unusual and their data volume was not remarkable. "We glossed right over this step."

The detail she offers against that reasoning is that companies much larger than Kenna were already running Elasticsearch 5, and none of them had hit this bug — so scale did not predict exposure. Struve's summary is what she calls her favourite SRE saying, "trust, but verify" — software may perform well for 99% of users while your particular use case contains the unoptimized or buggy path.

After the incident, Kenna built a mechanism to run shadow traffic against new clusters and indexes, so that a candidate cluster could be exercised with realistic load before taking real users. Shadow (or replay) traffic means duplicating production requests to a parallel system running the new code or configuration, without serving its responses to users. It is the cheapest way to observe how a change behaves under the traffic mix you actually have rather than the one a synthetic benchmark imagines. (Worth noting as an operational caveat not raised in the talk: shadowing writes or requests that touch third parties requires care, since the shadow system can cause real side effects or duplicate costs.)

At Netflix, Struve describes the same replay-traffic approach plus a second mechanism: long-running canaries. A canary is a small slice of production running the new version alongside the old for comparison. Many Netflix teams run canaries overnight rather than for a few minutes, which she presents as a good way to catch smaller regressions before they grow into bigger problems.

Her generalization mirrors the first lesson: do not reserve performance testing for large changes. From her SRE experience, "small dependency updates wreck more havoc than I care to admit". The goal is a release pipeline that can detect a performance regression on any change, not a special ceremony reserved for upgrades.

Lesson 3: Be Wary of Previous Experience Bias, Always

The third lesson is about the specific reasoning error that produced the overconfidence. The team's previous Elasticsearch upgrade had delivered large performance gains, which they generalized into "this one will too" — an implicit belief that software only gets better. A second bias reinforced it: the software was widely used and well known, so "it never entered our minds that that software could contain an application crippling bug".

Both are inductive arguments from a sample of one, and both are seductive precisely because they are usually right. Most upgrades do improve things and most mature software does not eat your cluster. The problem is that the rare counterexample is exactly the case where you needed the preparation.

Struve's counter-measure is deliberately unbureaucratic. She observes that teams on a streak of successful deploys become complacent, and suggests small ambient reminders rather than process: three emojis or a GIF at the bottom of a PR template or in a deploy confirmation dialog, prompting the engineer to pause and consider what could go wrong. Her explicit constraint is that this must not become heavyweight — "we don't need folks filling out TPS reports every time they want to push to production". The aim is a moment of skepticism, and ideally a smile, not a gate.

Lesson 4: Widen Your Circle, Early

The pivot in the talk is that the next three lessons are things the team did well, and that without them the outage would have been worse than six days.

The first is escalating outward. On the day after the outage started — the Friday afternoon on which the cluster had already collapsed entirely — the team posted to Elastic's public discuss forum. Struve estimates it took her about an hour to gather the data and write the post, and calls it "by far the most valuable hour of the incident, because this post is what got us our answer". Tedor replied the same day, and everything that followed was collaborative debugging with a core Elasticsearch developer rather than the team guessing alone. The counterfactual worth sitting with is what the remaining five days would have looked like without that hour.

Her argument is that most teams widen their circle as a last resort, after they have exhausted themselves. She references a related idea from Kyle at Meta on escalating quickly. Widening can mean external community forums, a vendor, or simply more teams inside your own company. The core claim from her SRE career: "as soon as you get the right person in the room, it is amazing how quickly things can turn around in an incident."

She also names the obstacle honestly — asking for help is hard for experienced engineers, and she includes herself. Her counter-argument is a second-order benefit senior engineers rarely notice: when you visibly ask for help, you show early-career engineers that not knowing is normal and that asking is a strength.

The Q&A supplied the necessary balance. An audience member working as an incident manager described the social cost of escalating: reaching out implies the other person should drop everything, and when it turns out not to be their problem, repeated escalation erodes goodwill. Struve's answer is severity-based triage. Reserve aggressive escalation for the genuinely critical incidents — SEV0 and SEV1 — where in her experience responders understand the stakes and are willing to be paged even when it turns out not to be their component. For lower-severity incidents, invest the time in investigating yourself, which also builds your own understanding of the system. She is also candid that the forum post working out was partly luck; public forums often return silence.

Two related practices came from the audience rather than the speaker. One participant described an "Incident of the Month" meeting created after their own upgrade incident, running for eleven months at the time of the talk, used to share findings across platform and product teams — a cheap mechanism for making one team's lesson available to everyone. The same participant's incident, a seemingly trivial logging-endpoint update to five-year-old untouched code, caused a five-to-sixfold logging spike and roughly $150,000 in licensing cost, which went unnoticed over a weekend until Monday afternoon; their fix included budget alerting. That is a useful reminder that not all outage cost is availability cost, and that spend needs alarms too.

That participant also asked whether Struve would avoid upgrading right before a weekend. Her answer is nuanced rather than a rule. Kenna's customers were enterprises who largely did not use the platform on weekends, so the weekend absorbed part of the impact and gave the team breathing room. She notes that for a week-long outage there is no good start day, and that she cannot imagine having fought it with every day being a full business day. Note the dependency on customer traffic patterns: the same choice is much worse for a consumer product whose peak is the weekend.

Lesson 5: Your Team Matters

Three engineers worked 15-plus-hour days for close to a week. Struve describes going through "every emotion in the book, from sad to angry to despondent", and the observation that mattered to her is that those emotions bonded the team rather than fracturing it.

Her conclusion is about hiring: "You can teach people tech... but you can't teach character." The interview question she proposes is whether this is someone you would want beside you with your back against the wall, and whether they will jump in without being asked. If yes, find a reason to hire them. The systems framing she puts on it is worth keeping: your team is part of your resiliency picture, alongside your redundancy and your runbooks. This is her opinion drawn from experience rather than a research finding, but it follows directly from the fact that a six-day incident is survived by people, not by architecture.

Lesson 6: Leader and Management Support Are Crucial

The lesson she saves for last is what the VP of engineering did. He was online late at night and at dawn every day for the duration. He contributed technically, but Struve identifies three other roles as more important.

He was the cheerleader: the team repeatedly wanted to give up, and he kept pushing because he genuinely believed they would solve it. He was the defender: he absorbed the constant calls and messages from upper management and customers, so the engineers could stay on the technical problem instead of context-switching into status reporting and panic management. And above all else he trusted them — he "never wavered in his trust that we would figure it out", and stayed "the epitome of calm, cool, and collected the entire time", which Struve says is what kept the team pushing forward. Her assessment is blunt — "If we had had a different VP, I am sure things would have turned out very differently."

Her direct address to leaders is that during an incident you are usually not the one pushing code or taking mitigative action, and the role you do play is larger than you realize: how you react sets the example for the rest of the team. Be their cheerleader, be their defender, be whatever they need you to be — and above all else, trust that they can do it, because that trust is what lets the team believe in itself and is critical to morale on a long-running incident.

Asked in Q&A to be specific about what the VP did to keep the team focused over five to seven days, Struve gave two answers. The buffering — fielding the calls and texts — and, upstream of the incident entirely, having hired a team that would band together under adversity and felt accountable for the work.

She then gives concrete guidance for leaders who want to join an incident channel without harming it. Announce your purpose immediately and narrowly: something like "Hey all, I'm here to support. I'll be camera off. Let me know if you need anything." The point of the announcement is to pre-emptively answer the question every engineer in the room is silently asking, which is whether they are about to be evaluated. Saying you are there to support signals you will not micromanage, and communicates solidarity.

A follow-up question pushed on this: even a silent observer can suppress people if psychological safety is thin — "I can talk in front of my team, but now the boss is watching." Struve's answer is that the safety has to be built on ordinary days, not manufactured during the incident. Leaders should talk about incidents regularly, not only when one is happening, framing them as learning opportunities and explicitly celebrating what came out of them. She calls the goal an "incident positive atmosphere". She adds an easily missed corollary: new hires may arrive from organizations with punitive incident cultures, so the message has to be restated to them explicitly rather than assumed to be absorbed by osmosis.

The reason this lesson lands hardest for her is captured in a saying she returns to: "People don't remember what you did, they remember how you made them feel." The technical improvements from the incident were real, but what she says lives on is that every engineer on that team watched leadership operate with compassion, empathy, and a we-will-figure-it-out attitude, and now tries to emulate it. It is why she became an SRE, and it shapes how she runs incidents today: she still mitigates and still handles comms, but "first and foremost, my priority is supporting those engineers around me".

Sustaining a Multi-Day Incident

An audience member raised the problem that incident practices designed for a two-hour event do not survive a multi-day one, and asked when to make that transition. Struve's threshold is the one-to-two-hour mark: past that, "your radar has to go up". What changes at that point is that you start managing the incident as a staffing problem alongside a technical one:

• Take stock of who is present and actually contributing, and who does not need to be there.
• Plan rotations before you need them, identifying who is in the wings and can be paged in.
• Enforce rest. She cites Log4j, where engineers had to be told directly to go sleep and page in a teammate.
• Attend to basic needs — checking whether people need food or a break.

She notes that Netflix is rolling out an incident management workshop for leaders covering exactly this, and that keeping tabs on team health in the background is a high-value contribution a leader can make during a long incident. Notice how much of this depends on the leadership behaviours in Lesson 6: an engineer will not admit exhaustion to a leader they think is measuring them.

Trade-offs And Limitations

• Pre-mortems and FMEAs cost time and do not guarantee coverage. Struve's own bug was in third-party source code and might well not have appeared on a failure-mode list. What an FMEA reliably buys is the rollback cost estimate and the bail-out signals — which in this incident would have been worth more than predicting the bug.
• Rollback exercises consume real engineering time and can themselves be risky in production. The counter-argument is that discovering the friction during a rehearsal is strictly cheaper than discovering it during an incident, as Struve's Netflix data-rollback story shows.
• Long-running canaries and shadow traffic cost infrastructure. You are paying to run duplicate capacity. Shadowing also needs care where requests have side effects or metered third-party costs.
• Widening the circle has social cost and uncertain payoff. Public forums frequently go unanswered — Struve calls her outcome lucky — and over-escalating internally teaches colleagues to discount your pages. Severity-based triage is her mitigation.
• A leader in the incident channel can suppress candour where psychological safety is weak. The stated announcement helps, but the underlying culture has to be built beforehand.
• The weekend-buffer choice is workload-dependent. It worked for Kenna's enterprise customers, who were not using the platform on Saturdays. It is not a general rule.
• These are one team's lessons from one incident. Struve presents them as hard-won experience, and the team-and-character claims in particular are personal judgement rather than measured results.

Practical Takeaways

• Before any significant change, write down the rollback procedure and its estimated duration. If the estimate is measured in days, treat that as a design problem, not a footnote.
• Extend the rollback question past your own code to data, schemas, and any stateful dependency. Ask explicitly: what part of this change cannot be undone?
• Run an FMEA or pre-mortem before large changes, splitting the ranked failure modes into "mitigate now" and "detect fast and bail out".
• Schedule a recurring exercise of your rollback mechanisms — quarterly or semi-annually — and treat any untested runbook as unproven.
• Add performance verification to the ordinary release path, not just to big upgrades: overnight canaries and replay traffic both work for this.
• Do not let a dependency's popularity substitute for testing your own workload.
• Add a lightweight prompt to PR templates or deploy confirmations that nudges engineers to consider failure. Keep it small enough that nobody resents it.
• Escalate outward within hours, not days. Budget an hour to write a well-evidenced request for help.
• Past the one-to-two-hour mark, start managing staffing: rotations, mandated rest, food, and a list of who can be paged in next.
• If you are a leader joining an incident, state your purpose in one line and then act as a shield for the team's attention.
• Publish and discuss incidents rather than burying them. An "Incident of the Month" forum, as one audience member described, spreads one team's lesson across the organization.
• Alarm on cost, not just availability. A logging regression can burn six figures over a quiet weekend without triggering a single availability alert.

Key Terms

• Elasticsearch — A distributed search and analytics engine built on Lucene, storing data in indexes split into shards across a cluster of nodes.
• Node / cluster — A single Elasticsearch server process, and the group of such processes that jointly hold and query the data. Kenna's cluster had 21 nodes.
• FMEA (Failure Mode and Effects Analysis) — A structured exercise, used at Netflix, in which a team enumerates ways a change could fail and ranks the failure modes by severity to decide what to mitigate and what to detect.
• Pre-mortem — The general practice of reasoning about failure before it happens; an FMEA is one format.
• Canary — A small production slice running a new version alongside the current one for comparison.
• Shadow / replay traffic — Mirroring production requests to a parallel system running new code or configuration, without serving its responses to users, in order to test it under realistic load.
• Widening the circle — Deliberately increasing the number of people or organizations involved in troubleshooting an incident, including external vendors and communities.
• SEV0 / SEV1 — The highest incident severity tiers, used here as the threshold above which aggressive escalation is warranted.
• Previous experience bias — Generalizing from a small number of past successes to assume the next similar action will also succeed.
• Psychological safety — A team condition in which people can admit uncertainty, ask for help, and report problems without fear of penalty.

Struve closes with the observation that the 2017 outage is infamous at Kenna, but not as a failure. It became a story the engineering culture pointed to — this is who we are — and in her account the psychological safety it created is why that team went on to build good software there and elsewhere. Her bonus lesson is to embrace incidents rather than quietly filing a post-mortem and hoping people forget. That embrace starts with ownership: "it was a team miss. We own that, and we embrace that." Her framing is that an incident is a withdrawal from your availability piggy bank — you have already spent the money, and the only question left is whether you squander it or invest it in your team and your software by learning from it publicly.

Reference: Molly Struve, Week-Long Outage: Lifelong Lessons, QCon San Francisco 2025, published by InfoQ on April 28, 2026.