Qantas 32: The Hub Was Measured. The Pipe Was Not
A Rolls-Royce factory had a machine that measured every engine hub it built and printed any error in red. In 2010 an A380 engine built with one of those hubs burst over Indonesia, because a small oil pipe inside it was too thin on one side. The machine measured a hole. It never measured the wall.
Chapters
- 0:00 The machine that printed errors in red
- 0:36 Flight 32, 7,000 feet
- 1:04 The landing everyone knows, and the question it skips
- 1:38 What the pieces reached
- 2:24 A pipe you could hold between two fingers
- 3:02 A pipe with a countdown
- 3:36 The last minute of engine No. 2
- 4:14 The engine that behaved differently from the model
- 4:54 A wall thickness nobody wrote down
- 5:26 The hub that moved
- 5:58 The measuring machine and the wrong datum
- 6:34 A limit ten times too loose
- 7:11 Either not detected or not declared
- 7:50 Where it could have lived
- 8:23 The concessions
- 9:04 April 2009: a sample of nine
- 9:46 The signatures that were missing
- 10:21 Every step had the information
- 10:57 Layers added after the failure
- 11:36 A measurement answers one question
Transcript
The machine that printed errors in red 0:00
A Rolls-Royce factory in Hucknall, England, had a measuring machine that checked the hubs it built. A probe touched each hole, and the machine printed a report. Any number outside its limit was printed in red. In 2006, a hub for one engine of an Airbus A380 went through it. Four years later, that engine came apart over Indonesia. The cause was an oil pipe inside it, and at its thinnest the wall of that pipe was just over a third of a millimeter. The Australian investigators found it probable that the machine had flagged the error. It never became a decision.
Flight 32, 7,000 feet 0:36
On November 4th, 2010, Qantas Flight 32 took off from Singapore's Changi airport. On board were five flight crew, twenty-four cabin crew and four hundred and forty passengers: four hundred and sixty-nine people. About four minutes after takeoff, climbing through seven thousand feet at two hundred and fifty knots, the crew heard two loud bangs, almost together. The number two engine, the inner one on the left wing, had just failed.
The landing everyone knows, and the question it skips 1:04
You may know how this story ends, because the crew landed. It took about fifty minutes to work through the first set of warnings, and the airplane touched down in Singapore at a quarter to noon, local time. It stopped about one hundred and fifty meters from the end of the runway. The investigators wrote that the crew managed the event as a competent team, and that is true. But it is not the question here. An engine broke, and parts of it went through the wing. The question is how a part so small, in an engine so carefully built, got there.
What the pieces reached 1:38
The turbine disc inside the engine had burst into three large pieces, with enough force to go through the engine casing. They tore holes in the left wing, through the upper skin and the front spar. The debris cut about six hundred and fifty wires. The airplane lost its green hydraulic system, the electrical power from two engines, the wing slats, the normal landing gear extension, and part of its braking. Fuel leaked from a tank, and a hot fragment started a short fire inside it, which died by itself. Debris fell over about one and a half square kilometers of Batam Island, and the report lists no injuries to anyone. The No. 1 engine would not shut down, and firefighters stopped it with foam three hours after landing.
A pipe you could hold between two fingers 2:24
The failure began in a part you could hold between two fingers. An oil feed stub pipe carries oil into the bearing chamber of the turbine. It is welded into a hub, and at the end where the oil filter sits, a wider hole is bored out inside the pipe. That bored section is called the counter bore. By design, the wall of the pipe there should be nine tenths of a millimeter thick. On this engine, the fractured pipe measured from one point four millimeters on one side to thirty-five hundredths of a millimeter on the other. A fatigue crack had grown across about a quarter of its circumference, exactly where it was thinnest.
A pipe with a countdown 3:02
The ATSB calculated how long a pipe like that lasts. At nine tenths of a millimeter, the worst case was about one hundred and fifty-nine thousand flights. At half a millimeter, five thousand four hundred. At the measured thin point, six hundred and seventy-seven flights. The engine had flown six hundred and seventy-seven. The peak stress in the wall rose from seven hundred and twelve megapascals at the intended thickness to more than a thousand at the thin one. A pipe that thin was not a weaker pipe. It was a pipe with a countdown.
The last minute of engine No. 2 3:36
The oil escaped into a space beside the bearing chamber, called the buffer space, that was designed to have no oil in it at all. Air there was between three hundred and sixty-five and three hundred and seventy-five degrees Celsius, and the synthetic oil can ignite at two hundred and eighty. The ATSB concluded that the oil most likely ignited by itself when it sprayed in. The fire started about a minute before the failure. About ten seconds before it, it reached the seals of the bearing chamber. The arm that holds the intermediate pressure turbine heated, weakened and let go, and four seconds later the disc burst.
The engine that behaved differently from the model 4:14
There is a second thing the investigators found, and it explains why a fire became a burst. When a turbine loses its connection to its compressor, it can overspeed. The manufacturer's models predicted that the airflow through the compressor would reverse, causing the engine to surge, and that this would hold the turbine back. On this engine, the high pressure system recovered partly instead. The ATSB wrote that this recovery was unexpected and had not been seen before. The certification analysis had also assumed that every part was made to its design. That assumption had no place for a pipe made too thin.
A wall thickness nobody wrote down 4:54
So how did the wall get thin? Start with the drawing. The wall thickness of that pipe was not written anywhere on it. The designers had asked for two things. The hole the pipe passes through had to sit within five hundredths of a millimeter of a reference point, and the counter bore had to be centered on it within a tenth. If both were met, the wall would come out right. In the worst case of those tolerances the minimum was eighty-two hundredths of a millimeter. The thickness was a result, and no line on the drawing asked for it.
The hub that moved 5:26
The reference point on the drawing, called datum AA, was the hole in the outer hub. But the pipe had to be welded in before the counter bore was cut, and once it was there, the machine could no longer reach that hole. So the shop used a pin in the inner hub instead. A probe measured the pin, and several hours later the counter bore was milled. In between, the hub moved. The counter bore was cut around the pin's axis and not the pipe's, so it came out off center. On one side of the pipe, the wall was thin.
The measuring machine and the wrong datum 5:58
Every hub then went to the measuring machine. It was a bridge over a fixed table, with a probe that touched each feature and compared it with a nominal value. If a number was outside its limit, the report printed an error statement, in red if printed in color. It checked the hub twice, once after the first holes and once at the end. But it measured the position of the hole against the inner hub counter bore, datum M, and the drawing said datum AA. The machine was answering a different question from the one on the drawing. And the report did not say anything about wall thickness.
A limit ten times too loose 6:34
The program that ran the machine was written and checked by inspectors at the plant. In 2006, when this engine was built, there was no formal process for creating or validating those programs. One was introduced in August of 2007. In that program, the allowed position for the hole was five tenths of a millimeter, ten times the five hundredths on the drawing. It was tightened to the drawing's value in December of 2007. The investigators could not find out why the tolerance had been loosened. A number that is checked against the wrong limit still looks like a check.
Either not detected or not declared 7:11
Here is the sentence from the report this episode is built on. The ATSB found it probable that the machine reported the non-conformance, but that it was either not detected or not declared by inspection personnel. Read it slowly. The report cannot say that anyone saw it. The printout for this hub, and for others from that time, was not kept. So the investigators could not tell whether an error was printed at all. They listed what could have happened: a problem with the printout, a line simply missed in review, or a second measurement that showed no error. What they could say is that the numbers made it likely.
Where it could have lived 7:50
What the ATSB did say is where this could have lived. In 2007, two audits at the Hucknall plant found many non-conformances on inspection reports that never reached the formal system. The report concluded that a culture existed there in which it was acceptable not to declare what staff judged to be minor non-conformances. The same report is careful: a staff survey found real concern for quality and no evidence of intentional violations. And the survey did not include the team that made this structure.
The concessions 8:23
After the audits, a different door opened. A non-conformance could be accepted through a concession, signed by engineers who judged that the part was still fit for use. Concessions for this assembly went from none in 2006 to fifteen against fourteen built in 2007. One of them listed hole positions up to point two nine eight millimeters from true position, with a note that a minimum gap of eighty-five hundredths had been achieved. A wire gauge check showed the pipe was centered in the hole. The engineer and the authority who approved it would not have known that the machine had measured against datum M, not AA. The report says it was reasonable for them to accept the part.
April 2009: a sample of nine 9:04
On March 27th, 2009, an engineer checking a revised machine program found that the counter bore on two earlier hubs was off position by about half a millimeter. It was the first time anyone at the plant had seen the offset, and the first time anyone had seen that it thinned the pipe wall. A retrospective concession was sentenced on April 3rd, covering about one hundred pipes already in service. It rested on nine unfinished hubs, a sample the investigators called not statistically representative. It assumed an offset of no more than seven tenths of a millimeter, which gave a wall of half a millimeter. The offset in this engine was larger.
The signatures that were missing 9:46
Company procedure required that a retrospective concession be approved by the Chief Engineer and by the Business Quality Director. Neither had signed it. It was not reported to the quality review board. It was assessed as having no effect on the customer, so the pipes in service were not pulled. In the ATSB's words, that denied them the opportunity to assess the risk to the fleet. And the problem was wider than one form. Of one hundred and thirty-eight retrospective concessions between 2009 and 2011, only seven followed the process.
Every step had the information 10:21
The ATSB did not blame a person, and it said so. It described a complex combination of events and missed opportunities that began with the design of the engine. A wall thickness left out of the drawing. A datum that had to change in manufacturing. A measuring program that checked a different datum, with a loose tolerance, and nobody to say what it was measuring. An error that was probably printed and perhaps never read. A concession decided on nine parts. At every step, the information existed. At no step did it arrive where a decision was made.
Layers added after the failure 10:57
After the failure, forty engines were taken out of service because the pipe wall in them measured under half a millimeter. The manufacturer withdrew every unit of the earliest production standard, because its manufacturing records were unavailable. On November 10th, European regulators ordered inspections of the engine for oil leaks, and an engine showing any could not keep flying. Then came the fix that matters most: software that shuts the engine down before its turbine disc can reach the speed at which it bursts. It was on every Trent 900 by December 6th. It does not need the pipe to be right.
A measurement answers one question 11:36
A measurement answers one question. This one answered where a hole was, relative to a feature that moved. The question that mattered was how thick the wall was, and no one was asked that. The fix did not trust the measurement any more than the old process did. It assumed that some part would one day be wrong, and shut the engine down before that mattered. Next case is a windscreen. British Airways Flight 5390, June 1990: a windscreen blew out in flight, held by bolts of the wrong size. Most of the bolts were thinner than specified, picked by matching the old ones, not by looking in the parts catalogue.
Description and sources
Qantas Flight 32, an Airbus A380, took off from Singapore on 4 November 2010 with 469 people on board. About four minutes later, climbing through 7,000 feet, the intermediate pressure turbine disc of the No. 2 Rolls-Royce Trent 900 burst into three pieces that tore through the left wing and damaged hydraulics, electrics, fuel and brakes. The crew landed back at Changi; the report lists no injuries.
The cause was an oil feed stub pipe with a thin wall on one side, from a counter bore that had been machined off-centre when the hub moved during manufacture. Oil leaked into a space designed to be oil-free, ignited, and burned through the arm holding the turbine until the disc let go. The wall thickness was never a feature on the drawing; it was a result of two position tolerances. The factory's measuring machine measured the hole's position against a different reference from the drawing's, with a looser limit, and its printouts for this hub were not kept. The ATSB found it probable that an error was reported and either not detected or not declared. A retrospective concession in April 2009 rested on nine parts and lacked required signatures. After the failure, 40 engines were removed and the Trent 900 fleet received software that shuts an engine down before its turbine disc can burst.
PRINT-READY, FROM THIS CHANNEL
The Failure Atlas, Vol. 01 — Tacoma Narrows · Citicorp Center · Millennium Bridge · Apollo 13 · the 2003 blackout · Hyatt Regency
https://therepository.gumroad.com/l/failure-atlas
PRIMARY SOURCES
- Australian Transport Safety Bureau, Aviation Occurrence Investigation AO-2010-089: In-flight uncontained engine failure, Airbus A380-842, VH-OQA, overhead Batam Island, Indonesia, 4 November 2010, final report 27 June 2013 (CC BY 3.0 AU) -- read directly (executive summary, sections 1.1-1.4, 2.2-2.4, 3.1-3.5, 4.3-4.13, 5.2-5.5, 6.1-6.3, 7.1-7.2): https://www.atsb.gov.au/publications/investigation_reports/2010/aair/ao-2010-089 -- text read from the copy hosted by the FAA: https://www.faa.gov/sites/faa.gov/files/QantasA380_Accident_Report.pdf
- Next episode: British Airways Flight 5390, 10 June 1990 (UK AAIB report 1/92, to be read directly): https://aviation-safety.net/asndb/326018
ARCHIVE PHOTOGRAPHS — the real structure, with title, author, date, archive and licence. CC BY items are used under that licence, cropped and reframed for the screen.
- Fig. 77 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 78 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 8 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 10 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 41 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 43 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 47 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 63 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 64 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 74 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 76 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 88 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 85 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 87 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 90 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 92 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 91 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
- Fig. 46 — ATSB — 2013 | AO-2010-089 | CC BY 3.0 AU
The 5 technical plates in this video are illustrations generated for the channel by a diffusion image model, styled to match its cyanotype identity. They are diagrams of the system — the photographs above are the archive, the drawings are ours, not photographs of the hardware, and no person is depicted in any of them.
Root Cause investigates why engineered systems fail, using official investigation findings and primary technical literature. Sources for this episode are linked above.