News The inside story of SpaceX’s historic rocket landing that changed launch forever

[News]

“It’s hard to describe how epic this comeback was after our first Falcon 9 launch failure.”


SpaceX lands a Falcon 9 rocket for the first time in December 2015. Credit: SpaceX

On Dec. 21, 2015, SpaceX launched the Orbcomm-2 mission on an upgraded version of its Falcon 9 rocket. That night, just days before Christmas, the company successfully landed the first stage for the first time. The story behind this remarkable achievement is nowhere more fully told than in the book Reentry, authored by Ars Technica Senior Space Editor Eric Berger and published in 2024. To mark the tenth anniversary, Ars is reprinting a slightly condensed chapter from the book that tells the inside story of this landing. The chapter begins in June 2015 with a tragedy, the disintegration of a Falcon 9 rocket carrying the CRS-7 cargo supply mission for NASA. It was the first time a Falcon 9 had been lost in flight.

Seconds after the Dragon-bearing Falcon 9 rocket broke apart over the Atlantic Ocean, David Giger shouted into his headset, “Dragon is alive!”

In the decade since he joined the company straight out of graduate school, Giger had taken on management of the entire Dragon program, reporting directly to Elon Musk. He watched the CRS-7 launch from mission control in Hawthorne not with a particular role, but rather providing a leadership presence. Giger could sense the Dragon mission team, mostly younger engineers, freeze up as video showed debris from the rocket showering back to Earth. A lot of the people involved in the hairy early flights of Dragon, including the C2 mission in 2012, had moved on to other positions at SpaceX or departed.

“They were a great team, but I think everyone assumed it was over,” Giger said. Unlike a lot of his colleagues, Giger had endured some trying times at SpaceX, including three failures of the Falcon 1 rocket. After the Falcon 9 shattered, Giger noticed that Dragon continued to send data back. It had separated from the rocket and was flying some thirty miles above the Atlantic Ocean.

The key to saving Dragon was opening its parachutes before it got too close to the ground. SpaceX had not anticipated such a contingency, nor planned to send commands to Dragon as it rode on the Falcon 9 rocket. But in an emergency, the Dragon control center could talk to Dragon using ground-based antennas. So controllers in California frantically worked to configure this communications system and command the two drogue parachutes to open. These are the small, precursor parachutes that stabilize the vehicle prior to deployment of Dragon’s three main parachutes.

The command was sent, but nothing happened. Dragon continued to dive.

For about two minutes after the rocket’s breakup, the plucky spacecraft faithfully relayed data. Then, less than a mile above the ocean, below the horizon from the Florida coast, the data stopped. The spacecraft and its 4,000 pounds of cargo plunged into the sea.


Opening the parachutes was not as simple as pressing a button. It required operators to send more than a dozen specific commands, in the correct order, to manually deploy the parachutes. In their haste—the operators had only seconds to act before the spacecraft fell too close to the ocean for its parachutes to have a meaningful effect—they had forgotten to turn on power for parachute deployment.

The unnecessary loss of Dragon was one lesson SpaceX took away from the failure of the nineteenth launch of the Falcon 9 rocket. In the aftermath of this CRS-7 failure, during long and intense meetings, Musk focused most of his time and energy on the rocket. But more than a few times, Musk would turn toward Giger and other Dragon officials and complain about its loss. “Dragon shouldn’t be fucking stupid,” Musk would admonish the team. “It should have saved itself.” By the next Dragon mission, this emergency scenario would be baked into the rocket’s launch procedures. If need be, Cargo Dragon could be saved.

The high-profile failure of the CRS-7 mission in June 2015 hit SpaceX hard. After five years, of challenging work with Falcon 9 and Dragon, the company had started to hit its stride. The Falcon 9 won a string of lucrative commercial satellite contracts. NASA, too, invested billions of dollars in SpaceX to launch humans onboard Dragon one day. Then it all fell apart. Carrying cargo for its most important customer, SpaceX blew up its rocket in spectacular fashion.


The loss of CRS-7 came at an inopportune time for NASA, as well. Half a year earlier the space agency’s other commercial provider of cargo services, Orbital Sciences, lost a NASA mission when its Antares rocket exploded just above the launch pad. Critics of the agency’s support for commercial spaceflight reemerged in Congress, underscoring the severity of the setback and raising questions about whether private companies could be trusted with human spaceflight.

Acquiring hallowed ground


In the first three years SpaceX flew the Falcon 9 rocket, from mid-2010 to mid-2013, it launched just five times. This flight rate was completely unacceptable to Musk, and after introducing version 1.1 of the rocket, he expected this cadence to increase substantially. He charged the Florida site director, Brian Mosdell, with delivering the capability of launching more than one Falcon 9 rocket a month.

But that was not Mosdell’s only task. He also spent much of 2013 spearheading SpaceX’s charge to obtain a lease for a second launch pad in Florida. This was to be none other than the most historic launch pad in the Western hemisphere, NASA’s Launch Complex 39A. Surrounded by swampland and rising just a few feet above the Atlantic Ocean, the sprawling 200-acre site includes the hallowed ground where Neil Armstrong and Buzz Aldrin took their last steps on terra firma before walking on the Moon. Later, dozens of shuttle missions also launched from there.

Following retirement of the shuttle in 2011, NASA no longer had any use for the launch site. The agency’s inspector general characterized the pad as “unneeded infrastructure,” and leasing the facility to a commercial launch company would offload millions of dollars a year in maintenance costs. SpaceX, already flying, seemed like the obvious choice as it sought a suitable site for Falcon 9 and Falcon Heavy launches and eventually crew flights for NASA. However, in the spring of 2013, another bidder emerged. Jeff Bezos held a reverent fondness for space history and sought to lease the site for Blue Origin and its New Glenn rocket.


Exterior view of Launch Complex 39A prior to its acquisition by SpaceX. Credit: NASA

To sweeten his proposal, Bezos offered to share the launch complex with SpaceX or another company. While this was a generous offer, Blue Origin did not have an orbital rocket in 2013, nor was it close to having one. Because of this, NASA awarded a twenty-year lease to SpaceX in September.


Musk was giddy after winning, having secured the landmark site. But he was also tweaked by Bezos’s bid, which he believed was intended to block his access. In an email to Space News, Musk mocked his would-be competitor. “If they do somehow show up in the next five years with a vehicle qualified to NASA’s human rating standards that can dock with the space station, which is what Pad 39A is meant to do, we will gladly accommodate their needs,” Musk wrote. “Frankly, I think we are more likely to discover unicorns dancing in the flame duct.”

Musk also mocked Blue Origin during internal meetings at SpaceX. He would say things like, “A company must really stink to call themselves BO.” History vindicated him. Blue Origin did not have an orbital rocket ready to fly from Launch Complex 39A within five years. Or ten. As of the writing of this book, the company’s New Glenn orbital rocket has yet to make a single launch attempt (Editors note: Blue Origin did launch New Glenn for the first time in early 2025). SpaceX, in the meantime, has launched more than 100 rockets from the old NASA pad.

Mosdell wrote a majority of SpaceX’s proposal to win the lease for 39A, including all of its technical materials, its schedule, and its budget considerations. After the company acquired the lease, he fully understood all of the work entailed in demolishing the old shuttle infrastructure and building a launch tower capable of supporting the Falcon 9 and Falcon Heavy.


His relatively small staff included seventy-two full-time employees and about eighty welders and other contract hires working on a new transporter. They already were maxed out by Musk’s push to launch monthly from SLC-40, putting in 80- to 100-hour weeks to keep up. Accordingly, Mosdell traveled to Hawthorne in January 2014 to meet with Musk and Shotwell about staffing up his operations for the buildout of Launch Complex 39A. During the meeting, Mosdell outlined what he believed to be a “lean” team to complete the design, procurement, building, and testing of the site. He then asked to hire sixty-eight people over the next six months.


Musk rejected the idea. The meeting spiraled downward from there, and Musk’s parting words to Mosdell were, “Go home and work harder.”

As a leader, Mosdell keenly felt the demands he placed on his team with intense work schedules. Eventually they would burn out and leave, or the quality of their work would suffer. When he raised these concerns with Musk and Shotwell, they would always say the money was tight, and if Florida could just buckle down and get the next payment milestone, some of the burden could be lifted off the team. Mosdell’s meeting in Hawthorne convinced him things really were never going to change. SpaceX would never enter a cruise phase; it would only accelerate.

“There was not going to be a genuine effort to fix the problem,” he said. “This was going to be a shut up, go home, and color kind of thing. And I decided I was no longer going to go to the Cape and tell my team to just keep going because things would change. Because I knew it was bullshit.”

During his six years at SpaceX, Mosdell had delivered. He built the team that scrappily built the SLC-40 launch pad for about a tenth the cost of SpaceX’s competitors. He worked as a launch director for six missions. They started with almost entirely manual launch operations, and by the end of 2013 reached the point where about 90 percent of the countdown was automated. Under Mosdell’s watch, the launch site had a perfect record. He also led the campaign to win NASA’s competition for its historic LC-39A site. He had laid the foundation for the success that ultimately awaited SpaceX in Florida.

But it had not been enough. He resigned.


Musk was not displeased. In his mind, Mosdell and the team at Cape Canaveral had not gone hardcore enough. Launching once a month was not enough. Musk believed Falcon 9 rockets should be flying from Florida once a week. In hindsight, this is a cadence the company would not reach for nearly a decade.

To fill Mosdell’s role, Musk turned to one of his people. Ricky Lim had joined SpaceX in 2008, spending months on Kwajalein during the final three flights of the Falcon 1 rocket. He came of age during the early years at SpaceX, surviving the crucible of the Falcon 1 and near death of the company. He then helped out at Vandenberg, alongside Zach Dunn and Lee Rosen. In the wake of Mosdell’s departure, Lim was asked to fill in as site director at Cape Canaveral for a few weeks. Three weeks would turn into six years.

The launch team at the Cape consisted of two halves. One was built from converts from the legacy rocket companies, including United Launch Alliance, where Mosdell had come from. The other half consisted of young engineers who had typically taken their first job with SpaceX. Musk felt Mosdell was too closely aligned with the legacies. Lim, he felt, would rally the younger blood.

And so he did. From mid-2014 onward, SpaceX started to hit a groove at the Cape. There were bumps along the way, of course, and eventually SpaceX would hire not sixty-eight additional employees as Mosdell had requested, but hundreds more for launch and pad rebuild work in Florida. Yet by April 2015, Lim and his team managed to launch a pair of missions—the sixth operational cargo mission for NASA and a Turkish communications satellite—within just thirteen days of one another from Florida. Eagerly, they moved to the next mission on the manifest, another cargo supply mission for NASA: CRS-7.

A painful, shocking moment


On the morning that CRS-7 lifted off, Lim set aside his site leadership role to serve as launch director. All went well for the first two minutes of the flight. However, at 2 minutes and 19 seconds, Lim started to hear chatter on the network about “data drops” from the second stage. He glanced up at a video showing the rocket far down range, a white streak climbing into an azure sky. While there was some sort of vapor cloud around the upper stage, the rocket’s nine engines were burning just fine.

“It was surreal,” Lim said. “What we saw in the data did not match what we were seeing. The first stage was still going. On the long-range cameras, it looked a lot like prior launches. I thought the second stage people were mistaken, to be honest.”

He believed it might be a ground software display issue or some other minor problem. A second or two more clarified matters. The large white cloud expanded, engulfing the rocket. As it dissipated, the tracking camera showed a shower of debris as pieces of the rocket began tumbling back to Earth. A hushed silence fell over the flight control room.

Video of CRS-7 being lost in flight.

Musk watched the launch from England, where he was celebrating his forty-fourth birthday. Seeing his rocket disintegrate was a lousy birthday present. His first call was to Dunn. After working for five years launching rockets, Dunn had taken over running SpaceX’s propulsion department just a few months earlier and observed the launch from inside the Mission Control Center in Hawthorne. Dunn told Musk he did not have an immediate explanation for the failure and passed his phone to Jon Edwards, an engineer supervising the launch. Edwards said it looked as though some kind of pressure event had occurred on the second stage.


During past traumatic moments, Dunn had felt empowered to take immediate action. When a Falcon 1 rocket had started imploding during transport on a C-17 rocket over the Atlantic Ocean, Dunn had risked his life to climb inside the booster to open a pressure valve. But this was different. “In this case, I didn’t really know what to do immediately,” he said. “That was kind of the creepy thing about it. We knew we had to go and see if there was any debris in the ocean. But there was not a ton of immediate action other than starting to figure out what took place. It was a really painful, shocking moment.”

Dunn walked out of mission control and back to the propulsion area of the office. He called his team together and gave a quick talk about what had just happened. Like Giger, Dunn was now one of SpaceX’s veterans and had persevered through the Falcon 1 failures. He explained that launching rockets was a high-risk business and that the way to get through this challenge was to work methodically to find out what happened, fix the problem, and fly again.

Musk returned to California and dove into the accident investigation with frenzied energy. At least daily, and often multiple times a day, he convened meetings in his executive conference room on the first floor of the company’s headquarters. A black conference table dominated the middle of the room, around which nine chairs were situated in a large U. People presenting at the meeting filled the chairs. More engineers sat or stood around the sides of the room. Musk always stationed himself at one end of the table closest to the door, facing a wall that featured a large canvas print of the Falcon 1 rocket launching from Omelek Island, a cautionary tale of the company’s desperate roots.


“The vast majority of the people at the company today have only ever seen success, and so you don’t fear failure quite as much,” Musk said after the accident, which he publicly described as a “huge blow” to SpaceX. “To some degree I think the company as a whole maybe became a little complacent.”

Indeed, the company had grown ten-fold since its last failure in 2008. As the Falcon 9 had streaked through its first eighteen launches, some of the Kwajalein veterans like Dunn and Lim would discuss how few of the company’s employees had experienced the pain of launch failure. By the time of the CRS-7 mission, only about 5 percent of employees had that experience, so they were somewhat numb to Musk’s mantra that “only the paranoid survive.” Before every Falcon 9 launch, Musk would email the entire company and say that if anyone had a serious concern about the upcoming launch, he wanted to hear it. And he was genuine about this. He wanted to hear risks and, if they sparked wider concern, act on them.


Because of SpaceX’s growing size, downtime was expensive. With a staff of approximately 4,000 employees at the time of the failure, SpaceX needed to fly frequently to pay the bills. Personnel costs alone exceeded $70 million a month. So Musk pushed for a thorough, but rapid, investigation. During the CRS-7 failure analysis meetings, he would listen as department leads shared their latest information on the investigation. Often he interrupted with a question, a comment, or a sharp critique. These were tense and frustrating and difficult discussions because the true cause of the accident remained a mystery for several weeks. Underlying it all was a sense of unease and uncertainty about the fate of SpaceX.

“Elon brought a tremendous amount of energy to those meetings,” Dunn said. “You always have to bring your best with Elon, but in these sort of tense moments you had to be right there. One hundred percent focus. No bullshit. You better be able to explain what you know, what you don’t know, take direction very crisply, and understand what he means with precision. If not, he pounced.”


The responsibility for investigating the failure fell on one of Musk’s oldest and most trusted lieutenants, Hans Koenigsmann. At the time, Koenigsmann served as vice president of Flight Reliability and Mission Assurance, which essentially meant ensuring safe and successful launches.

“I feel like if I’m responsible for the risk, I’m also responsible for when it goes wrong,” he said. “It took five months, and I worked around the clock. I took one weekend off, I think. Otherwise, I worked every single day of those months. I worked my ass off, and my team did, too.”

Part of the challenge in understanding what had gone wrong is that the rocket had gone from flying normally to a conflagration in just 800 milliseconds. Koenigsmann’s team knew almost immediately that the problem originated on the second stage, which had not yet separated from the first stage. But it had all happened so fast, in a fraction of a second. As a result, engineers and scientists from SpaceX and NASA had just 115 pieces of telemetry data (that is, measurements from various sensors onboard the rocket during that critical second when the upper stage failed). From this, they ultimately determined that the liquid oxygen tank had ruptured after a container containing pressurized helium broke free inside the tank.

Why are there bottles of helium in the LOX tank? As a rocket launches into space, it steadily burns propellant and oxidizer. As these fluids drain, helium gas is released to fill the vacated volume and maintain a downward pressure. This ensures that propellant and oxidizer continue to flow into the engines. The helium container is called a composite overwrapped pressure vessel, or COPV, because a strong fiber is wrapped around a metallic container to ensure its integrity. The more difficult question for Koenigsmann, therefore, became understanding why a COPV bottle containing helium had broken loose and struck the upper dome of the oxygen tank with catastrophic effect.


Eventually, some structures engineers pinned the cause of the failure down to a small $4 part about the size of a Tootsie Pop. This stainless steel eye bolt, also known as a rod end, helped secure the COPV to the wall of the oxygen tank. It had broken during the rocket’s ascent. Koenigsmann said these rod ends were rated to withstand 10,000 pounds of force, but one of them inside the ill-fated upper stage had broken under less than 2,000 pounds of force.

Nearly all rockets and spacecraft undergo a rigorous design process. The last checkpoint before a project moves into fabrication is known as “critical design review.” During this review for the Falcon 9, SpaceX had required the use of a more expensive rod end, Koenigsmann said. This part costs about $50. Somewhere between this design review and the actual flight, however, a cheaper rod end had been substituted. This steel rod end cost significantly less and was manufactured by a casting process. This meant it was made by pouring steel into a mold, where it solidified, and then was ejected from the mold. The more expensive rod ends worked fine at cryogenic temperatures inside the liquid oxygen tank, but cast materials were more problematic when pulled under the force of tension as they could have unknown flaws hiding inside.

SpaceX tried to find the actual rod end that had broken, to ensure it had nailed down the original cause of the failure. Searching for a thumbsized part fifty miles off the coast of Florida, hundreds of feet below the surface of the ocean, proved to be a rather Don Quixote–like quest. The company even hired a remotely operated submarine to look for wreckage. In the process, it found long-lost hardware from the Apollo and Space Shuttle Programs, but no Falcon 9 parts. However, Koenigsmann was confident that a cast rod end was the culprit because SpaceX tested similar parts from the same purchase order and found they were subject to breaking under cryogenic conditions.


So why had SpaceX switched to a cheaper cast rod end? Musk had instilled a culture of always looking to cut costs. Someone decided that a $50 rod end was too expensive and would be substituted with a cheaper part. Every strut on the rocket used rod ends, so there were hundreds of them on the vehicle, meaning this single change saved more than a thousand dollars on a launch vehicle. In their zeal to control costs, Musk and his lieutenants made such decisions thousands of times. Had Musk not been so judicious about costs, the price of a Falcon 9 would have ballooned. And in nearly every case, the approach worked—except this one.


In the fall of 2015, Koenigsmann wrote a detailed report about SpaceX’s findings and presented it to NASA and the Federal Aviation Administration. The report concluded that “material defects” were the most probable cause for the broken rod end. This essentially put the blame on the rod end’s supplier. NASA’s own, independent findings leveled a harsher judgment more directly on SpaceX. NASA attributed the failure to a “design error” by SpaceX. The space agency also said SpaceX’s quality control process should have identified the substandard rod ends before they were installed on the rocket.

“The implementation was done without adequate screening or testing of the industrial grade part, without regard to the manufacturer’s recommendations for a 4:1 factor of safety when using their industrial grade part in an application, and without proper modeling or adequate load testing of the part under predicted flight conditions,” the report stated.

In other words, NASA said SpaceX had messed up, not the supplier. Koenigsmann said he accepts that SpaceX should have done a better job screening the rod ends. But he said the supplier deserves blame as well. “SpaceX and the supplier screwed up,” he said.


Koenigsmann still keeps one of these rod ends in a desk drawer at home, to not forget the lessons of CRS-7.

Despite disagreeing over the fundamental cause of the supply mission failure, NASA and SpaceX continued to work together well. When Dragon sank into the ocean, NASA lost $118 million in cargo, including a crucial docking adapter needed to enable future astronaut missions to the space station. The failure also increased the US space agency’s reliance on Russia. For several months, the only means America had of getting its astronauts to the space station, and feeding them, came via a pair of small spacecraft designed during the Soviet era that launched from Kazakhstan.

Publicly, however, NASA did not chastise SpaceX for these problems. Rather, its officials remained supportive. During a US Senate hearing in 2016, when some elected officials would have celebrated an opportunity to lambaste the company, NASA’s chief of human spaceflight stood up for SpaceX when asked about the failure.

“They turned around very quickly,” the official, Bill Gerstenmaier, told Congress. “Within a matter of days, they were actually in a test facility on the ground testing the failure that they thought had occurred. That getting into test was much faster than I could have ever done on a NASA side. By the time I would have had the ability to get contracts written and done the proposals and put the test sequence in place, it would have been a half a year.”

SpaceX, in turn, sought to make NASA whole for its losses. A few months after the accident, the company quietly agreed to fly five future cargo missions, its sixteenth through twentieth flights to the space station, at discounted prices. SpaceX also increased the amount of cargo each Dragon mission would carry, giving NASA more bang for its buck.


This assumed, however, that SpaceX could get the Falcon 9 rocket flying safely once again.

Musk’s risky decision on liquid oxygen


Even had the cargo mission not failed, the Falcon 9 still faced months of downtime during the second half of 2015. Musk had decided the rocket needed another significant upgrade, to version 1.2, which later became known as Falcon 9 Full Thrust. This represented a massive evolution to the rocket’s capabilities. Drone ship landings were essential to making the economics of first stage reuse work, but they were not the only step. SpaceX also needed to squeeze every ounce of performance out of the rocket. No part of the Falcon 9 was spared a ruthless revision, and in the end, SpaceX engineers produced a new machine that increased the lift capability of the Falcon 9 by nearly one-third.

The propulsion department designed an upgraded version of the Merlin 1D engine that raised the thrust of each engine by about 15 percent. The structures department built a lighter rocket that was easier to manufacture. And all of the lessons learned from the Grasshopper program and landing attempts in the Atlantic Ocean were poured into the design of the new rocket legs and control systems.


However, the real linchpin of the upgrade involved a technology known as propellant densification, or squeezing as much fuel as possible onto the rocket. This sounds wonky and wholly uninteresting—but it is not. The science and engineering of super-chilling rocket fuel is fascinating, and its implementation tremendously risky. Within a year of seriously starting work on densification, SpaceX would blow up a rocket, destroy a launch pad, and lose a $195 million Israeli satellite. Some former Apollo astronauts viewed SpaceX’s approach as so dangerous they urged NASA to never let its astronauts fly on rockets fueled this way. Musk knew the risks, he accepted the risks, and in the end, SpaceX beat the risks.


Densification, however, was a tremendous challenge heaped on SpaceX at the same time the company’s employees were scrambling to recover from the loss of the CRS-7 mission, satisfy NASA’s concerns, and work through the finer points of landing first stages.

“Elon grasped the essentials of the reuse problem,” (senior SpaceX engineer) John Muratore said. “He kept telling us we’ve got to get more performance. We’ve got to get the liquid oxygen colder. He just kept driving us.” Somewhat understatedly, Muratore added, “It was quite an intense time.”

SpaceX densified both oxygen and kerosene, but since the former had to be chilled to much colder temperatures, it was far more difficult to handle. Oxygen is the most abundant element on Earth and essential for life. Humans cannot breathe without it, and in our bodies, it chemically reacts with molecules from food to produce energy. Similarly, this process of oxidation occurs when oxygen combines with a fuel. Firewood, for example, cannot burn without oxygen. And so oxygen is an essential component of producing combustion within a rocket engine. In fact, most rockets burn more oxygen than fuel on the way to orbit. Onboard the Falcon 9, in terms of mass, there is more liquid oxygen than kerosene fuel.

Musk reasoned that by packing more liquid oxygen into the rocket, it could get better gas mileage. He was certainly not the first person to think about forcing oxygen into a denser state and thereby increasing the amount that a rocket’s tanks could hold; NASA had previously studied propellant densification over the decades.

Recently, the agency had dismissed it, yet again, for the Constellation Program. But this decision was not based solely on physics. Rather, it was due to politics and rivalries between the agency’s field centers. Marshall Space Flight Center, in Alabama, already had its bread and butter with existing propulsion technology. And NASA management had little appetite for the exploding test articles that would necessarily accompany densification development. Neither of these were barriers at SpaceX, which could afford to fail—and indeed publicly celebrated its test failures as evidence of pushing beyond the bleeding edge.


By densifying liquid oxygen and kerosene onboard Falcon 9, SpaceX could squeeze an extra 8 to 10 percent of performance out of the vehicle. This was not trivial. It meant carrying two more tons of payload to orbit. This was extremely important for a reusable rocket, which was paying a significant mass penalty for returning to Earth due to its landing gear and other added components. For economic viability, therefore, Musk believed densification was as important as drone ship landings. If he could accomplish both, the Falcon 9 could truly be the world’s first twenty-first- century rocket—reusable, high-performing, and cost-effective.

So how does one densify oxygen? One way is to use liquid oxygen instead of gas. Liquid oxygen has a pale blue, ghostly color. It condenses at –297.33 degrees Fahrenheit (–182.96 degrees Celsius), far, far colder than the coldest temperature ever recorded on Earth in Antarctica. It is colder than even the darkest areas of the Moon that never see sunlight. This makes working with liquid oxygen difficult. However, the upside for rockets is worth it: Liquid oxygen is 1,000 times more dense than gaseous oxygen, so most rockets use liquids.

What Musk wanted to do was make this liquid oxygen still more dense by chilling it down, almost to a solid. This is basic chemistry, as the cooler a substance gets the more its constituent molecules slow down, thereby bringing them slightly closer together. So the colder SpaceX could get its liquid oxygen, the more that could be packed onto the rocket.


This is how one day, in 2015, Muratore and another engineer named Vincent Werner found themselves on the phone with the National Institute of Standards and Technology, a Maryland-based agency that is the world leader in measuring physical properties. Werner and a handful of SpaceX engineers had been poring over tables published by the agency that showed the various temperatures and pressures at which oxygen, nitrogen, and liquid air—a mixture of mostly oxygen and nitrogen— turned into solids.


“We called them because they had generated the tables,” Muratore said. “And they were like, ‘You know, guys, these were extrapolated tables. Nobody’s ever worked down here before. The tables are approximately accurate, but they could be off by a degree or two, or a psi or two.'”

SpaceX was not just looking to experiment with liquid oxygen at its coldest temperatures; it planned to produce vast quantities of the stuff. For a single rocket launch, the company needed to make hundreds of thousands of gallons. The actual work of producing densified oxygen fell to a small team of about eight engineers in Cape Canaveral, including Phillip Rench.

He seemed an unlikely hire for SpaceX. Rench earned a degree in mathematics from Southern New Hampshire University, which is not known for aerospace greatness. Rench then spent nearly a decade working at SeaWorld in Orlando, where he performed an odd assortment of jobs, from underwater maintenance to fixing amusement park rides. While working at SeaWorld, Rench discovered a knack for devising solutions to challenging problems. In 2010, a veteran trainer at the park, Dawn Brancheau, was dragged to her death by a killer whale named Tilikum while gently rubbing the creature. After the incident, Rench helped build a giant submersible floor that lifted the whales out of the water, to make it safer for trainers to interact with the creatures. It was Rench’s first time working with complex valves and other components used in control systems.

After seeing a promotional video that depicted a Falcon 9 rocket launching and landing in Florida, Rench was blown away. So he applied to SpaceX and was hired early in 2014 to help modify Launch Complex 39A. Rench watched the fateful CRS-7 launch from the vantage point of this pad, alongside the other engineers, technicians, and interns working on the old NASA site.


“Everyone was super depressed,” he said. “But the next day we came back at 150 percent, with energy and passion. You know the five stages of grief? Yeah, we went through that really quick.”

Engineers at McGregor had performed preliminary densification tests, and some of the early work in Florida was led by Brian Childers and Gavin Petit. Rench worked with a team that included Petit, David Ball, Chris Wallden, and others. Because the Florida crew had no practical experience with super-chilled oxygen, they more or less just started connecting equipment and seeing what happened. SpaceX used liquid nitrogen to chill liquid oxygen because the colorless gas turns into a liquid at –320 degrees Fahrenheit (–196 degrees Celsius), below that of LOX. To further chill the liquid oxygen, the engineering team flowed it through a pipe, around which was wrapped a coil of tubing filled with liquid nitrogen. The two never mixed, but the warmer LOX would shed heat into the liquid nitrogen. As this heat moved in, some of the warmer bits of nitrogen started to boil off. SpaceX used very powerful vacuum pumps to suck this heat away. Over time, as the pressure dropped, the temperature of nitrogen fell below –340 degrees, and the liquid oxygen followed. They could not go much colder, as nitrogen freezes at –346 degrees Fahrenheit (–210 degrees Celsius).

Rench loved the work. He had spent years working on valves and other systems to control the flow, temperature, and pressure of liquids. When his team was pushing liquid oxygen to its extreme, it was not so different from SeaWorld. Over the course of a few weeks, he and the other engineers developed procedures by which this super-chilled LOX could be made and stored in a large, insulated tank at the LC-39A launch pad. They worked in pairs, for eight-hour shifts, seven days a week. The nights were eerie, with a soundtrack from purgatory.


“Liquid oxygen does not want to be densified,” Rench said. “Densification makes this low, horrible growl. When we first started densifying LOX, the Praxair delivery drivers would be pumping the warm LOX into the sphere and it would make all kinds of crazy noises. They were getting nervous to be around it, and these are people who have worked with liquid oxygen for pretty much their entire lives.”

NASA had been skeptical about SpaceX’s plans for densification at Launch Complex 39A, so it asked for a demonstration. After Rench’s team delivered and NASA signed off, the fluids team at the launch pad started stripping the parts and pumps away from the LOX chiller system. They were needed at SLC-40 to make the densified propellant for the debut flight of the Falcon 9 Full Thrust.

SpaceX desperately tries to save Christmas


After two failed drone ship landings, Musk felt ready to try landing on land. This had a major advantage over the ocean, as the rocket need not contend with high seas. Ground was ground—flat and unmoving. But there was a major disadvantage, too. In returning the Falcon 9 rocket to land, it would fly near cruise ships in Port Canaveral, the National Reconnaissance Office’s multibillion-dollar Eastern Processing Facility, and numerous other launch pads and valuable assets.

SpaceX acquired an old Cape pad, Launch Complex 13, in February 2015 for the purpose of returning the rocket. Trip Harriss, who had been with the company since its days on Kwajalein and the Falcon 1, now bore responsibility for Falcon recovery efforts and led the build-out of Landing Zone 1. He and Bala Ramamurthy also worked to convince the Range commander that SpaceX should be allowed to aim rockets at the Air Force station, a first.

“As the Range commander, you’re used to rockets going away from you,” said General Wayne Monteith, who commanded the 45th Space Wing at Cape Canaveral from 2015 to 2018. “So when you see one that’s 180 feet tall and coming back, as the person responsible for the safety of everyone on that installation, you start to get a little worried. Your career dissipation light starts blinking.”


Harriss and SpaceX provided data to convince Monteith and other Air Force officials of the project’s safety. It helped that the ocean landings, although not successful, came close to hitting the drone ship. So Monteith felt confident that if SpaceX damaged any property at the Cape, it would be the company’s own equipment. SpaceX also demonstrated that the vast majority of the booster’s return flight profile was over water. If something went wrong, a destruct signal could be sent to the first stage before it threatened anything on shore.

Before he signed off on a landing attempt, however, Monteith had to convince his supervisors the plan was safe. As the weeks ticked down toward SpaceX’s return to flight, opposition started to get louder from the National Reconnaissance Office, which was concerned that vibrations from the rocket’s sonic boom—as it slowed from supersonic to subsonic speeds—would damage the delicate work being done in its payload processing facility.

Range safety analysts predicted the Falcon 9 flyback would produce a sonic boom comparable to the major 2013 Chelyabinsk meteor event in Russia, damaging buildings and homes in the Cape Canaveral area and causing widespread damage. There was little data to refute these claims, which came as part of a lengthy and official-looking 100-page report defending the analysis. Alongside those claims came a stark warning that the United States would lose assured access to space, possibly for years, due to damage of critical launch facilities.

Why was there such caution? Unless the military is in the midst of a war, it is a risk-averse operation. Asses are on the line if there is a screwup. Monteith knew the buck stopped with him and that by making the call to allow SpaceX to land at Cape Canaveral, it was his particular ass in the line of fire.


“During a meeting, a commander’s call, I stood up and said I believed this was the right thing to do,” Monteith said. “In doing so I understood that if anything went wrong, I would be fired.”

In early December, SpaceX received a green light from the Air Force to not just launch a missile, but to bring one back to the station. This is pretty remarkable, as SpaceX was flying a brand-new version of its Falcon 9 rocket, which was returning to flight after a launch failure, with densified propellant onboard for the first time.

Predictably, the run-up to the launch was chaotic. After SpaceX solved the rod end issue with NASA and the Federal Aviation Administration and obtained permission from the Air Force to dive-bomb its rocket back to the Cape, it still had to refine new procedures for densified oxygen.

One challenge with the densified oxygen was the inability to “recycle” a launch attempt if there was some technical or weather problem at the appointed time for liftoff. Once the super-chilled propellant was loaded onto the rocket, SpaceX had minutes to launch, or the liquid oxygen would become too warm. Although there was spare oxidizer in the LOX ball, offloading the warmed liquid oxygen from the rocket to this storage vessel would spoil the colder oxygen there. Dumping all the rocket’s LOX was not an option, as this would damage pipes and other launch site infrastructure.

As launch director, Lim also kept a concerned eye on the calendar. SpaceX had targeted the night of December 21, 2015, for the return to flight launch. The rocket would loft eleven satellites for the telecommunications company ORBCOMM into low-Earth orbit, with a total mass of about 4,500 pounds. This was a light enough load for the Falcon 9 to have plenty of spare fuel to return to Landing Zone 1. Everyone had worked intensely to get ready for the launch and was counting on a few days off over the holidays. Many talked of quitting if they did not get a break soon.


“We were desperately trying to save Christmas,” Lim said. “Our employees had been working months on end, and I worried that about a third of them might leave. If we scrubbed and plowed through the holidays, it would just have been murder.”


The Falcon 9 rocket for the ORBCOMM-2 mission undergoes final processing in December 2015. Credit: SpaceX

Once again, Lim directed the launch from inside the company’s control center about eight miles from the pad. In the early years, there were two principal leaders during a launch, the director and the chief engineer. This created a tension on launch day, as the director served as the “gas pedal” and the chief engineer had a more cautionary role as the “brake.” Koenigsmann typically served as chief engineer of launch, but due to his focus on the CRS-7 failure, he delegated the role to Robb Kulin.

Koenigsmann and Musk watched proceedings from inside the control room. The countdown was tense. In the final minutes before liftoff, scheduled for 8:29 pm local time, a camera inside the interstage area between the first and second stages showed drops of a pale blue liquid dripping down. This was a novel problem due to working with densified propellant for the first time, and it might indicate a number of bad things. Kerosene leaking might result in a fire. Liquid oxygen could lead to an explosion. Reviewing data and video, the launch team determined it was probably “liquid air,” or air that had been cooled down to cryogenic temperatures by the frigid tanks. Hastily, the launch team discussed whether to scrub and investigate the leak.

At T-1 minute, Koenigsmann turned to Musk. “You’ve got to make a decision,” he said.

Almost invariably, Musk delivered his decisions with confidence. He liked to command while others obeyed. But in this instance, with everything on the line for SpaceX, he responded casually, almost dreamily. “Well, I guess we’re going,” Musk said.


The upgraded version of the first stage performed perfectly. After dropping off the second stage, the rocket burned for home, dropping out of the black night down to the Florida coast. Nearer to land, from the vantage point of the launch control center, the rocket disappeared behind the tree line with a spectacular orange glow and a huge cloud of dust. Then there was a huge, building-shaking blast.

“That scared the shit out of us,” Koenigsmann said. He and Musk thought the rocket exploded. Musk’s countenance sank, despondent and disappointed.

Someone on the launch team suggested they check the video feed from the landing site. It told a happier story. The Falcon 9 rocket? It was there, standing upright on the landing pad, smoking in the mild Florida evening.

They had been fooled by the reentering rocket’s sonic boom, which had been delayed a few seconds traveling to the launch control center. The room erupted in applause and cheers.

Musk’s mood reversed entirely. He became delirious with joy, absolutely smashed full of happiness and pride for persevering long enough to see this moment. His faith in bringing rockets back from orbit and landing them, so often questioned, had been validated. Like a kid in the candy store, he kept pressing Lim and the launch team to go out to the landing pad and see his beautiful rocket. Three different people who had been with Musk for years said they had never seen him happier.

SpaceX had negotiated range safety protocols with the Air Force in the event of a landing. The rocket still had explosives onboard, including TEA-TEB ignition fluid, the rope-like flight termination system, as well as liquid oxygen and kerosene. A safety team had to secure the rocket first. But in less than an hour, Musk, Koenigsmann, and others, including Kulin, Harriss, Shana Diez, and Lee Rosen, donned hard hats to go running and skipping and dancing across the landing pad. As they danced about, they noticed there had been no apocalyptic meteor event, nor property destruction of any kind. Even the basic windows in an office trailer by the landing site bore nary a scratch. The launch of the new rocket and its unprecedented landing were a complete success.


“It’s hard to describe how epic this comeback was after our first Falcon 9 launch failure,” Koenigsmann said.

As he, Musk, and the others marveled up at that sooty rocket, illuminated by flood lights beneath dark and starry skies, they must have wondered if this moment could ever be eclipsed.

“It just felt so massive.”


They were pretty excited back in Hawthorne, too. As the rocket touched down, hordes of employees crammed into the factory floor just outside mission control started chanting, “U-S-A! U-S-A! U-S-A!” A raucous celebration ensued.

And why not?

The four thousand employees of SpaceX had wrought nothing short of a miracle in the six months preceding that night. The company worked on four separate, massive projects in parallel, packing their final exams into that single launch. Riding onboard the Falcon 9 rocket in late December were the company’s return to flight mission, a significant upgrade to the Full Thrust version, an unprecedented oxygen densification program, and the first landing. They saved Christmas, to boot.


The historic ORBCOMM launch and landing delivered one of the most cathartic and breathtaking moments in SpaceX history. I do not believe it is possible to overstate the significance. With its fate on the line, the company roared back from a terrible and financially disastrous failure. And, on the very same flight, SpaceX accomplished something no company, or country, had ever done before. Until then, SpaceX had followed in the footsteps of NASA and others in launching rockets, flying satellites into space, and landing spacecraft in the water. Sure, it did so in cheaper and innovative ways. But these were well-trodden paths. No one had ever, ever launched an orbital rocket and landed it back on Earth minutes later.

Until that night.

Catriona Chambers came to SpaceX in early 2005 as an electronics engineer. Within months on the job, she picked up responsibility for the Merlin engine computer on the Falcon 1 rocket. On that small rocket’s very first launch, there was a sensor that measured atmospheric pressure. After reaching space, the first stage would descend back to Earth, and when the sensor detected a thickening atmosphere, it would command deployment of a parachute. She and everyone who worked on the rocket knew this was preposterous. The rocket would probably never survive, and the parachute would be practically useless. But Musk pushed hard for reuse from the very beginning of SpaceX. Now here she was, almost eleven years later, observing it actually happen. As director of avionics, she watched with her team as the first stage landed, feeling the weight of history as she hugged and high-fived her friends.


Robb Kulin stands in front of the first landed Falcon 9 rocket in December 2015. Credit: Hans Koenigsmann

“That was the point where it really sunk in that we had been working on this for so long,” Chambers said. “It just felt so massive, and I was so excited. And then I realized I needed to calm down.” She was eight months pregnant, after all.

Like a lot of SpaceX employees, Zach Dunn felt both exhilaration and relief at the launch of the ORBCOMM mission. He had taken over the propulsion department in February, with the aim of completing the Merlin engine upgrades for the Full Thrust version of the Falcon 9. Within the first couple of weeks on the job, two engines blew up. Then the CRS-7 launch failed, and Dunn was thrown into the tortuous investigation. Finally came the arduous campaign to ready the new rocket and update the launch site for densified propellant.

This gave the propulsion team fits right up until the launch date. On December 18, the company had to abort three separate attempts at completing a static fire test. The launch team was still learning, on the fly, how to load and offload super-cold liquid oxygen, when Musk came into the control room. Invariably, his presence raised the level of tension and urgency. Dunn explained to Musk that by the time the rocket was ready to ignite its engines, the propellant was warmer than the engines were expecting.

Musk told him to run the test anyway.

“My engine team was telling me this was not the right thing to do,” Dunn said. “That we weren’t going to get the data we needed from the test. The pressure from Elon was just absolutely intense.”

On launch day, Dunn sat next to Shotwell in Hawthorne’s mission control. As soon as the booster touched down, Shotwell leapt to her feet, joining the merriment. After a few minutes of also celebrating, Dunn left and walked across the factory floor to the propulsion area. About five dozen engineers were there, almost all of his propulsion department.


“It had been a hard fucking year,” Dunn said. “This was my hardest year at SpaceX, leading propulsion, going through those failures and trying to keep the team together, and the pressure of getting back to launch. It pushed my leadership and technical abilities to their limits. My interface with Elon was more direct and more intense than it had been before. It had taken a toll.”

As Dunn walked toward his desk, the other engineers, one by one and then in a rush, stood up and applauded him. A standing ovation. It was completely unexpected. Dunn had come into the department as an outsider, having led pad operations at Vandenberg. There were a lot of egos and a lot of brains in SpaceX’s propulsion department. During the preceding ten months, Dunn had fought with this team as well as for this team. He’d won some. He’d lost some. But after that night, he was no longer just their leader. He was one of them.

“Man, I’ve never felt better in my life,” Dunn said. “It felt incredible to experience that after the hardest fight that I’ve ever had professionally.”