NASA’s Exploration Ground Systems (EGS) program and prime launch processing contractor Jacobs completed the Wet Dress Rehearsal (WDR) countdown demonstration test for Artemis 1 at the Kennedy Space Center (KSC) on June 20. The fourth run of the test “cut off” at T-29 seconds, several seconds before the desired stop time but after demonstrating the desired objectives of the test.
A second leak in the liquid hydrogen (LH2) tail service mast umbilical (TSMU) connection from the Mobile Launcher to the Space Launch System (SLS) Core Stage prevented a fully identical terminal countdown sequence and troubleshooting abbreviated the two planned countdown runs down to one. Still, the EGS and Jacobs launch team, along with support from the SLS and Orion programs and contractors was able to isolate the leak, workaround the issue, and complete test objectives.
After reviewing the results of the June 20 run NASA declared the test complete. The Artemis 1 Orion/SLS vehicle is now being prepared for launch inside the Vehicle Assembly Building, with a first launch attempt still possible as soon as late August.
Hydrogen leak forces terminal countdown workaround for RS-25 engine start criteria
During the fourth WDR run on June 20, the Exploration Systems Development (ESD) programs and contractors achieved the ground-based launch control systems test objectives necessary to complete the test, reaching T-29 seconds in a single terminal countdown run before the onboard SLS flight computers cut off the countdown.
Another leak in the ground to Core Stage liquid hydrogen umbilical connection prevented the vehicle from being in a perfect configuration for launch, but ground system commanding and choreography was the focus of the countdown demonstration, and the launch team was able to work around the issue for the purposes of the test.
The launch team was able to fully load the SLS Core Stage at KSC for the first time on June 20; although the stage had been fully-fueled three times at Stennis Space Center during the Green Run campaign, the new EGS launch control system is a separate development program and manages not just the Core Stage, but the full SLS launch vehicle, the Orion spacecraft payload, and the large infrastructure required for a full launch campaign.
SLS will take the Orion spacecraft (seen here shrouded by its Launch Abort System) on a 3 to 6 week mission to the Moon on the Artemis 1 mission. (Credit: Julia Bergeron for NSF)
During the June 20 test, both the Core Stage and Interim Cryogenic Propulsion Stage (ICPS) were fully loaded with their liquid oxygen (LOX) and liquid hydrogen (LH2) propellant, kept replenished for most of the remainder of the countdown, and then pressurized for flight in the final minutes of terminal count.
The leak was discovered with the start of liquid hydrogen thermal conditioning of the RS-25 engines. A “bleed” flow of LH2 is circulated through the engines once the Core Stage liquid hydrogen (LH2) tank is full and in stable replenish. During the process of starting that, a leak was detected at the quick disconnect of the LH2 bleed line, which is one of the connections on the LH2 tail service mast umbilical (TSMU) to the Core Stage.
“The bleed function is where we take liquid hydrogen from the Core Stage and flow it down to through the engine section to cool down the inlet of the engines and the criteria is that you’ve got to have the engines at a specific range of temperature and pressure in order for those engines to start,” Phil Weber, Senior Technical Integration Manager for EGS, said during a June 24 media teleconference. “In stable replenish we went ahead and tried to activate that bleed function and we found that we had a hydrogen leak.”
The RS-25 engines are flight veterans of multiple Space Shuttle launches, but the SLS Core Stage Main Propulsion System (MPS) is different than the Shuttle orbiter MPS and the liquid hydrogen bleed flow is one of the differences. The reusable Shuttle orbiters carried LH2 recirculation pumps that routed the hydrogen bleed flow back into the External Tank, but those were deleted from the expendable Core Stage design.
For SLS, the hydrogen bleed is dumped overboard through a vehicle-to-ground line in the LH2 TSMU. During the Green Run campaign, Boeing and NASA engineers discovered that a “kick start” procedure was needed to establish a full LH2 bleed flow.
“When they initially designed the bleed system, they thought that by just opening up the [overboard bleed] valve that the gravity would be enough to draw the hydrogen down through the engine inlets and out over this overboard dump,” Weber explained in a June 28 interview with NASASpaceflight. “But what they really found at Stennis was that because of the routing of the lines inside the engine section, [for] a couple of the engines it didn’t work.”
The two Mobile Launcher tail service masts and their umbilical connections to the SLS Core Stage are seen during the July 2 rollback to the VAB. Credit: Stephen Marr for NSF.
“[The Green Run team found] they had to close the vent valve on the top of the hydrogen tank and elevate the pressure a couple of [pounds per square inch (psi)] and that’s what they call a kick start. That way you get all four engines going at once, getting the bleed [flowing] through them.”
“We had the valves open and so some of the hydrogen was coming out through the lines and we had no leak, but as soon as she [the LH2 operator] closed the vent valve and bumped [the pressure] up a couple of psi to do the kick start that’s when we saw [the leak rate] start climbing and it was a pretty rapid increase,” Weber said. The LH2 console operator was closely monitoring the situation and reduced the pressure before the leak reached the four-percent concentration limit.
“She was right on it and opened the vent valve on the top of the hydrogen tank just in time, so we got to 3.9,” Weber, the Lead Launch Project Engineer (LPE) at the Integration Console for the Artemis 1 launch, said. “That’s what we tell them to do; she did it exactly right.”
The leaking hydrogen was detected by the ground-based hazardous gas detection system. “The bleed quick disconnect and the fill and drain quick disconnect are adjacent to each other on the ground plate that we pull away, and they actually share a common cavity that gets purged with helium,” Weber noted. “We have a line that draws out of that [and] goes down to our haz gas mass spectrometer.”
The leak in the LH2 bleed line was measured in the cavity between the umbilical plates, supporting all the lines between the LH2 tail service mast and the Core Stage engine section. “There’s an onboard valve that you can open and then there’s a quick disconnect (QD),” Weber explained.
“It wasn’t any valve malfunction, it was the QD itself and so it’s right at the outer moldline, the skin of the vehicle, where the ground plate mates up and the ground half of this quick disconnect presses up against the side of the flight-half. The flight half of the QD flies with the vehicle; the ground half stays with the plate and comes back into the tail service mast with us.”
(Photo Caption: A view of the liquid hydrogen tail service mast umbilical connection used at Stennis Space Center during the March 18, 2021, Green Run Hot-Fire test. The two main propellant lines can be seen, the larger fill and drain line on top and the bleed line on the bottom. A leak at the umbilical quick disconnect for the bleed line at KSC prevented the engines from meeting their start conditions for the June 20 WDR test.)
“That’s where the leak was, right at the quick disconnect so we were seeing it inside the cavity between the two plates, the flight plate and the ground plate,” he added.
“When we were doing the troubleshooting and we started closing the valves, we shut off the fill and drain and then isolated it so only the bleed was open and at that time did a pressurization and the leak came right back so we’d isolated it and said there’s no fuzz on it whatsoever, it’s the bleed.
That’s the kind of isolation you do to make sure you’re chasing the right QD.”
After trying troubleshooting ideas like a thaw/freeze cycle on the QD to see if it would stop leaking, the launch team decided it was getting late in the work shift for many team members and to press ahead towards a single terminal countdown run instead of two.
Rather than holding at T-33 seconds and recycling for a second run, they would try to take a single terminal countdown run as far as possible.
The onboard bleed valve was closed in order to prevent hydrogen from leaking through the quick disconnect, but that also meant that the LH2 engine conditioning could not be performed.
The two LH2 storage spheres at LC-39B. The new 1.25 million gallon tank is on the right while the original 850,000 gallon tank is on the left. (Credit: Nathan Barker for NSF)
Thermal conditioning of the RS-25 engines with LOX and LH2 takes hours and occurs during the fueling and replenish sequence for the Core Stage propellant tanks.
The conditioning continues into the final countdown sequence, which helps the RS-25 engines reach their required temperature range and pressure range for start in the seconds before ignition.
Without the LH2 engine conditioning, the engines would not be in their ready-to-start condition when the ground and flight sequencer applications checked their status.
The engine controller unit on each RS-25 captures, monitors, and reports its own current engine status, from overall health to valve positions to temperatures and pressures at critical locations.
In preparation for running the 10-minute terminal countdown sequence, the launch team had to “mask” those launch commit criteria checks in the Ground Launch Sequencer (GLS) that controls all but the last 30 seconds of the countdown.
By masking the checks, the GLS would see the criteria was not met but would continue counting rather than stop. Control of the vehicle is handed off from the GLS to the SLS flight computers at T-33 seconds and the SLS flight software begins the onboard Autonomous Launch Sequence (ALS) at T-30 seconds.
The onboard flight software cannot be masked to proceed if it finds a launch commit criteria violation during ALS and the launch team knew before they started the terminal countdown that if it reached that point at T-30 seconds, that would be where it came to a halt.
The engine controllers saw that the pre-start conditions weren’t being met and late in the countdown they would be reporting a Major Component Fail (MCF) condition to the ground and flight computers that they were not in a ready-to-start configuration.
“We knew in advance that we were going to see the MCF hits as soon as we start monitoring for it and we knew ALS was going to pick those up and as well, so that’s why [the cutoff at T-29 seconds] wasn’t a surprise,” Weber said.
A separate issue with heaters on the liquid oxygen feedlines leading to the LOX inlets for the RS-25 engines meant that neither LH2 nor LOX start conditions would be met for the test; however, the engines were never intended to be fired during the Wet Dress Rehearsal.
SLS Core Stage test objectives demonstrating the complete engine pre-start conditions and the ignition sequence were completed twice during two Hot-Fire tests in the Green Run campaign at Stennis in January and March, 2021.
In addition to masking certain launch commit criteria parameters, the launch team also had to discuss the implications of the secured Core Stage LH2 bleed system being in a non-flight configuration. One of the issues was pressurizing the LH2 tank for launch.
Crawler Transporter-2 rolls the Artemis 1 vehicle and its Mobile Launcher back to the VAB on July 2. Credit: Thomas Burghardt for NSF.
The LOX and LH2 propellant tanks for both the Core Stage and the ICPS are pressurized before ignition and liftoff, but the Core Stage LH2 system was not in a standard configuration due to securing the bleed to stop the quick disconnect leak.
“One of the things that we really wanted to see was [LH2] pre-press, that our ground system that puts helium into the [LH2] tank to pressurize it for flight would really function and meet the mark,” Weber said. “But the way that was supposed to happen was with the bleed open, and so that was part of what the guys had to run over to Firing Room 3 and go test and make sure that was really going to work.”
“The Boeing [engineers] that do stress analysis had to go through and dig out their [documentation] and make sure this wasn’t really going to do anything detrimental to the flight hardware if we proceeded through this,” he added. “I was really impressed with how the team [handled that].”
“[I said] we’re going to give up on chasing this leak and we’re going to go ahead and close the valve, the bleed valve, and I want us to get all the way down to ALS if we can get there. And so [NASA SLS Chief Engineer Dr. John] Blevins said ‘I got it’ and he went off and huddled with his team and started working that while our guys started working the procedure to make sure that we could actually pull it off and do it and we did, we got the [LH2] tank pre-pressed.”
As expected, immediately after starting ALS at T-30 seconds the SLS flight computers made their first check of vehicle status to see if it was in a launch ready configuration. Weber said that they cutoff the countdown at T-29.98 seconds when they saw that the engines were not ready to start and handed control back to the Ground Launch Sequencer, which fully stopped the countdown at T-29 seconds.
After analyzing the results of the June 20 test run, NASA declared that the test was complete and that the team could begin to reconfigure systems from a launch countdown set up to get the vehicle and ML ready to rollback to the VAB, which occurred on July 2.
Gaseous nitrogen switchboard — not the supply — delays the start of fueling
For the third time in a row, starting the hours-long process of tanking the two SLS cryogenic stages was delayed by an issue with the gaseous nitrogen (GN2) purge. The volumes in the SLS vehicle around the propellant tanks are kept under a dry purge most of the time; they are often work areas before launch and low-humidity, room-temperature dry air is used to keep moisture off of the equipment in those areas.
A few hours before the vehicle propellant loading starts, the purge is switched from dry air to GN2 for the additional need to inert the areas, given the presence of rocket propellant. The cryogenic propellant also lowers the temperature in the volumes, in some cases close to zero degrees Fahrenheit, so the GN2 is also warm and dry so that the electronics distributed through those areas on the rocket stages stay dry and don’t freeze.
On June 20, a “go” was given to proceed with the tanking process, beginning with Core Stage liquid oxygen loading; however, an issue with a controller of a valve that helps remotely manage distribution of the GN2 purge supply came up at that time. “It’s not like a controller card, circuitry-type thing,” Weber noted.
“It takes an electric signal and it’s a mechanical actuator that moves the valve and the regulator and that had failed. They couldn’t control consistently with it and so that was the problem that they annunciated to us right before we got into tanking.”
In contrast to previous problems originating at an offsite Air Liquide plant that supplies GN2 to the space center, the problem controller and valve were at the Converter Compressor Facility on KSC grounds. “There’s a pipeline [from the Air Liquide plant] that runs into the center and it runs roughly 6000 psi,” Weber explained.
“When you go down the road next to the crawlerway you pass the Launch Control Center and about a mile down the road there’s a series of buildings off to the left on the north side of the crawlerway. That’s where the center [manages] the gases coming in for all the uses on the center. It’s like a utility kind of thing where [Exploration Ground Systems is] one company in town and the town really supplies the utilities to us.”
Water is sprayed on the crawlerway in front of the crawler returning the Artemis 1 vehicle to the VAB on July 2. Credit: Stephen Marr for NSF.
“That’s where the [high-pressure] GN2 pipeline comes in and once it gets there they split it into what they call a north leg and a south leg and that’s so that they have redundancy. You can run on only one at a time but if one were to fail you can switch to the other and get yourself through the situation, so it’s one of those cases where you have built-in redundancy.”
Under the circumstances, the launch team decided to replace the controller before beginning the tanking process so they would have redundancy in case something else broke down later in the process. “There is not any kind of a launch commit criteria or anything like that that says you have to have that redundancy, so it was more of an engineering judgment call,” Weber noted.
“I said I really think we ought to fix this before we get started and they’d estimated it might take them an hour or two and I’d bite the bullet and say you’ve got to get this fixed because I really want to make sure we get through today, we can’t just keep doing this. [Launch Director] Charlie [Blackwell-Thompson] agreed, that made sense to her as well.”
“We never switched back to air, we were just running on the alternate leg and GN2 purge was up the whole time,” he added.
On June 20, the countdown remained in the pre-tanking hold at T-6 hours and 40 minutes for almost an extra two hours, but with a maximum two-hour-long launch window to work with for Artemis 1, the launch team is looking at what they can do to stay on schedule when it comes to launch day. “This is the kind of stuff that Charlie and I have been sitting down this week [to talk about],” Weber said.
“[Management wants] to know exactly what we’re doing to make sure that we’re going to get started on time when we get out to launch, so we’ve been working through this plus a number of other things to make sure.”
Fixing leak, getting ready for launch
The vehicle was rolled back to the VAB on July 2 to fix the hydrogen leak and finish launch preparations. NASA has not yet picked a single date for a launch attempt for the Artemis 1 lunar test flight, but is still targeting a launch period that opens on August 23 and extends through September 6.
The Mobile Launcher with the Artemis 1 vehicle approaches High Bay 3 of the VAB near the end of its July 2 rollback from Launch Pad 39B. Credit: Stephen Marr for NSF.
Initial thinking about the possible cause of the leak at the LH2 bleed line quick disconnect is with the teflon seals. “It turns out that the teflon seal that’s in that QD was the same one that was used at Stennis and it’s got some run-time on it as well as a couple of trips to the pad,” Weber said in the June 24 teleconference.
“We’re going to go in and change out those soft goods and we’re still talking with John Blevins’ team about while we’re in there and got it apart, it probably/maybe makes sense to go after the 8-inch soft goods, as well, that’s the fill and drain line. It’s going to be more than just popping it open, swap out the seal, and put it back together.”
“We’re going to take it apart and take a lot of measurements, do a lot of imagery, and there’s the potential we may pull the ground-half of the QD that has a bellows in it with like a spring-function,” Weber added. “That’s what actually forces the QD together to create the compression to make the seal, so we may take that off on the bench and use a tool to measure the compression force itself and if that’s a little low maybe we’ll swap that out with one that has a higher compression.”
With the vehicle and Mobile Launcher “hard down” on the pedestals in the VAB, moveable access platforms are being extended around the vehicle to begin the final stretch of work. For the hydrogen TSMU, the Mobile Launcher deck also serves as a work platform.
“What we’ll do is we’ll re-establish access, we have access stands that’ll go in and then we’ll demate the umbilical plate,” Cliff Lanham, senior vehicle operations manager for NASA’s EGS Program, said in the June 24 media teleconference. “We’ll pull that back and then obviously we have access right there to get into the QD on the flight side.”
“We do expect to put an enclosure around it, from a contamination [protection] standpoint, so we’ll have to build that up as well.” The hydrogen TSMU supplies several services to the Core Stage besides fill and drain and bleed lines, so those functions will need to be reverified after the repairs are completed.
“There’s a pretty significant retest that we have to do because it’s not just these quick disconnects for hydrogen but there’s electrical and pneumatic systems that go through that as well, so we’re going to re-run the Interface Verification Test on that [umbilical], end-to-end, just to make sure that we didn’t disturb or damage anything else in the process of fixing this,” Weber added.