Progress made in restoring station's crippled computers
BY WILLIAM HARWOOD
Posted: April 26, 2001
See our latest story here.
The "extraordinarily good news" surprised station flight director Robert Castle and a team of troubleshooters at the Johnson Space Center, giving flight controllers welcome insight into the system and improving prospects for developing a fix.
"I think we can help you here," Helms said.
"You are making my day," Castle replied. "It's been a long night."
All three of the station's command and control - C&C - computers experienced problems accessing their internal hard drives Tuesday and Wednesday, dropping off line one at a time in domino fashion to the bafflement - and increasing alarm - of ground controllers.
While the station's critical life support systems, its stabilizing gyroscopes and its two huge solar arrays continued operating normally, engineers on the ground lost all telemetry from the C&C system and were unable to send commands to the system.
The computer glitch forced the station astronauts to delay critical tests of the lab's new Canadian robot arm while engineers launched a massive troubleshooting effort to figure out what had gone wrong and what might be needed to regain use of the C&C computers.
Shortly before 11 p.m. Wednesday, the ground successfully uplinked a command that effectively pulled the plug on the machine that had been in control - C&C-3 - forcing C&C 2 to take over as the station's primary computer.
The computer switch over set off an alarm, as expected, that woke up the station crew and for a few minutes it appeared the team was on the road to recovery.
But about 40 minutes later, C&C 2 suddenly stopped sending telemetry, leaving ground controllers in the dark once again. Subsequent efforts to gain insight into the system failed and shortly after the crew woke up today, Castle presented a detailed status report.
"I wish I had better things to talk about, we're still having significant problems with the C&C MDMs," he said. "I know we woke you guys up last night when we finally got C&C MDM 2 up and running. About 40 minutes after that, we lost telemetry from it again and any of our command tests so far have failed.
"So we really don't have any insight into what's going on with any of the C&C MDMs right now. We do have some steps coming up to you in the execute package to try and figure out what is going on."
The station is equipped with three command and control multiplexer-demultiplexer computers t- C&C MDMs - that control the lab's operation. Only one computer is in charge at any given time. A second machine runs in backup mode, ready to take over if the primary computer shuts down or malfunctions, and the third machine is in standby, programmed to shift into backup if the designated backup computer switches to primary mode.
All three computers apparently ran into a similar software-related problem accessing data from their internal hard drives. And that took all three down, one at a time. This morning, Castle gave the astronauts a brief synopsis of the events that led to the electronic impasse.
"We lost C&C 1 yesterday (Tuesday night), of course, due to apparently a mass storage device problem," he said. "C&C 2 ran for a little over eight hours and then e lost to due to again, a mass storage type problem of some sort. It transitioned to 3 and then while operating on 3 we lost telemetry.
"We finally recovered C&C 2 by sending a command to just pull power to C&C 3. It went down and C&C number 2 came up. We then lost C&C 2 telemetry and we have been unable to recover.
"As far as the overall vehicle status, the momentum management (gyroscope system) continues to run and attitude is holding nicely," Castle continued. "When we had 40 some minutes of telemetry, all the lab systems and the U.S. segment systems appeared to be in very good shape."
But without the C&C system in operation, he went on, "we can't do robotic (arm) ops on the station."
Instead, Castle told the crew they would spend the morning continuing work to transfer equipment to and from the Raffaello cargo module while ground controllers continued efforts to recover the C&C system. If those efforts worked, he said, the astronauts would be able to use the new Canadarm 2 space crane to hand a Spacelab cargo pallet back to the shuttle's robot arm for reberthing in Endeavour' cargo bay. Other tests would be deferred to Friday.
"What's the status of the other two MDMs?" asked station astronaut James Voss. "We're on C&C 2 right now. I know you don't have any data, but what's the status of the other two MDMs? Are they up and running, like in standby or backup?"
"Both of them are powered," Castle said. "We don't know where they are. We're guessing they're in diagnostic (mode) but they're not up in standby."
And then Helms, using a laptop computer in he Destiny lab module, gave the team a welcome and unexpected surprise.
"Bob, we are connected to the MDM and it looks like the data is good," she reported. "It says C&C 1 is in standby and C&C 3 in backup."
"OK, well the fact that you're connected is extraordinarily good news!" Castle said. "Since we didn't have IO (input-output) with C&C 1 and 3, I don't know if we can believe their status,. But we can believe that C&C 2 is up and running, which is extraordinarily good news."
"Yeah, I just launched the C&C summary and it launched, the purple Ds disappeared and we received data," Helms said.
"I just tried to launch the C&C overviews display and it also, the purple Ds disappeared and we have data. So I think we can help you here," Helms said.
"You are making my day. It's been a long night."
The computer troubleshooting has wrecked the crew's schedule, forcing the astronauts to put robot arm testing on hold. The new arm cannot be operated without the C&C computers running because those computers are used to provide needed software to the crane's computer control system.
The arm currently is extended, holding a 3,000-pound Spacelab cargo pallet on one end. The pallet, which held the arm during Endeavour's launch, must be reberthed in the shuttle's cargo bay for return to Earth. That work had been scheduled for Wednesday and currently is on hold.
If the C&C system is recovered this morning, Helms will use the arm to hand the pallet over to the shuttle's robot arm for reberthing. Other tests, including a dry run to rehearse the techniques needed to install the station's main airlock would be deferred to Friday.
One other major task that requires an operational C&C system is undocking of the Raffaello cargo module currently attached to the Unity module's downward facing port. Castle told the crew today that in a worst case scenario, they could use a computer in the node itself to operate the latches holding Raffaello in place.
Mission managers are holding open the option of extending Endeavour's mission by one day to complete delayed testing and other work depending on what happens with the C&C recovery effort.
Exclusive video from Baikonur (a bonus for our current mission theater subscribers):
The Soyuz TM-32 spacecraft and booster roll to the launch pad on Thursday in the first rays of the morning sun. The rocket, which will carry space tourist Dennis Tito, is being prepared for launch on Saturday.
PLAY (434k, 34sec QuickTime file)
The Soyuz taxi crew raises the American, Russian and Kazakh flags during a traditional preflight ceremony near the Cosmonauts Hotel in the city of Baikonur.
PLAY (374k, 10sec QuickTime file)
Workers fit the emergency escape rocket to the fairing of the Soyuz booster.
PLAY (369k, 11sec QuickTime file)
More video from Baikonur will appear on this page shortly. Become a mission theater subscriber today.
See the Status Center for full play-by-play coverage.