The Ultimate in Debugging

Mark Rainey: Engineers are currently debugging why the Voyager 1 spacecraft, which is 15 billions miles away, turned off its main radio and switched to a backup radio that hasn't been used in over forty years!

I've had some tricky debugging issues in the past, including finding compiler bugs and debugging code with no debugger that had been burnt into prom packs for terminals, however I have huge admiration for the engineers maintaining the operation of Voyager 1.

Recently they sent a command to the craft that caused it to shut off its main radio transmitter, seemingly in an effort to preserve power and protect from faults. This prompted it to switch over to the backup radio transmitter, that is lower power. Now they have regained communication they are trying to determine the cause on hardware that is nearly 50 years old. Any communication takes days. When you think you have a difficult issue to debug, spare a thought for this team.

No different

[lgordon]

I used to work with outsourcing in India in the early 2000's, I think the communication with Voyager is probably faster.

Re:

[Foundryman]

I just had an issue with Veeam support to debug a veeam powershell command that couldn't be cancelled. That took them over 30 days to get me the solution.

Re:Feature

[Sebby]

So the main transmitter was down, and it switched to the backup transmitter. That sounds like something it should do. Are they sure this is a bug and not a feature?

It wasn't expected to do that - and that's what (why that happened) they're trying to figure out; from earth.com [earth.com] (linked in the story link):

The command aimed to activate one of the spacecraft’s heaters on October 16. However, when there was no response to the command two days later, the mission’s flight team sensed something was off.

Dr. John R. Spencer from the NASA‘s Planetary Science Department and his team discovered that the spacecraft, in a delightful twist, had gone rogue.

Choosing the path of energy efficiency, Voyager 1 turned off its primary X-band radio transmitter and switched to its secondary S-band radio transmitter, conserving power in the process.

Sounds like it safely did a 'fallback' to a state they didn't expect it to perform ('went rogue').

Re:

[jonadab]

They didn't expect that fallback to be *necessary* and so now they have to figure out why the system was lower on power than expected, after executing the previous command, and what can be done about it.

Re:

[RockDoctor]

so now they have to figure out why the system was lower on power than expected

From the information presented so far, they don't know (or even have strong grounds to suspect) that the system power is low. The monitoring software seems to think it is low on power, but is that due to a bug in the software for adding up the various (monitored) power drains? Or due to one of the sensors monitoring the device's power use? Or due to the introduction of a new power drain which isn't monitored? (I'm thinking specifi

Re:

[taustin]

I suspect the baud rate is so slow that it takes days to send - and verify - commands. At this distance, it may be literally slower than a fast typist, and everything has to be verified before implemented or they lose the spacecraft entirely.

There's a lot more involved than just transmission time.

Re:

[iggymanz]

Nasa's site says otherwise for commands and engineering data which is different than science data feed, "almost two days"

https://blogs.nasa.gov/voyager... [nasa.gov]

A command couldn't be any significant length of time (not big chunk of day I mean), there would be too much risk of losing part of it

Re:communication "days"?

[Waffle Iron]

Voyager is 23.0 light hours from the Earth. Round trip comm would be less than 2 days, by two hours. Saying "days" for communication is misleading.

Are you seriously claiming that it is more correct to say "1.92 day" than "1.92 days"?

Re: communication "days"?

[rkww]

Send a command, get a response; send another command which depends on the response; days.

Re:

[Waffle Iron]

No, I'm talking about implication of saying "days to communicate."

Specifically: 1.92 days.

Re:

[Waffle Iron]

Why is it so hard for you to understand that 1.92 > 1?

Re:

[redmid17]

Nah he's right, both technically and commonly.

Re:

[Waffle Iron]

If someone in a business says a project will take days, they do not mean even 2 days, but 3 or more.

That's pure bullshit.

Re:

[Waffle Iron]

You will make supervisors angry.

Only if they're perseverating imbeciles.

Re:communication "days"?

[localroger]

It takes time to transmit and receive messages, since they have to be sent at very slow baud rates with the low power backup system. Also once they craft the message to send, they have to schedule time on the radio telescope to send it -- radio telescopes aren't just sitting around waiting for NASA to call them, they're in constant use, and there are only a few in the world that can make the comm link. So yes, by the time you factor in all the overhead necessary to conduct an exchange, it's days.

Re:

[RockDoctor]

Also once they craft the message to send, they have to schedule time on the radio telescope to send it -- radio telescopes aren't just sitting around waiting for NASA to call them, they're in constant use, and there are only a few in the world that can make the comm link.

They don't use radio telescopes - though the machines may look like telescopes from a distance. NASA's "Deep Space Network" has 3 major site - somewhere in America, somewhere in Spain, and somewhere in Australia, at approximately 120 degre

Re: communication "days"?

[StarrEngineer]

One thing of note, round trip time does not equal communication time. Sending the data takes lots of time, especially when hardware is old and interference slashes bitrate.

Re:

[viperidaenz]

There is only a 16 hour window each day in which data can be received from Voyager. You need to wait for the window to open before sending data. You can't always get a response back in 46 hours if the windows don't align either.
You also need to wait for the transmission of the data, which only happens at 160bps.

So more than "1.92 days" to communicate

Bravo

[Anonymous Coward]

I tip my hat to the engineers at NASA who keep this probe running. It's going to be a shame when Elmo and Vivek slash funding. Vivek still hanging out with the crowd who point blank said they wouldn't vote for him because he wasn't white. https://www.thedailybeast.com/... [thedailybeast.com]

Re:Bravo

[fropenn]

Elmo and Vivek slash funding

I am expecting they will slash funding for NASA while Space-X gets some nice fat government contracts. Bookmark this post and then come back in a year and tell me I'm wrong.

Re:

[jonadab]

You do realize Slashdot doesn't allow commenting on months-old discussions, yes?

Re:

[Tablizer]

We have DeLorean's.

The first command they sent

[Waffle Iron]

No matter what, the first step to try while trying to remotely diagnose any problem is:

"Unplug the power cord, then wait for 30 seconds. Then plug it back in and let us know if you still have the problem."

Re:

[Bu11etmagnet]

In other words, Have you tried turning it off and on again? [youtube.com]

Look on the bright side

[RUs1729]

With such a long time between issuing a command and getting a response they have plenty of time to think and ponder how to proceed. That is an invaluable asset.

The voyager tech is very cool

[Petersko]

When I read about the troubleshooting of voyager last year I was really impressed. The ability to relocate critical code around failed memory just lit up my nerd brain. After almost 25 years in critical systems support, reading this gave me deep respect for the designers.

https://arstechnica.com/space/... [arstechnica.com]

Re:

[AmiMoJo]

Relocating code around failed memory is a nice feature to have, but quite an old one. I remember back in the 90s there were Linux patches to support mapping out failed chunks of RAM, but it goes back much further than that in fact.

It can be done in software if your code can be relocated. Basically you have to either have a CPU that supports long relative jumps, very small code, or a loader that copies the code somewhere and then fixes all the jump addresses. The last one has been around since the 1960s.

You

Re:

[Petersko]

True. The thing I was impressed by was the complete round trip. The ability to peek the memory to identify the failed blocks, redistribute the code into non-contiguous chunks, upload and write the new code, all through the antenna connection at such unbelievable distances.

As you said, not new. But brilliantly done.

Re:

[aRTeeNLCH]

It's like the designers prepared ahead for "ultimate MacGyver" back in the day. "So, you have one Swiss pocket knife, one pair of shoestrings and some non conducting handcreme, what can break that cannot be fixed with that?" "The side radiometric thingamajig, to fix that shoestrings are useless, that would need wooden gambling dice!"

Now that I think of it, considering the timeline, some of these people may have actually helped out the scriptwriters of MacGyver....

Nothing compared to PS3

[VaccinesCauseAdults]

That’s nothing. They should try debugging the SPU vector processors on the PS3 CPU (Cell Broadband) using the horrendous Sony tools and the unmitigated binfire that was the SN Systems debugger.

Real Programmers

[smoot123]

This seems an opportune time to revisit a classic: Real Programmers Don't Use Pascal [ed.ac.uk]. It even mentions Voyager so apparently Real Programmers are alive and well.

Re:

[ShanghaiBill]

Voyager was originally programmed in Fortran.

Some has been replaced by transmitted code written in C.

Re:

[angel'o'sphere]

Funny!!!