'Extreme Programming' Controls Phoenix Mars Lander

pbd points out a story discussing the challenges faced by NASA engineers in designing the code sent to operate the Phoenix Mars Lander. Quoting Computerworld: "On Wednesday, engineers sent up the code to run an actual analysis, but a satellite orbiting Mars, which transfers the data down to the Lander, was offline. Robinson explained that the satellite had been hit with radiation, knocking it into safe mode. 'Space is a harsh environment, and sometimes they just go into safe mode,' he noted. 'It's a minor problem. [The satellite] aborts whatever it was doing and waits for future commanding.' Engineers successfully resent the code on Thursday."

That's coolness

[zappepcs (820751)]

Not sure about others, but designing software and systems (software & hardware) that understand the data they are using, and when conditions are not right AND know how to go to safe mode rather than just fall apart is awesomeness in physical form.

Then just considering it a minor problem is an even higher form of awesomeness :)

As a hobby I build small robotic creations. I can tell you that working with limited resources, and having to deal with the completely unexpected is just brain damage for the fun of it. Testing your new code on the same hardware you've had for months and thinking "I really did not know that you could bend this metal piece with so little force" or "why the hell is it doing that" and find out that you missed a decimal point on force calculations or a typo leaves you reacting with subroutine x instead of y.

Those engineers have to be fscking proud of their work. I know I would be. Some days I look at code I've had running 24/7 for several years and am amazed, not just at myself, but that I managed to find the bugs and fix the "I didn't know that would happen" issues. It's a lot of type and bounds checking to be certain, but something always creeps in.

I say they should be proud. The Mars missions have been nothing short of exemplary awesomeness. When they figure out the failure they did have, I'm certain that the absolute goodness they built into it will help reveal totally amazing discoveries about why there was a failure. Nothing simple like metric conversions, or wrong alloy for the screws.

Makes me want to work with them. :)

Re:

[Innova (1669)]

The Mars missions have been nothing short of exemplary awesomeness.

I couldn't agree more....with one exception [wikipedia.org]

Re:That's coolness

[zappepcs (820751)]

Actually, considering it a minor problem is not irresponsible risk. When you are that cock sure of the equipment you designed that you can think of it as a minor problem when space radiation knocks your system into safe mode for a bit, you have awesome mad design skills, not risk taking behaviors.

Yes, they used tax payers' money, and I'm pretty glad that I helped contribute in some way. Exploring our solar system and beyond with instrumentation is an absolutely needed step to find more space for all the humans on this planet. Not sure if you have noticed, but there are a few things we are running out of here despite China's one child rule.... room, food, fuel, and some other less dramatic things. Mother nature has a way of balancing things, so she'll kill a lot of us off. If we want to continue growing, expanding, space is the only viable option with current trends and technology. Improving the technology we command and the information that we have is the only way to viable improvements. The space exploration programs aim to do this in very calculated ways, very methodical steps to discover new information on a cost per answer type basis.

It is NOT irresponsible. There are a lot of irresponsible uses of tax payers' money, but trying to expand human knowledge and capabilities is not.

Re:

[iminplaya (723125)]

Not sure if you have noticed, but there are a few things we are running out of here despite China's one child rule.... room, food, fuel, and some other less dramatic things.

We are not running out of them. We are only mismanaging them. There's lots of fresh water falling onto the oceans and empty land we can bring together.

Re:

[homer_s (799572)]

Yes, they used tax payers' money, and I'm pretty glad that I helped contribute in some way.
That is the issue, isn't it? You might be glad, but there are people who do not want to contribute and they were forced to do so.

Not sure if you have noticed, but there are a few things we are running out of here despite China's one child rule.... room, food, fuel, and some other less dramatic things. Mother nature has a way of balancing things, so she'll kill a lot of us off.
This [wikipedia.org] might interest you.

Re:That's coolness

[satoshi1 (794000)]

Linux may not be ready for the desktop, but it is ready for the surface of Mars.
-Me

You know, I am actually cool with that. A stable operating system running a mission critical device thousands of miles away is important, everyone who has contributed to the Linux kernel in some way should be proud that their work is enabling us to learn amazing new things about our red neighbor. I think it says a lot that Linux was chosen over some internallly developed OS.

Re:That's coolness

[zappepcs (820751)]

For a Friday night, that is a very nice thought. A project funded by the public, and at least in part, built by the public. Nothing says OSS works better than that. At least I can't think of anything that does... very nice.

Re:That's coolness

[kperson (771747)]

"A stable operating system running a mission critical device thousands of miles away..."

Just where the hell do you think Mars is??

Re:

[roguetrick (1147853)]

Hey, after it gets larger than 10 basements, we lose the picture.

Obligatory PA reference...x2

[adamofgreyskull (640712)]

Google > NASA for directions
Mars [google.com] is roughly 3715 [indo.com] miles from me, as the deepcrow [penny-arcade.com] flies.

Re:That's coolness

[funwithBSD (245349)]

I just hope none of it was LGPL 3.

I think that would mean that RMS owns Mars.

Re:

[Tolkien (664315)]

Better him than the gov't, odds are they'll claim ownership, despite not having the right to do so.

Re:

[funwithBSD (245349)]

So he has Mars ownership given to him through the rights granted him by a government that has no right to claim it for itself...

So how exactly does he have a right conferred to a government that does not have the power to do?

Interesting theory, but not sure it would stand up. Also, good job ruining a joke, eh.
 

Re:

[RobBebop (947356)]

I think it says a lot that Linux was chosen over some internallly developed OS.

No. Internally developed OS's still have their place in complex military and/or science systems. What I think is important to note is that Linux was chosen over other COTS options such as VxWorks and Green Hills.

The project I am working on uses Linux as a development platform to create a device with a fully internally developed OS. I am fairly certain Linux was not chosen because it wouldn't be able to meet timing requirements without being heavily modified/stripped-down, anyway.

Re:

[johannesg (664142)]

Did they release the drivers for all the onboard hardware, though? What if we want to build our own Mars lander and mod the digger a bit, or use a different landing sequence?

Re:That's coolness

[harlows_monkeys (106428)]

TFA appear to be wrong. It runs VxWorks 5.2 [windriver.com].

The confusion probably arose because Wind River also sells a Linux version, and the press sometimes confuses that with VxWorks.

No mention of Extreme Programming

[by Anonymous Coward]

This article doesn't state the lander was programmed with Extreme Programming techniques.

There's just one sentence which says something about "presents extreme programming challenges" which is the closest this article comes to mentioning Extreme Programming.

Re:

[kcbanner (929309)]

I was dissapointed as well. I thought they might have been writing code on the fly to counter some bad thing happening. All we basically got was a compile delay :(

Re:

[serge587 (1038264)]

Then simply add some suspensefully dangerous music in the background and you have the next generation of thriller movies!

Re:No mention of Extreme Programming

[Ethanol-fueled (1125189)]

...whilst hand-coding a real-time, multitasking OS in assembler while suspended over a tank of water filled with hungry sharks. Laser beams optional.

Re:No mention of Extreme Programming?

[conureman (748753)]

I gathered that if they uploaded a bad string, the lander gathers dust for a wasted day. Beaucoups dinero riding on correctness. Matthew Robinson's team look like heroes, this is like the triumph of the coders.

Re:

[by Anonymous Coward]

Having a plan that requires 1000 lines of new C code every day is f'ed up beyond all belief. Someone doesn't know what they are writing about.

Wow, yeah. You actually inspired me to read the article. I thought "LOL, yeah right, AC must've made a typo". But I'll be god damned, the article really says that.

I always thought I was a fairly decent coder, but I might have to re-evaluate if other people can do 1000+ lines of debugged, production quality code every day.

Re:

[Hognoxious (631665)]

Having a plan that requires 1000 lines of new C code every day is f'ed up beyond all belief.

Not really. Now if it was Visual Basic...

Embedded programming is hard!

[compumike (454538)]

Sure, there's complicated software systems, but when something has to deal with hardware too, there's just a lot more to think about in every action. That's why engineers make abstractions -- enough to keep their part of a particular project in their head.

But shouldn't the "safe mode" limits be independent of the particular operation as much as possible? In software engineering, the people writing test cases are often not the same as the people developing the code, and for the reason that they want to match both to the spec, not to each other.

--
Hey code monkey... want to learn electronics? [nerdkits.com]

Units?

[pythonist (1289628)]

I'm wondering what units they are using? American? SI? or Martian?

Call me a pragmatist...

[Dice (109560)]

... but when I hear terms like "EXTREME PROGRAMMING!!!1eleven" I just roll my eyes and stop listening.

Re:

[Jeff DeMaagd (2015)]

There has to be some good jokes there. I think some members here can come up with solid promo dialog for an MTV-style code-off.

Don't miss it! It's EXTREME!

Re:

[justinlee37 (993373)]

I don't think the actual article has to do with the XP methodology, though, the headline writer was just too dumb to avoid referencing XP 'cause he didn't know about it.

Why do they have to do this much coding?

[by Anonymous Coward]

According to TFA, each member of a team of 30 engineers must write ~1500 lines of code, per day, in order to control the robotic equipment on board. This coding is done in C. Does this strike anyone else as a brain dead way to control a robot on a day to day basis? Why do they have to write the commands in such a low level language. Can't they abstract this into a more manageable format?

Re:Why do they have to do this much coding?

[Tablizer (95088)]

I agree, there's something odd about using C to "program" daily instrument movements. They just need coordinate tables, not C programming. If there are conditionals, they cannot be complex enough to justify C, can they? The rovers didn't have very many conditionals for their sequences based on what I read. If something is not as expected, they usually have the rover stop and wait for Earth confirmation, NOT make the robot select contingency options (unless automated driving is on, but it still has low tolerances before going into wait-mode). I suspect the article is confusing the work of two different teams: the "operating" software developers and the instrument sequencers. Some journalists don't know C from a horse shoe......oh wait.
     

Re:

[GrayNimic (1051532)]

From what I understand, Phoenix's on-board memory is sufficiently limited that they cannot keep the entire set of "normal" operations on-board (ie, the library would take up too much space). So for each day's actions, they have to include the relevant subsets of the library in with the command code. If those library excerpts are included in the line count (it would make sense from a "upload size" perspective, if not from a "programming" perspective), that could explain it.

Also, be careful with comparis

Re:

[Saeculorum (547931)]

I'd suggest looking at pricing for radiation-hardened parts sometime. A radiation-hardened 2 Megabyte SRAM on a programs I am working on costs around $40,000, and it's not even particularly quick. There are very real code size considerations for space programs, especially when all of this has to run on a battery!

Re:Why do they have to do this much coding?

[dodobh (65811)]

That close to the metal, with the limited memory space and CPU power? C is probably the highest level language which works.

!extreme programming

[religious freak (1005821)]

Aside from having a strong general dislike for anything 'extreme', this is wrong because this is obviously (at least from what I saw) NOT extreme programming, it's programming for harsh environments (no not Windows :).

Crappy title.

Buzzword Soup: Reliable Computing

[RobBebop (947356)]

I agree. It is not "extreme" to expect a device in space to have stand-by systems in place to withstand radiation. I believe the currently accepted buzzword for this is simply "reliable computing".

The idea is to anticipate failure events, and designing the system to survive them. It is cool... but certainly not extreme.

English language more complex than first thought..

[adamofgreyskull (640712)]

Words [reference.com] can have different [reference.com] meanings. Film at 11.

I hope they mean...

[by Anonymous Coward]

I hope they mean this [uncyclopedia.org] type of Extreme Programming.

Why didn't they automate Dumping The Scoop, etc?

[by Anonymous Coward]

Some of the things shouldn't require detailed programming each day,
because they could have been programmed into a firmware library!

The geometry between the arm, scoop, & the 8 ovens doesn't change,
so that could have been automated.
Instead of coding the whole damn thing
& hoping the signal can get there that day,
they could have told it
Invoke "dump scoop into oven # 1 routine"
and saved the interplanetary bandwidth+delays
for something that required customization...

Also, this business of EVERYTHING waits

Say What!?

[Sam36 (1065410)]

Why did they use vxworks instead of linux?

Re:Say What!?

[Xiaran (836924)]

There can be lots of reasons. I like Linux as much as the next guy and have been using is since back in the days when I installed it on my 386 from dozens of floppies(and nearly fried my CRT monitor several time configing X :) Those were the days :) ). But I used to work on a project that used QNX. I was often pestered by people saying "Why cant you do that in WIndows" and then "Why cant you do that in Linux". Frankly we didnt because

1. We had years of libraries that were QNX specific
2. There were real time requirements that Plain Linux was not up to(real time linux may be... but you are talking a major porting project)
3. We had literally man-decades of programming experience in our team.
4. We were using a database soultion that was not available on Linux(and is still not)

SO the upshot is... it is often a decision to either have something working now to do the job or wait another few man-years before a (probably buggier) version is done for Linux. Its not anti-Linux. Its just common sense.

Re:

[JamesP (688957)]

I believe the main issue with using linux is memory protection.

"Real Time OS" is (kind of) "marketing speak" for a OS that does pretty much nothing. Not because it is bad or poor, but because it was designed that way.

And really, up in Mars you don't need hundreds of processes, user interaction, etc, etc

What you need is a basic framework for working with (and that VxWorks does), and you ABSOLUTELY NEED the OS to STEP OUT OF THE WAY when you need it!

Because sometimes the only way to fix a problem is binary pa

lucky for them

[someone1234 (830754)]

>And they have a lot of interplay between different instruments, so you have to make sure the sequences are not just working, but working together.

I would have some interplay too, but Mass effect isn't out here, yet. Lucky Nasa engineers :)

Some other interesting points about that article

[Fallen Andy (795676)]

1. Only 8 ovens, which can't be used more than once. Hence all the painstaking deliberations

about when to really go for a scoop of soil.

2. Only 3 months before it will get too cold and the lander will (probably) die.

3. Martian day, (roughly 24hrs 40mins).

The NASA programmers have been my heroes ever since the hacks they did to Voyager.

I guess after they've finished the programmers will take up something more relaxing (like working for EA).

Andy

Re:

[Lumpy (12016)]

Exactly, I get a kick out of guys that second guess programmers that are so more advanced than they are that they can only HOPE to aspire to be as good as the guys at NASA.

That and From my experience, less than 10% of all programmers have ever done anything embedded or robotics wise. The rules change when you are writing software for engine management, robots, or space probes, than when you are writing a new consumer toy app or spreadsheet program.

... sure scares this old timer.

[Fallen Andy (795676)]

I started out my programming career porting the UCSD p-system. Bad enough being given a processor I'd never seen or programmed before , a box of bad photocopies and burning my way through 5 prototypes . Motorola 68K based with an "advanced" I/O processor (6809) which had bugs in the firmwire handshake with the MC68K. No parity on memory. Double sided board big enough to play golf. That was my first port. (It really helped that it used a blurry color Sinclair QL monitor at 110 column resolution - aarrgghh my

X-*

[conureman (748753)]

Think X-Games dude.

Re:

[FooAtWFU (699187)]

Extreme Programming is a subset of the "agile" software methodologies which features test-first programming, pair programming, customer conversations / "stories" rather than specifications, to "do the simplest thing possible that could work" (and fix it up later... assisted by comprehensive test suites), and a 40-hour work week.

And somehow I doubt NASA is doing all that.

Re:Um what

[K. S. Kyosuke (729550)]

"And somehow I doubt NASA is doing all that."

And you might be right. That is because this submission was processed with an editorial process called "Extreme Submission Editing", which is a subset of the "agile journalism" methodologies, driven by principles like "test-first publishing" (first publish it, then fix the bugs if there is a stronger "boo!" than usual), "pair editing" (pick any two editors, each one will believe that the other one is going to fix it) and "do the simplest editing possible that could work" (cross yourself, close the eyes and click the "OK, Publish the damned thing!" button). Thanks to the Extreme Submission Editing, Slashdot continues to bring you quality news at an unbeatable price!

Re:Um what

[roman_mir (125474)]

It's definitely more extreme than the so called 'Extreme Programming'. Extreme in a sense that you have to get things right or an extreme amount of time and money will go to extreme waste. That's Extreme. 'Extreme Programming' is mostly about covering your ass, not about getting the code right.

Re:

[K. S. Kyosuke (729550)]

It may appear in the headline, but in the text, I see just one place mentioning an "extreme programming challenge". Sadly, my ESL-programmed brain parses that as "extreme (programming challenge)", not as "(extreme programming) challenge", but what do I know - IANANES. :-)