Fault Tolerant Shell 234
Paul Howe writes "Roaming around my school's computer science webserver I ran across what struck me as a great (and very prescient) idea: a fault tolerant scripting language. This makes a lot of sense when programming in environments that are almost fundamentally unstable i.e. distributed systems etc. I'm not sure how active this project is, but its clear that this is an idea whose time has come. Fault Tolerant Shell."
You're dealing with the problem too high up (Score:5, Interesting)
The real benefit is in having a system which is sufficiently distributed that any program running on top of it can continue to do so despite any sort of underlying failure.
Wouldn't be much work in Tcl (Score:5, Interesting)
It doesn't actually seem to grok the commands that are being run, so something like
proc try {times script} {
if { [catch [uplevel $script] err] } { cleanup ; retry }
}
is all that's needed (of course to do it right you'd need a bit more, but still...).
try {5 times} {
commands...
}
Although Tcl is a bit lower level, and would require you to do exec ls, you could of course wrap that too so that all commands in the $script block would just be 'exec'ed by default.
In any case, better to use a flexible tool that can be tweaked to do what you need then write highly specialized tools.
Sounds like good way to do some serious damage (Score:3, Interesting)
Nice, which brings me too.... (Score:5, Interesting)
Re:Bad Idea (Score:2, Interesting)
Re:Bad Idea (Score:3, Interesting)
According to the HTML 4.01 spec [w3.org] </td> and </tr> tags are optional. So you can code a standards compliant page without them, as long as you declare your doctype properly.
this can essentially already be done in /bin/bash (Score:5, Interesting)
what they seem to essentially want is
- a try statement and error catching and
- a fortran like syntax for testing and arithmetic
I think the authors were a bit misguided. Instead of creating a whole new shell how about just extending a good existing shell with a new try statement a described.it can even be done without extending the shell:
as for the new syntax of .eq. .ne. .lt. .gt. .to.
certainly looks like fortran-hugging to me , yuck
as for integer arithmetic, that can be done with by either using backticks or the $[ ] expansion
Once again, Babbage was thinking ahead... (Score:5, Interesting)
Babbage, if you put into the machine wrong figures, will the right answers
come out?' I am not able rightly to apprehend the kind of confusion of ideas
that could provoke such a question."
-- Charles Babbage
I don't see why eveyone is complaining... (Score:4, Interesting)
Well.. besides pipes of course
A distributed shell ... (Score:2, Interesting)
I think the shell was called dsh. I believe this is the project site: http://dsh.sourceforge.net/
Are the aims of this fault tolerant shell and dsh the same? I'm not a programmer, but I'm trying to teach myself *nix system administration.
Eventually I'm hoping to cluster some older x86 systems I'm going to get at auction together for a Beowulf cluster. It sounds to me like one if not both of these two shells might come in handy!
OK, wise guys... (Score:5, Interesting)
ftsh has great utility in the realm it's written for. Obviously, it's not a basis for installing kernels or doing password authentication. In a Grid (not just distributed) environment, things break for all sorts of reasons all the time. You're dealing with a Friendly Admin on another system, one who may well be unaffiliated with your institution, project or field of study. He doesn't have any particular reason to consult with you about system changes.
Now you find yourself writing a grid diagnostic or submitter or job manager. One does not need strongly typed compiled languages for this. Shell scripts are almost always more efficient to write, and the speed difference is unimportant. Right now, most Grid submitters are being written in bash or Python or some such. Bash sucks for exception handling of the sort we're talking about. Python does better with its try: statements, but there's room for improvement. ftsh is a good choice for a sublayer to these scripts. One writes some of the machinery that actually interacts with the Grid nodes and supervisors in this easy, clear and flexible form.
Now there are a lot of specific points to answer:
One needs a Windows port to be able to make the Grid software we write in Linux available to the poor drones who are stuck with Win boxes.
This is not a code spellchecker or coding environment. At all.
This is not a crutch for inadequate programmers. This is a collection of methods to deal with a specific set of recalcitrant problems.
As I was pointing out before, this is, after all, an unstable system. One is using diverse resources on diverse platforms in many countries at many institutions. I appreciate the comment made by unixbob about operating in heterogeneous environments.
This isn't a substitute for wget. One uses wget as an example because it's clear.
The "pull" model breaks down immediately when there is no unified environment, as is described on infrastructures.org. When you're not the admin, and your software has to be wiped out the minute your job is done, "push" is the only way to do it. This is the case with most Grid computing right now (that I know about)
:) Use the right tool for the job and save time.
:)
All the woe and doom about the sloppy coding and letting the environment correct your deficiencies is... ill-thought-out. That's what a compiler is, folks. Should we all be coding in machine language?
I do agree, however, that one should indeed hone one's craft. Sloppy coding in projects of importance is inexcusable (M$). There is no reason to stick to strict exception handling, however, in the applications being discussed by ftsh's developers (the same folks who brought you Condor). When code becomes 3/4 exception handling, even when the specific exceptions don't matter, there's a problem, IMHO.
"set -e" will go a long way to helping you (Score:5, Interesting)
That shell script can be improved a lot by using " set -e " to exit on failure, as follows:
#!/bin/sh
set -e # exit on failure
cd
rm -rf bar
cp -r
This means that, if any command in the script fails, the script will exit immediately, instead of carrying on blindly.
The script's exit status will be non-zero, indicating failure. If it was called by another script, and that had "set -e", then that too will exit immediately. This is a little bit like exceptions in some other languages.
Re:Bad Idea (Score:5, Interesting)
The same logic could be applied to any security system, from the automatic door lock on the front of your house to Airbags in your car. Spell checkers discourage people from learning to spell. Antibiotics prevent the growth of the immune system. Why have a lock on your trigger, if it will encourage you to leave it in a place where your kids can find it.
The fact of the matter is, if the code works, it's good code. This is a shell scripting language we're talking about here... Not exactly assembly. Programmers would be better off spending more time thinking about the higher structure of their applications and less time hunting down trivial mistakes.
Of course, I know that this isn't quite what the article is talking about, but it's the principle of the thing. Augmentation would be an improvement.
Re:Worst idea since spell checkers (Score:2, Interesting)
Also, this isn't about covering up mistakes. I am sure good script programmer will _allways_ assume a command can fail. Using the example of the "cd" command in the article, should I really just assume it worked before removing files? Of course not. How ftsh helps is that the necessary error checking code is made more readable and brief. I still have to trap errors whether I use ftsh or bash, the difference is ftsh is easier to understand.
Simply making code less convoluted and more readable is not the same as sloppy programming.
Re:Why is there a Windows compatible port? (Score:2, Interesting)
some people, (I myself too) use bash as a daily basic for Windows, this new stuff can be interesting and maybe usefull for the unsafe windows enviroment.
Re:Let's draw a line in the sand... (Score:3, Interesting)
ACID Filesystems (Score:3, Interesting)
Remounting a filesystem with ACID on, a process sets a rollback point , executing a series of commands with the operating system keeping a record of the changes to the filesystem made by the process and its children. The process would inform the OS to either commit or rollback the changes.
This still raises questions on how to deal with with two or more competing "transactional" processes which rely on read information which another process chooses to rollback to an early state.
Re:It's got the concept backwards (Score:3, Interesting)
The thing is, if you run a distributed system with a thousand servers, and your patch distribution points drop of the network, you don't suddenly want a thousand servers hammering the networks endlessly. You want things like exponential backoff, timeouts after which the system will change behaviour (stop requesting updates until explicitly requested to start again, start triggering alarms etc.), and that is exactly the kind of scenario ftsh makes easy to do in scripts without having to write all the logic yourself for every bloody script.
What's with all the negativity? (Score:5, Interesting)
I think perl is where it is because so many people use it as "super script." To me that says, a) we recode all the Bourne and csh and bash in perl or b) we look at why people do shell scripting in perl or other languages and add that to the shell. I couldn't tell you which is right. It's a neat idea though and I'm glad they made it.
A real example I can think of, I had a test machine that had some kind of ext3 corruption and so it mounted up in read-only mode when it booted. I spent time diagnosing an application error in our application because nothing caught that; these are redhat type startup scripts. I noticed that our app couldn't write logs and began to debug the system. More interestingly, a dozen or so start-up scripts failed to start up critical components and their failure wasn't noticed. If you can't write to the filesystem, you can't create a socket(AF_UNIX) and all sort's of things go tits up then. If that's how you debug it's only going to get more difficult as you add more and more complexity, you have to detect the lower level failures and report them. Perversely, this wouldn't have been noticed had a different partition been read-only. Turns out that a drive was going bad. Had it been a different partition, it would have been noticed at catastrophic system failure time when the drive died.
I've done a fair amount of embedded work and there is always a test for new guys, you can tell the new guy (new college grad, whatever) because he skips half or more of the error checking in his code. You know printf returns a value? Funnier still, if you develop something like a consumer app in embedded space, you'll eventually see things like printf fail. We know it never should, but with 20,000+ users in different environments and what not, things like that can and do fail and usually point to a greater problem, like a dead drive or something. Instead of logging/alerting something to the critical and unusual printf failure, the app fails in a different way because this printf failed. Heaven forbid that it was sprintf that failed and then you shove bad data in to a database or configuration file and not just fail the system but corrupt the data too. Inspite of all of that, even veterans will forget error checking at times, it's a common bug and so having higher level tools to help assist, like exception in the shell can only be a good thing.
Re:Wouldn't be much work in Tcl (Score:3, Interesting)
Your example only does a fraction of what ftsh does.
Why there are no pipes (Score:4, Interesting)
Funny you should mention this, because I was going to write something about pipes. Getting pipes right with good error semantics is hard. For all the "just use set +e in bash" weenies out there, try running
Where's your error?If you think about the unix process and pipe primitives, you will see the difficulty. To create a pipeline, you normally fork, create the pipe, fork again, and run the two ends of the pipeline in the two sub-processes. This is scalable to deeply nested pipelines, but has a cost: Only one of the sub-processes is a child of the shell, so only one exit status can be monitored. To work around this, you really need to build a mini-OS environment on top of unix.
This demonstrates that unix was fundamentally not designed with concern for error semantics (consider Erlang as a diametric example). And this, I'm sure, is why ftsh doesn't have pipes (yet).
Re:Bad Idea (Score:3, Interesting)
Same problem with english. Most people have a high degree of fault tolerance when parsing natural language, which means any old crap will still just about be readable.
(sorry - I had to after reading that statement heh)
Re:It's got the concept backwards (Score:4, Interesting)
Check once every 1,2,4,8,16,32,64,whatever,mins *all the time anyway* whether it fails or succeeds and you *absolutely don't* want to have to explicitly tell 1000 machines to start again.
You simply generalise the update process, get rid of the special cases. In the case of patches, you know you're going to have to distribute them out to clients at some point anyway so have all the clients check once a day, every day. If the distribution server is down for a couple of days it's pretty much irrelevant.
My error detection code is trivial the network traffic is negligible unless the job's actually being done and I still haven't been given a good case for ftsh. I have a good case for a better randomising algorithm within a shell and a decent distributed cron (which is simple BTW), but not for a specifically fault tolerant shell.
You've got to stop thinking of these things as individual systems. The network is the machine.
dwim? (Do what I mean) (Score:2, Interesting)
Re:Missing the point (Score:2, Interesting)
http://c2.com/cgi/wiki?FailFast
It's not about ignoring errors, it's about the central idea that you'll never, _ever_, be able to write 100% perfect code, and if you could your code will be so full of error checking that it's both unreadable and, as a result, unmaintainable, masking logic bugs and similar. It's a better economy to come up with better ways to deal with failure than trying to prevent it altogether. And the final solution will be more stable.
This is an important realisation: Failure is inevitable, how you deal with it is what matters.