Multi-threaded Programming Makes You Crazy?

gduranceau writes "Help! My program deadlocks! I got several concurrent threads that write the same variable! Everything goes well on my mono processor but becomes an incredible mess on that 16 CPU monster! And of course, as soon as I add traces, problems disappear... Don't panic! Calm down and take a deep breath. "

Use the right tool

And of course, as soon as I add traces, problems disappear... Don't panic! Calm down and take a deep breathe...

...and get yourself a technology [sun.com] designed for multi-threaded programming. Java will give each thread its own cache of variables to prevent deadlocking on concurrent modifications. If you need to do something that requires more than one statement (thus creating a race condition), then you need to create yourself a semaphore-based lock:

synchronized(objectToModify)
{
    if(objectToModify.getX() == myObj.getX()) objectToModify.setY(myObj.getY());
}

Of course, such synchronizations can carry a huge penalty on multi-CPU systems. i.e. If you manage to stop every CPU, you could be wasting MASSIVE amounts of CPU time. As a result, you should always strive to push locks down as far in the code as possible. They must execute extremely quickly, and should only be called when absolutely necessary. Follow those guidelines and you'll find it fairly easy to write multi-threaded code.

Oh wait. I was supposed to praise the NPTL tool, wasn't I. Um... well... it's very nice. And they've got... um... penguins on the homepage. Oh, and look! It's GPLed! Wow. Just... um... wow. Hey, did you know that the author of Minix wrote a book [amazon.com] on OS Design? Really. It even covers the basics of multi-threading. It's pretty cool, you should... um... check it out. Yeah, that's the ticket!

Java no panacea -- must know what you're doing

Java's "builtin thread saftey" is simply a poor hack. The idea is to give _every_ structure a mutex. Any access to the structure requires a mutex lock.

First off, that in itself will not prevent deadlock. Secondly, it's damned inefficient.

Look: there's just no way around it. If you want to do effective (i.e. low bug, high performance) multithreaded programming, you simply have to understand what you're doing. Ultimately, the tools of your trade will be mutexes, condition variables, semaphores, etc -- the O/S primitives. Don't rely on your programming language to "automatically" use these for you, blasting out mutexes machinegun-style. Instead, figure out the logic of your program. You probably need only a small number of mutexes.

A key to effective multithreaded programming is to adhere rigidly to certain programming practices. It must _NEVER_ be the case that 2 threads have write access to a given item at the same time. Duh. But you can use fancy programming tricks to, in effect, automatically add run-time assertions to your code which assure that this practice is being adhered to. In production mode, you remove these runtime assertions.

Another good practice is, if you really need to have multiple mutexes, to arrange them into a hierarchy. When a top-level mutex is locked, no other mutex can be locked. When a second-level mutex is locked, only top-level mutexes can be locked. Etc. This hierarchy can be verified at runtime, in debug mode. Adhering to this regime will go a long way to removing the possibility for deadlocks.

Bottom line: you really have to know what you're doing in order to write good multi-threaded code. You should take the time to really study that problem space. An excellent book I've found for this purpose is "Concurrent Programming in ML". (I know -- nobody uses ML. So what? Learn the language just for the purpose of understanding the book. Then, you can apply your knowledge to any domain you're working in).

Re:Java no panacea -- must know what you're doing

No, Java does not give every structure a mutex (as if Java even had structures). It only creates a mutex if synchronization is used on an object, so if you never use locking there are no locks. And no, it is actually perfectly fine for 2 or more threads to have read or write access to the same variable as long as it's atomic read/write... you just have to know what that means for your program.

Batman was right that after using Java's threading this NTPL trace looks pretty lame. Not only is the threading and locking in Java braindead simple, but the JVM actually tells you what is wrong. For instance it detects deadlocks and gives you the complete call trace of each deadlocked thread.

Other languages have good locking too (ruby for instance), so it's more that everything is difficult and crappy in C and its kind. I guess if you are stuck writing a threaded application in C in the first place this tracing library could be useful. Of course if you use the heap you're going to also want to replace malloc/free with a fast multithreaded version and then do a bunch of hacks so that it isn't ridiculously slow (locking on every free()... now *that's* inefficient).

Re:Java no panacea -- must know what you're doing

Yes what you say it true, but these are all implications of atomic read/write... It follows logically from having caches

Umm, no, it has nothing to do with atomicity, nor is it because of caching.

Atomicity just guarantees that the atomic change is seen

Re:Java no panacea -- must know what you're doing

It must _NEVER_ be the case that 2 threads have write access to a given item at the same time.

Two clarifications:

First, it's okay to allow multiple threads write access as long as the writes are guaranteed to be atomic and as long as the order of atomic writes doesn't matter. In practice, that second restriction usually means you need locking.

Second, it's often important that one thread not be writing an object while one or more threads are reading it. In other words, multiple writers aren't the only problem.

Re:Use the right tool

I agree, java has great tools for avoiding deadlocks and great tools for multithreaded programming. But i have still seen people who manage to create incredibly bad code on java in multithreaded environments and also people who don't take advantage of the

Re:Use the right tool

Use the right tool

The correct tool is called a brain, but first the brain must be configured properly.

Deadlocks are one symptom of poor program logic, and are designed into the program due to lack of proper controls. They frequently occur when a program is not designed before it is written.

See "dining philosophers [google.com]" for an explanation of this, and several methods to prevent this situation.

Tracing tools are all well and good, but if one starts out with correct logic in the first place then one won't spend more time debugging than programming.

Always remember that a digital computer is a logical computing device. If you give it a series of instructions which do not ALWAYS have a logical solution, then it will choke ... eventually.

-Adam

Re:Use the right tool

There are some good tools for the job. Relatively speaking, Java isn't one of them.

While Java does include some built-in support for multithreading primitives, its underlying model (using locks on data to prevent simultaneous access) is the same as many

Re:Use the right tool

I'm shocked, just shocked! All this time I thought programming was for the masses and it turns out that when you just copy stuff books and websites, it doesn't always work.

Really when are people going to get over this multithreading problem? Concurrency

Re:Use the right tool

If you're going to tout Java as a concurrent programming platform, the books you should link to should be about Java. Such as:

• Concurrent Programming in Java: Design Principles and Pattern (2nd Edition), by Doug Lea.
• Java Concurrenc

All good programmers...

Are a little mad anyway ;)

Re:All good programmers...

No proberen and no verhogen make software something something

Go crazy?

Don't mind if I do!!!

Multi-threaded Programmation Makes Me Crazy?

No, this title does. Is a Sentence? Is a Question? Why There a Space Before the Question Mark? What 'Programmation'?

Re:Multi-threaded Programmation Makes Me Crazy?

The poster is a French-speaking person. Programmation is the French word for "programming".
Notice also: take a deep *breathe*.

But I agree... multi-threaded programming can drive people crazy. Message passing-based programming is less prone to nastiness than shared-state concurrency. (Languages like Erlang come to mind).
http://en.wikipedia.org/wiki/Concurrent_programmin g_language [wikipedia.org]
http://www2.info.ucl.ac.be/people/PVR/bookcc.html [ucl.ac.be]

You can also do Erlang-style message passing in Python using Candygram
http://candygram.sourceforge.net/ [sourceforge.net]

Re:Multi-threaded Programmation Makes Me Crazy?

Title is description! Describes producte! Tells you what does! How it helpes you! Programmation is programming with exclamation!

Link to the home page would be nice

I was wondering what was this program was about. Fortunately, here is there website. http://nptltracetool.sourceforge.net/ [sourceforge.net]

Probably going to download it anyway...

Looking at the list of functions that it hooks into, I don't see pthread_rwlock*. Are the pthread_rwlock functions implemented using other pthread_* funcs? I haven't run into any problems yet with the project I'm working on, but it would be nice to run through this and make sure everything's working as expected.

Bah

Aha, so I can only do multithreaded programming on GNU/Linux with NPTL'ed glibc or what? Other programming langunguages than C/C++ don't exist or don't do threading. What about other operating systems? Specific solutions to general problems only apply to specific manifestations of the general problem and are therefore useless for most of us.

The only good general advice about learning how to develop software on distributed systems I can give is: Read some of Andrew S Tanenbaum's books about operating systems and distributed systems in particular. The books contain knowledge you'll be able to apply to almost every system you develop software for.

Why all the negativism in the posts?

Developing multithreaded is infact difficult, and any tool claiming to make it easier is worth looking at. If it works, these guys have done us all a favor. If it doesn't, at least they've made an attempt, and it may inspire others to do improve on it. Better tools are always welcome.

Multithreaded Haiku

There was a time when I was doing a lot more multithreaded programming (using the pthreads API, FYI). At the time, I was inspired to write some Haiku:

Threaded programming
is not for the faint of heart
or for the sober.

A small locked mutex
mistakenly left alone
results in deadlock.

valgrind used to be able to do this

http://valgrind.org/ [valgrind.org] used to include a tool called hellgrind for finding just such problems. unfortunately, hellgrind has gone away for a bit (it broke when the VM was re-done to support non-x86 platforms), but julian & co are working hard to get it working again Real Soon Now (tm). if you're using x86, you can use an old release of valgrind (2.2.0 i think) and you should be fine.

personally, i can't say enough good things about valgrind. there are a couple non-obvious issues (support for sse/sse2/sse3 is still in the works, so if you get an inexplicable SIGILL, this is probably the problem), but it's saved me hundreds of hours over the past year (and i'm sure it'll save me even more in the future).

that all said, my (admittedly limited) experience with threading is that it's best to design the deadlocks away before you even touch the editor. i wonder if there are any design tools which support deadlock / contention checking at the model or design level?

No that's just common sense.

that all said, my (admittedly limited) experience with threading is that it's best to design the deadlocks away before you even touch the editor.

That's not "limited" experience. That's common sense. Trying to find deadlocks, race conditions, and accident

Re:valgrind used to be able to do this

At my previous company we built a system with on the order of 10 threads working on a combined dataset consisting of many hundreds of thousands of objects and occupying a couple hundred meg of memory in a large installation. There could be hundreds of thou

Re:Here's your problem...

Realized that link wasn't as helpful as I remembered. But here are some other good general links that should get you going.

http://en.wikipedia.org/wiki/Concurrent_computing [wikipedia.org]
http://en.wikipedia.org/wiki/Message_passing [wikipedia.org]
http://en.wikipedia.org/wiki/Actor_model [wikipedia.org]

Re:Here's your problem...

Yes, there is a real reason. Sometimes it's inherent in the algorithm that the amount of data that must be shared is impractical to send using messages. Parallelization does not come for free; there are communication costs. If the communication costs are greater than the benefit you get from doing the computation in parallel, then you get no benefit from parallelization. Message passing will always have more overhead than shared memory multithreading. Hence, shared memory multithreading allows you to exploit finer grain parallelism than message passing.

Your point that message passing is generally a cleaner design choice is valid, but it's not always a practical option.