More Interest In Parallel Programming Outside the US?

simoniker writes "In a new weblog post on Dobbs Code Talk, Intel's James Reinders discusses the growth of concurrency in programming, suggesting that '...programming for multi-core is catching the imagination of programmers more in Japan, China, Russia, and India than in Europe and the United States.' He also comments: 'We see a significantly HIGHER interest in jumping on a parallelism from programmers with under 15 years experience, verses programmers with more than 15 years.' Any anecdotal evidence for or against from this community?"

Duh?

Old programmers don't want to learn new things -- trust the tried and true.

Young bucks want to be on the cutting edge to get the jobs that the old people already have.

----

Oh, and the people see the benefit in the other countries more than those in the U.S.? Probably not, we're just lazy American's though.

Re:Duh?

Spoken like a completely ignorant child. How the hell do you think (if you even can) we older guys got into this business? We were tinkering with new things, using them when appropriate, before many of you were born, and the joy of new ideas hasn't worn off. The only difference is we don't do it quite so impulsively, just because it seems new.

For one thing, multiple homogeneous cores is NOT new (hetero- either, for that matter), just fitting them into the same die. I've used quad 68040 systems, where, due to the ability of the CPUs to exchange data between their copy-back caches, some frequently-used data items were NEVER written to memory, and on System V you could couple processing as tightly or loosely as you wanted. There are some problem sets that take advantage of in-"job" multi-processing better than others, just as some problem sets will take of advantage of multiple cores by doing completely different tasks simultaneously. Simple (very) example: copying all of the files between volumes (not a block-for-block clone); if I have two cores, I can can either have a multi-threaded equivalent of "cp" which walks the directory tree of the source and dispatches the create/copy jobs in parallel, each core dropping into the kernel as needed, or I can start a "cpio -o" on one core and pipe it to a "cpio -i" on the other, with a decent block size on the pipe. More cores means more dispatch threads in the first case, and background horsepower handling the low-level disk I/O in the other. In my experience, the "cpio" case works better than the multi-threaded "cp" (due, AFAICT, to the locks on the destination directories).

Oversimplified

That's an oversimplified view.

It's more like when you've got enough experience, you already know what can go wrong, and why doing something might be... well, not necessarily a bad idea, but cost more and be less efficient anyway. You start having some clue, for example, what happens when your 1000 thread program has to access a shared piece of data.

E.g., let's say we write a massively multi-threaded shooter game. Each player is a separate thread, and we'll throw in a few extra threads for other stuff. What happens when I shoot at you? If your thread was updating your coordinates just as mine was calculating if I hit, very funny effects can happen. If the rendering is a separate thread too, and reads such mangled coordinates, you'll have enemies blinking into strange places on your screen. If the physics or collision detection does the same, that-a-way lies falling under the map and even more annoying stuff.

Debugging it gets even funnier, since some race conditions can happen once a year on one computer configuration, but every 5 minutes on some hapless user's. Most will not even happen while you're single-stepping through the program.

Now I'm not saying either of that is unsolvable. Just that when you have a given time and budget for that project, it's quite easy to see how the cool, hip and bleeding-edge solution would overrun that.

By comparison, well, I can't speak for all young 'uns, but I can say that _I_ was a lot more irresponsible as the stereotypical precocious kid. I did dumb things just because I didn't know any better, and/or wasted time reinventing the wheel with another framework just because it was fun. All this on the background of thinking that I'm such a genius that obviously _my_ version of the wheel will be better than that built by a company with 20 years of experience in the field. And that if I don't feel like using some best practice, 'cause it's boring, then I know better than those boring old farts, and they're probably doing it just to be paid for more hours.

Of course, that didn't stop my programs from crashing or doing other funny things, but no need to get hung up on that, right?

And I see the same in a lot of hotshots nowadays. They do dumb stuff just because it's more fun to play with new stuff, than just do their job. I can't be too mad at them, because I used to do the same. But make no mistake, it _is_ a form of computer gaming, not being t3h 1337 uber-h4xx0r.

At any rate, rest assured that some of us old guys still know how to spawn a thread, because that's what it boils down to. I even get into disputes with some of my colleagues because they think I use threads too often. And there are plenty of frameworks which do that for you, so you don't have to get your own hands dirty. E.g., everyone who's ever wrote a web application, guess what? It's a parallel application, only it's the server which spawns your threads.

Re:Duh?

the latest thing

1960: E. V. Yevreinov at the Institute of Mathematics in Novosibirsk (IMN) begins work on tightly-coupled, coarse-grain parallel architectures with programmable interconnects. ( c.f. [vt.edu] )

An extra core or so on a desktop is nice, beyond that they really won't be anywhere near the speedup its hyped.

And of course any virtual reality scenario will not profit from extra power.

CC.

Re:Duh?

Ugh.

Yes you can parallelize a VR system quite well. You can simulate a couple dozen NPCs per core, then synchronize on the collisions between the few that actually collide. You still get a nice speedup. It ain't 100% linear, but it can be pretty good. The frame-by-frame accuracy requirements are often low enough that you can fuzz a little on consistency for performance (that's usually done already. ever heard "If it looks right, it's right?").

Parallel programming is how we get more speed out of modern architectures. We're being told that we're not going to see Moore's law expand in GHz like it used to, but in multiple cores. Nobody things it's a panacea, except maybe the 13yr old xbox kiddies, but they never counted.

As for making impossible into possible, sure it will. There are lots of things you couldn't do with your machine 10-15 yrs ago, you can do now. Many systems have performance inflection points. As we get faster (either with a faster clock or a larger number of cores), we're going to cross more of them. I remember when you couldn't decode an mp3 in real time on a desktop machine. With the I/O and space costs of uncompressed music, that meant that you didn't really listen to music from your computer. Impossible -> Possible.

Threads: Threat or Menace

It always surprizes me how many people say "we have to multithread our code, because computer are getting more cores," not realizing:

1. There are often other ways to do it, e.g. multiple processes communicating over sockets, or multiple processes that share memory.
2. Threads are hard to get right. Really, really hard. [faqs.org]

When your library of mutexes, semaphores, etc. doesn't have exactly the construct you need, and you go to write your own on top of them, it's really, really hard not to introduce serious bugs that only show up very rarely. As one random example, consider the Linux kernel team's attempts to write a mutex, as descried in Ulrich Drepper's paper "Futexes are Tricky." [redhat.com]

If these people take years to get it right, what makes you think *you* can get it right in a reasonable time?

The irony is that threads are only practical (from a correctness/debugging point of view) when there isn't much interaction between the threads.

By the way, I got that link from Drepper's excellent "What Every Programmer Should Know about Memory." [redhat.com] It also talks about how threading can slow things down.

Re:Duh?

I work in parallel programming too.

Most problems do not parallelize to large scales.

I'm getting tired of this nonsense being propagated. Almost all real world problems parallelize just fine, and to a scale sufficient to solve the problem with linear speedup. It's only when people look at a narrow class of toy problems and artificial restrictions that parallelism "doesn't apply". e.g. Look at google; it searches the entire web in milliseconds using a large array of boxes. Even machine instructions are being processed in parallel these days (out of order execution etc.).

Name a single real world problem that doesn't parallelize. I've asked this question on slashdot on several occasions and I've never received a positive reply. Real world problems like search, FEA, neural nets, compilation, database queries and weather simulation all parallelize well. Problems like orbital mechanics don't parallelize as easily but then they don't need parallelism to achieve bounded answers in faster than real time.

Note: I'm not talking about some problems being intrinsically hard (NP complete etc.), many programmers seem to conflate "problem is hard" with "problem cannot be parallelized". Some mediocre programmers also seem to regard parallel programming as voodoo and are oblivious to the fact that they are typically programming a box with dozens of processors in it (keyboard, disk, graphics, printer, monitor etc.). Some mediocre programmers also claim that because a serial programming language cannot be automatically parallelized that means parallelism is hard. Until we can program in a natural language that just means they're not using a parallel programming language appropriate for their target.

---

Advertising pays for nothing. Who do you think pays marketer's salaries? You do via higher cost products.

Re:Duh?

What do you mean by "parallelize well"? Normally people would use that phrase to imply fine-grained parallelism in the problem but I suspect that you are using it differently. For example when you say that compilation parallelizes well are you talking about coarse-grain parallelism across compilation-units? This is not really a single instance of a problem being computed in parallel, but rather many independent instances of a problem. If you are willing to use many independent instances of a problem then the coarse-grain parallelism will always scale "well" - i.e linearly in the number of cores. But this does not provide a speedup in many real-world applications.

In your compilation example, it is easy to get speedups at the compilation level using something like make -jN. But this assumes that each unit is independent. If you want to apply advanced global optimisations then this is not always the case, and then you hit the harder problem of parallelizing the compilation algorithms rather than just running multiple instances. It's not impossible but I'm not aware of any commercial compilers that do this.

real world problem

> Name a single real world problem that doesn't parallelize.

Childbirth. Regardless of how many women you assign to the task, it still takes nine months.

(feel free to reply with snark, but that's a quote from Fred Brooks, so if your snarky reply makes it look like you haven't heard the quote before you will seem foolish)

Re:real world problem

True, but it scales well, you can have multiple child instances in the same nine months by throwing more women at the problem.

Re:real world problem

There is, however, the problem of process upkeep. One child process is a resource-hog. Multiple child processes seem to consume resources exponentially rather than linearly. How do you propose to optimize bandwidth to the mother process(es), supply sufficient inputs to the child process(es), or redirect the child process(es)' regularly scheduled core dumps? As a note, mother and child processes don't respond well to preemptive multitasking.

Re:Duh?

Most people who do anything are mediocre. Otherwise, mediocre would be redefined. It's like saying, half the people in the class scored below average. The fact that half the people scored below some value determines what the value of average is.

Questions

Q1) Why did the multithreaded chicken cross the road?
A1) to To other side. get the

Q2) Why did the multithreaded chicken cross the road?
A4) other to side. To the get

It is funnier in the original Russian.

Re:Questions

Whoa..

Q: How many multithreaded person(s) does it take to change a light bulb?
A: 5, 1 at each of the 4 ladders and 1 to pass the light bulb to the lucky one.

Q: How many multithreaded person(s) does it take to change a light bulb?
A: 4, each trying to screw the lightbulb.

Q: How many multithreaded person(s) does it take to change a light bulb?
A: I don't know what happened to them.

Experince

One reason could be that software engineers with more experience simply already know about these things, and have faced off against the many problems with concurrency. Threads can be hell to deal with for instance. So because of things they don't show any interest.

That being said I think that if you want to actually make use of many cores you really do have to switch to a language that can give you usage of many threads for free. Writing it manually usually ends up with complications. I find Erlang to be pretty nifty when it comes to these things for instance.

More than a trend, it's a necessity

Parallel programming is going to go mainstream, not because people find it interesting, but because that's the way hardware is forcing us to go. First mainframes, then workstations, and now desktops and laptops have moved away from single CPU cores. In every case, it has been a necessary evil due to the inability to pack enough power into a single monolithic processor. Does anyone actually think Intel, if they could, wouldn't happily go back to building single-core CPU's with the same total power as the multi-core CPU's they're making now?

Right now, parallel development techniques, education, and tools are all lagging behind the hardware reality. Relatively few applications currently make even remotely efficient use of multiple cores, and that includes embarrassingly parallel code that would require only minor code changes to run in parallel and no changes to the base algorithm at all.

However, if you look around, the tools are materializing. Parallel programming skills will be in hot demand shortly. It's just a matter of time before the multi-core install base is large enough that software companies can't ignore it.

Reinders Is Wrong: Threads Are Not the Answer

One day soon, the computer industry will realize that, 150 years after Charles Babbage came up with his idea of a general purpose sequential computer, it is time to move on and change to a new computing model. The industry will be dragged kicking and screaming into the 21st century. For over 20 years, researchers in parallel and high-performance computing have tried to come up with an easy way to use threads for parallel programming. They have failed and they have failed miserably. Amazingly, they are still continuing to pursue the multithreading approach. None other than Dan Reed, Director of scalable and multicore computing at Microsoft Research, believes that multithreading over time will become part of the skill set of every professional software developer (source: cio.com [cio.com]). What is wrong with this picture? Threads are a major disaster: They are coarse-grained, they are a pain in the ass to write and hard to debug and maintain. Reinders knows this. He's pushing threads, not because he wants your code to run faster but because Intel's multicore CPUs are useless for non-threaded apps.

Reinders is not an evangelist for nothing. He's more concerned about future-proofing Intel's processors than anything else. You listen to him at your own risk because the industry's current multicore strategy will fail and it will fail miserably.

Threads were never meant to be the basis of a parallel computing model but as a mechanism to execute sequential code concurrently. To find out why multithreading is not part of the future of parallel programming, read Nightmare on Core Street [blogspot.com]. There is better way to achieve fine-grain, deterministic parallelism without threads.

Re:Reinders Is Wrong: Threads Are Not the Answer

Fine-grained parallelism works fine. It works in your NVidia, SIMD-based graphics coprocessor, does it not? Locking and syncing is a problem only in a non-deterministic environment like multithreading. Fine-grained parallelism is temporally deterministic because the temporal (concurrent or sequential) order of code execution can be precisely determined.

It really really depends on the problem and the algorithm. Some things are easy to parallelize, especially if they don't need (much) shared writable memory, but others are furiously hard.

you'll probably have to wire the registers between cores directly into each other in order to avoid the enormous overhead of an external chip.

Not really. There is a way to design a multicore processor such that only neighboring cores cooperate on related computation. It is part of the self-balancing mechanism. I can't go into detail but suffice it to say that if you keep your inter-core communication performance penalty at a fixed level regardless of the number of cores, you have a winner.

As I said above, it really depends on what you're doing. Some classes of problems just don't and can't have nice communication patterns, and if you've got one where you've got these inherent non-local effects, no amount of cleverness is going to let you avoid the hard fact that communication costs will dominate them. Other problems are much more tractable though; it's definitely not all doom and gloom. Just don't sound off and claim that it's all solved (nope!) or that some simple hardware-level cleverness will save us (nope again!) but instead study what's really known so that you can sound more knowledgeable. A good place to start reading up is with the thirteen dwarfs paper [gigascale.org] (PDF).

Old dogs and new tricks

Whether it's a good idea or not, you will have a VERY hard time convincing an old dog programmer that he should jump on the train. Think back to the time when relational models entered the database world.

Ok, few here are old enough to be able to. But take object oriented programming. I'm fairly sure a few will remember the pre-OO days. "What is that good for?" was the most neutral question you might have heard. "Bunch o' bloated bollocks for kids that can't code cleanly" is maybe more like the average comment from an old programmer.

Well, just like in those two cases, what I see is times when it's useful and times when you're better off with the old ways. If you NEED all the processing power a machine can offer you, in time critical applications that need as much horsepower as you can muster (i.e. games), you will pretty much have to parallelize as much as you can. Although until we have rock solid compilers that can make use of multiple cores without causing unwanted side effects (and we're far from that), you might want to stay with serial computing for tasks that needn't be finished DAMN RIGHT NOW, but have to be DAMN RIGHT, no matter what.

Parallel programming has been with us of years!

One of the reasons more seasoned programmers are not particularly interested is that in most cases someone else has already doen the hard work.

Want to serve multiple user on multiple cpus with your web pages? Then write a single threaded program and let Apache handle the parallelism. Same goes for JavaEE, database triggers, etc. etc. going all the way back to good old CICS and COBOL.

It is very rare that you actually need to do parallel programing yourself. Either you are doing some TP monitor like program which schedules tasks other people have written in which case you should use use C and POSIX threads (anything else will land you in deep sewage) or you are doing serious science stuff in which case there are several "standard" fortran libraries to do multithreaded matrix math -- but if the workload is "serious" you should be looking at clustering anyway.
         

Not So Great

Been there, done that. Good from far, but far from good.

As an engineer straight out of college, I was very interested in parallel programming. In fact, we were doing a project on parallel databases. My take is that it sounds very appealing, but once you dig deeper, you realise that there are too many gotchas.

Considering the typical work-time problem, let's say a piece of work takes n seconds to complete by 1 processor. If there are m processors, the work gets completed in n/m seconds. Unless the parallel system can somehow do better than this, it is usually not worth the effort. If the work is perfectly divisible between m processors, then why have a parallel system? Why not a distributed system (like beowulf, etc.)?

If it is not perfectly distributable, the code can get really complicated. Although it might be very interesting to solve mathematically, it is not worth the effort, if the benefit is only 'm'. This is because, as per Moore's law [wikipedia.org], the speed of the processor will catch up in k*log m years. So, in k*log m years, you will be left with an unmaintainable piece of code which will be running as fast as a serial program running on more modern hardware.

If the parallel system increases the speed factor better than 'm', such as by k^m, the solution is viable. However, there aren't many problems that have such a dramtic improvement.

What may be interesting are algorithms that take serial algorithms and parallelise them. However, most thread scheduling implementations already do this (old shell scripts can also utilise parallel processing using these techniques). Today's emphasis is on writing simple code that will require less maintenance, than on linear performance increase.

The only other economic benefit I can think of is economy of volumes. If you can get 4GHz of processing power for 1.5 times the cost of a 2GHz processor, it might be worth thinking about it.

WTF question is this????

Those of us doing server side development for any medium to large company will have already been doing multi-threaded and/or multi-process applications for ages now:
- When Intel was still a barely known brand, other companies were already selling heavy-iron machines with multiple CPUs for use in heavy server-side environments (didn't ran Windows though). Multi-cores are just a variant of the multiple-CPU concept.

The spread of Web applications just made highly multi-threaded server-side apps even more widespread - they tend naturally to have multiple concurrent users (<rant>though some web-app developers seem to be sadly unaware of the innate multi-threadness of their applications ... until "strange" problems start randomly to pop-up</rant>).

As for thick client apps, for anything but the simplest programs one always needs at least 2 threads, one for GUI painting and another one for processing (either that or some fancy juggling a la Windows 3.1)

So, does this article means that Japan, China, India and Russia had no multi-CPU machines until now ... or is this just part of a PR campaign to sell an architecture which, in desktops, is fast growing beyond it's usefulness (a bit like razors with 4 and 5 blades)

Re:What are the applications?

Multiple cores plus less experienced programmers results multiple infinite loops able to run at the same time. I don't quite see how this helps quality software, regardless of the synchronization problem.

Re:What are the applications?

It depends on the application. Some applications simply benefit from running in realtime. And some applications don't really scale well by breaking them up into individual processes. Some applications want to use as much CPU as you can throw at them -- a web app, for instance, had better be able to handle another few (dozen?) app servers if you get Slashdotted.

Also: Management of threads is mostly hard because we're mostly still using such low-level tools to do it. Semaphores, locks, and threads are the equivalent of GOTOs, labels, and pointers. While you can build a higher-level system (message-passing, map/reduce, coroutines) out of semaphores, locks, and threads, I'd argue that's like writing everything in GOTOs instead of doing a proper for loop -- or using a for loop instead of a proper Ruby-style "each" iterator. At the risk of abusing my analogy, yes, sometimes you do want to get under the hood (pointers, threads), but in general, it's much safer, saner, and not that much less efficient to use modern parallel programming concepts.

There's still some real problems with pretty much any threaded system, but I suspect that a lot of the bad rap threads get is from insufficient threading tools.

Oh, and about that webserver -- using more than one process is pretty much like using more than one thread, and I'd say weighs about the same in this discussion. Webservers, so far, are able to get away with using one thread (or process, doesn't matter) per request (and queuing up requests if there's too many), so that's a bit less of a problem than, say, raytracing, compiling, encoding/decoding video, etc.

Re:You can still make large apps without concurren

When your only tool is a hammer...

... everything looks like a skull.