Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
Programming

What Makes Parallel Programming Difficult? 196

Posted by Unknown Lamer
from the double-the-threads-double-the-fun dept.
An anonymous reader writes "Intel's Aater Suleman writes about why parallel programming is difficult. ... I was unaware ... that a major challenge in multi-threaded programming lies in optimizing parallel programs not just getting them to run. His analysis is insightful and the case study is very enlightening if you are unfamiliar with parallel code debugging. "
This discussion has been archived. No new comments can be posted.

What Makes Parallel Programming Difficult?

Comments Filter:
  • Easy! (Score:4, Funny)

    by Anonymous Coward on Friday May 27, 2011 @02:25PM (#36266270)

    Motherboards rarely have parallel ports these days!

    • Re:Easy! (Score:5, Funny)

      by sgbett (739519) <slashdot@remailer.org> on Friday May 27, 2011 @02:33PM (#36266352) Homepage

      is that things sometimes happen in the wrong order.

      • Re:Easy! (Score:5, Funny)

        by sgbett (739519) <slashdot@remailer.org> on Friday May 27, 2011 @02:36PM (#36266376) Homepage

        The problem with parallel programming

        • by mybecq (131456)

          Difficult Parallel Makes Programming What?

          (Prior message was optimized for concurrent throughput).

      • Yes it is happen when that terrible's.
        • by Acius (828840)

          - The nazi grammar parallel

          No combination above gramatically words possible of the is correct. You 's drop the should.

    • by oztiks (921504)

      Because its just as difficult as parallel parking?

    • by antdude (79039)

      Same with serial, PS/2 (mine only had one and using PS/2+USB adapters doesn't work well -- too many pauses and sometimes pause), PCI (not Express), etc. :(

  • "You're just not thinking fourth-dimensionally!"

    "Right, right, I have a real problem with that."

    • by dch24 (904899)
      McFly! But really, the first and second movies were way better.

      If you've read the article, you'll feel a little smarter today. Even if it was just a good review.

      If you haven't heard of Amdahl's Law [wikipedia.org], pay attention. (Since simple.wikipedia.org doesn't have a nice, short explanation, how about a car analogy?)

      Amdahl's Law
      You're street racing. But there's a school zone in the middle of the course, and you always slow down for the school.

      No matter how fast you go on the rest of the course, that school z
      • by ildon (413912)

        Your understanding of Amdahl's Law isn't quite correct. I guess if you want to use the street race analogy, it's more like you each have your own lane for part of the race, but for another part of the race you have to go single file into a single lane, and if you're parallel to someone as you approach the choke point you have to arbitrarily let them ahead of you or pass them based on some set of rules that may not necessarily be fair. And the goal isn't to win the race but you're all delivering something an

        • by dch24 (904899)
          Amdahl's law is just about speedups. It doesn't imply parallel operations at all. I know the wikipedia page says it's about parallel processes. It's not.
          • by ildon (413912)

            I didn't look at the wikipedia page. This is from memory from the parallel programming class I took. Either way the main point is that your analogy was poor.

      • Not quite, Amdahl's Law says that no matter how well you optimize your parallel tasks, the one that you cannot parallelize will gobble up your time. I don't think there's a good car analogy for that.

        Ask your project manager. He's dealing with that problem on a daily base.

        • by dch24 (904899)
          I am my project manager, you insensitive clod. :-)

          People are like cars. The faster you drive them, the more hot air you get out their rear end.
        • "I don't think there's a good car analogy for that."

          Sure there is.

          You have yourself a passel of old Dodge Chargers (painted orange for some weird reason), on a REALLY wide road, all tied together through the windows with some good ol' Southern one-inch hemp.

          Whichever car has the wrong mix of gasoline to moonshine will drag the whole pack down. To carry the analogy further: if it has real problems it could crash the whole lot.

          :o) There is ALWAYS a car analogy.

    • by jd (1658)

      Well of course you'd have a problem if using reals. For four dimensions, you need to typecast to quaternions.

  • There is a class of problems, the P complete problems, which are (probably) inherently hard to multithread in any advantageous way, and this class includes some pretty important real-world problems:

    http://en.wikipedia.org/wiki/P-complete [wikipedia.org]
    • I'll reply to you.

      You're a bit smarter than me but I think you're saying there are lots of easy tasks that are easy to run on a single linear code thread but are hard to split and recombine with any less loss of time / resources.

      • by blair1q (305137)

        he's saying there are lots of hard problems that are easy to parallelize, and also lots of hard problems that are hard to parallelize

        there are probably some easy problems that are hard to parallelize but since they're easy nobody even thinks to bother

        parallelization is itself an optimization, so when you start talking about optimizing after you've parallelized you've just gone back to the original problem

        maybe what's needed is an architecture that is structured to handle P-complete problems, rather than sin

  • I am kinda curious how anyone even tangentially involved in programming could not be aware that the problem with writing parallel programming was doing it for a gain in efficiency. Making a thread or process is generally just a couple lines of code, synchronization with data separation, mutex's and avoiding deadlocks and race conditions has been solved since almost the beginning of parallelism.

    • Re:unaware? WTF? (Score:5, Insightful)

      by Mongoose Disciple (722373) on Friday May 27, 2011 @02:39PM (#36266416)

      synchronization with data separation, mutex's and avoiding deadlocks and race conditions has been solved since almost the beginning of parallelism

      And yet people constantly get these details wrong in practice.

      It's an extra layer of complexity and it introduces extra chances to make mistakes, even around areas where a programmer could know better. There's not much way around that. If people only made coding mistakes around difficult problems software would be dramatically more bulletproof than it actually is.

      • by Hooya (518216)

        I tend to think of it as an extra dimension in code. With non-parallel code, the code you have (it's sequence) is the same as what it's sequence would be when run. With parallel code, the run-time sequence is different than the code as it's laid out in source.

        I see people have trouble with just async stuff (eg. AJAX) and have a hard time wrapping their mind around the fact that even though the callback function is in-sequence with the rest of the code, it's not actually called in that sequence - hence the '

        • by dgatwood (11270)

          I see people have trouble with just async stuff (e.g. AJAX) and have a hard time wrapping their mind around the fact that even though the callback function is in-sequence with the rest of the code, it's not actually called in that sequence - hence the 'callback'.

          In my experience, the reason people have so much trouble with async stuff is that every single JavaScript API I've seen that does things asynchronously is rather fundamentally designed *wrong*. They don't provide any clean mechanism for passing dat

      • by Nadaka (224565)

        I blame that mostly on the languages not being explicit about what operations and methods are or are not thread safe. And for capable programmers those errors generally only occur when you try to make things more efficient by avoiding excess mutex and data duplication in the pursuit of efficiency.

      • by blair1q (305137)

        And yet people constantly get these details wrong in practice.

        If they can't get those right innately, then they never even started out to do parallel programming, they just stuck a fork in their execution flow.

      • by CastrTroy (595695)
        I took a course in parallel programming. We worked with MPI in C if anyone is curious or interested. The hardest part was completely changing the way you thought about programming. It was like the first time you looked at prolog. Or the first time you tried a functional language like scheme. If you've only ever done procedural, it can be quite a big deal to switch. Also, it's quite a bit harder to debug multithreaded code, as the exact order of the instructions is different everytime you run it. Tracki
      • by steelfood (895457)

        The key to parallel programming is compartmentalization. With a good enough foundation that compartmentalizes properly, parallel programming would only be a matter of coding the compartments and synchronizing their results.

        That having been said, parallel programming these days solves a fairly niche problem. The speed of modern processors make faking parallelism with interrupts viable. The only area where parallelism truly makes sense is when working with extremely large amounts of large chunks of data.

        For e

    • by IICV (652597)

      I am kinda curious how anyone even tangentially involved in programming could not be aware that the problem with writing parallel programming was doing it for a gain in efficiency. Making a thread or process is generally just a couple lines of code, synchronization with data separation, mutex's and avoiding deadlocks and race conditions has been solved since almost the beginning of parallelism.

      Actually, interestingly enough, parallelizing code not only results in computational efficiency gains, but it frequ

  • by kvvbassboy (2010962) on Friday May 27, 2011 @02:36PM (#36266378)
    learn2efficientparallelalgorithms.

    Good tutorial for someone who wants to jump into some parallel programming, but it's mostly Operating Systems 101 (or 601).

    Honestly though, if you have not optimized your algorithm or code to for parallelism and you want to do it now, you might probably be better off writing the whole thing from scratch, and the tutorial explains why very nicely/.

    • Assuming that there is a good way to take advantage of multiple processors for the particular problem your code solves. For a course project once, we found ourselves trying to solve a linear program, and someone suggested using the research cluster to speed things up. As it turns out, linear programming problems are P-complete, and there is no known way to make meaningful use of multiple cores (it is very likely that no such method even exists, and that the problem is inherently sequential).
      • I'll take a stab and reply to you that "8 processors was enough for anyone", in the sense that multiplexing 8 programs is just insane. Better to just run 8 prorams each on their own core, and use some progs that can use 4 cores at a time. That leaves 4 free.

        (Overly simpistic) I agree, but 1028 cores is not the answer. We need the next generation in raw core power to move computing forward. 8 killer cores will beat 1024 mediocre cores.

        • by maraist (68387) * <michael...marais ... l...n0spam...com> on Friday May 27, 2011 @06:29PM (#36268774) Homepage
          While you're correct from a temporarily practical measure, I disagree in theory. OS theory 20 or more years ago was about one very simple concept.. Keeping all resources utilized. Instead of buying 20 cheap, slow full systems (at a meager $6k each), you can buy 1 $50k machine and time-share it. All your disk-IO will be maximized, all your CPUs will be maximized, network etc. Any given person is running slower, but you're saving money overall.

          If I have a single 8 core machine but it's attached to a netapp disk-array of 100 platters over a network, then the latency means that the round trip of a single-threaded program is almost guaranteed to leave platters idle. If, instead I split a problem up into multiple threads / processes (or use async-IO concepts), then each thread can schedule IO and immediately react to IO-completion, thereby turning around and requesting the next random disk block. While async-IO removes the advantage of multiple CPUs, it's MASSIVELY error-prone programming compared to blocking parallel threads/processes.

          A given configuration will have it's own practical maximum and over-saturation point. And for most disk/network sub-systems, 8 cores TODAY is sufficient. But with appropriate NUMA supported motherboards and cache coherence isolation, it's possible that a thousand-thread application-suite could leverage more than 8 cores efficiently. But I've regularly over-committed 8 core machine farms with 3 to 5 thousand threads and never had responsiveness issues (each thread group (client application) were predominantly IO bound). Here, higher numbers of CPUs allows fewer CPU transfers during rare periods of competing hot CPU sections. If I have 6 hot threads on 4 cores, the CPU context switches leach a measureable amount of user-time. But by going hyper-threading (e.g. doubling the number of context registers), we can reduce the overhead slightly.

          Now for HPC, where you have a single problem you're trying to solve quickly/cheaply - I'll admit it's hard to scale up. Cache contention KILLS performance - bringing critical region execution to near DRAM speeds. And unless you have MOESI, even non-contentious shared memory regions run at BUS speeds. You really need copy-on-write and message passing. Of course, not every problem is efficient with copy-on-write algorithms (i.e. sorting), so YMMV. But this, too was an advocation for over-committing.. Meaning while YOUR problem doesn't divide. You can take the hardware farm and run two separate problems on it. It'll run somewhat slower, but you get nearly double your money's worth in the hardware - lowering costs, and thus reducing the barrier to entry to TRY and solve hard problems with compute farms.
          amazon EC anyone?
  • In my opinion, many problems with software development, are just as applicable in other domains of our life, and parallel programming is definitely one of them. We equally well have problems managing large teams of people working in parallel. These are problems of logistics, management and also (and no I am not joking) - cooking. And we're as bad handling these as we now handle software development. It may be right, however, to start solving this with the computers - no need to throw away rotten food and/or

    • There has been quite a bit of work on formalizing parallel computing. NP problems are exactly that: problems that can be solved efficiently on a computer that can explore an unbounded number of solution paths in parallel. There is also the NC hierarchy, which can be thought of as problems that can be solved efficiently on a sequential computer and "much more efficiently" on a parallel computer (that is, polynomial time on a sequential computer, and polylogarithmic time on a parallel computer with a polyn
  • by Anonymous Coward

    That you have to do everything all at once. How would you tell 50 kids to sort 50 toy cars? How would you tell 50 footballers to line up by height all at once? How would you have 50 editors edit the 50 pages of a screenplay all at once so that it makes sense from a continuity perspective? All these problems are very easy, but become very hard when you have to do it all at once...

    • The ultimate solution to most of those appears to be, "have to do it fifty times, and assign one job to a person".

      That said, not all problems are so easily dealt with as bulk operations. Perhaps you have some real need to take one item and spit the work up among multiple people. The real driving force for all the scenarios I can think of is a desire or need for an upper bound on job latency.

      I do this kind of work as my day job. I've also got some experience in managing groups of coders (at work, not particu

    • by jbolden (176878)

      How would you tell 50 footballers to line up by height all at once?

      Easy break into n subgroups have them line up by height and then perform pairwise merges.

  • Lame (Score:3, Interesting)

    by oldhack (1037484) on Friday May 27, 2011 @02:46PM (#36266494)
    I bet there are hundreds of simple tutorials on concurrent processing on the web, and these two pieces bring nothing new/interesting.
  • I think the language Chapel [cray.com] being developed by Cray is taking concrete steps to make Parallel Programming easier. It is very similar in syntax to C++ and even supports some higher level constructs.

    This looks like an advertisement for Chapel but I have no relation to Cray. Having taken a graduate parallel programming course, I cannot agree more with the statement that "Parallel Programming is difficult". I struggled a lot with pthreads and MPI before doing the final assignment in Chapel which was a plea
  • 20 years ago when I was working with transputers we use Occam. It was a very pure parallel programming language and it wasn't too difficult. However, writing parallel code meant starting from scratch (getting rid of the dusty decks of old algorithms as my professor described it). However, this never really happened and we've ended up with primitive parallelisation nailed on to sequential code. The are many parallel architectures, SIMD, MIMD, distributed memory, shared memory and combinations of them all

    • by jd (1658)

      Occam still exists. KROC supports the latest version of Occam (Occam-Pi), which supports mobile processes, generics and other concepts acquired from the developments in serial programming and computer clusters.

      I consider it to be one of the finest languages out there for learning parallel programming and consider that most of the modern failures in the field are a result of people not knowing it and therefore not knowing the fundamentals. You can't run if you can't walk.

      The transputer was the ultimate in co

  • Efficiency is hard (Score:5, Informative)

    by tlhIngan (30335) <[ten.frow] [ta] [todhsals]> on Friday May 27, 2011 @03:30PM (#36266940)

    Using locks and the like make it very easy to do multithreaded and parallel programs.

    The big problem comes when you need multiple locks because you find your program is waiting more on locks than anything else which is gumming up the whole works, and that can easily lead to deadlocks and other fun stuff.

    Another way is to consider lockless algorithms, which don't have such blocking mechanisms. However, then you get into issues where atomicity isn't quite so atomic thanks to memory queues and re-ordering done in the modern CPU, and thus have to start adding memory barriers before doing your atomic exchanges.

    Raymond Chen (of Microsoft) did a nice write up of the lockfree ways to do things and what Windows provides to accomplish them.

    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/05/10149783.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/06/10150261.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/06/10150262.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/07/10150728.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/08/10151159.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/08/10151258.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/12/10152296.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/13/10152929.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/14/10153633.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/15/10154245.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/19/10155452.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/20/10156014.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/21/10156539.aspx [msdn.com]
    http://blogs.msdn.com/b/oldnewthing/archive/2011/04/22/10156894.aspx [msdn.com]

    • by Prune (557140)
      Worse, typical synchronization primitives such as in pthreads and Windows are optimized not for speed but for error handling etc. It is easy to beat their speed with custom implementations, sometimes with dramatic speed increases (see for example numerous articles at http://locklessinc.com/articles/ [locklessinc.com] [locklessinc.com] and I've implemented some of the mutexes and ticket locks and my Windows/Linux software has become faster as a result). In addition, using lock-free algorithms whenever possible can provide a f
    • by ShakaUVM (157947)

      >>Using locks and the like make it very easy to do multithreaded and parallel programs.

      Eh, the Dining Philosophers would like to ask you out to eat. Locks can introduce timing issues which can result in a program locking up at random.

      The real difficulty of parallel programming comes from two things (speaking as someone who has a Master's in the subject):
      1) The development environment isn't as well supported as single-threaded coding.
      2) It requires a different mindset to write code solidly. Remember ho

      • by Tacvek (948259)

        It also does not help that threads are very easy to mess up. Look at all the traditional programs that used multiple threads for nonparallel concurrency, and how much trouble the developers have with deadlocks, or forgetting to use locks on a shared variable access, or calling code intended only to be used by the other thread, etc.

        So even if you have the ideal parallel version of the algorithm all planned out, actually implementing it correctly can still be problematic.

  • that most of today's popular programming languages do not accommodate higher-level forms of expression required for easy parallelism. Declarative languages have a slight edge at being able to express where sequential dependencies are.
  • One more reason why functional programming matters. Many programs become trivial to parallelize when you avoid mutation and side-effects outside of limited, carefully-controlled contexts.

    It's truly a joy when you can parallelize your code by changing a single character (from "map" to "pmap"). There's usually a little more to it than that, but you almost never have to deal with explicit locks or synchronization.

    • by jbolden (176878)

      Agreed. Learning to isolate side effects is one of the best things a programmer will get from FP.

    • by emt377 (610337)

      One more reason why functional programming matters. Many programs become trivial to parallelize when you avoid mutation and side-effects outside of limited, carefully-controlled contexts.

      More specifically, the problem of parallel programming is the problem of structuring state for concurrent access. Everything else (the mechanics of locks etc) is trivial and mainly a typing exercise.

      The reason this is difficult is that it's an optimization; normally when we optimize code we write the naive version first (or near-naive), then optimize it. We might change a hash table to a vector, memcpy() to a few lines of inline assembler for a particular target, hand code CRC/checksumming/RC4 in assemble

  • What makes parallel programming hard is poor languages. Languages that allow state changes and don't keep them isolated. Isolate changes of state, all changes of state and be careful about what kinds of combinators to use. Google map-reduce works whenever

    a) You can organize your data into an array
    b) You can do computations on array element in isolation
    c) You can do computations on the entire array via. associate operations pairwise

    And most programs do meet those properties but they slip in all sorts of

    • by russotto (537200)

      There's a reason Mapreduce is said to operate on "embarrassingly parallel" problems. There are a lot of them. But there are also a lot of problems which are not embarrassingly parallel; for instance, they have nonlocal data dependencies.

      • by jbolden (176878)

        Nonlocal data dependencies are not a problem. Map reduce is fantastic for distributing to clients. Each client performs a mini map reduce and then the results are collected.

  • by ljw1004 (764174) on Friday May 27, 2011 @04:10PM (#36267536)

    The dirty secret of parallel programming is that it's *NOT* so widely needed. I think a lot of academics got funding to study automatic parallelization or other parallel techniques, and they latch on to multicore as a justification for it, but it's not.

    There is only one GOOD reasons to use multithreading -- because your work is compute-bound. This typically happens on large-data applications like audio/video processing (for which you just call out to libraries that someone else has written), or else on your own large-data problems that have embarrassingly trivial parallelization: e.g.

    var results = from c in Customers.AsParallel() where c.OrderStatus="unfilfilled" select new {Name=c.Name, Cost=c.Cost};

    Here, using ParallelLINQ, it's as simple as just sticking in "AsParallel()". The commonest sort of large-data problems don't have any complicated scheduling.

    There are also BAD reasons why people have used multithreading, particularly to deal with long-latency operations like network requests. But this is a BAD reason, and you shouldn't use multithreading for it. There are better alternatives, as shown by the Async feature in F#/VB/C# which I worked on, which was also copied into Javascript with Google's traceur compiler). e.g.

    Task task1 = (new WebClient()).DownloadStringTaskAsync("http://a.com");
    Task task2 = (new WebClient()).DownloadStringTaskASync("http://b.com");
    Task winner = await Task.WhenAny(task1,task2);
    string result = await winner;

    Here it kicks off two tasks in parallel. But they are cooperatively multitasked on the same main thread at the "await" points. Therefore there is *NO* issue about race conditions; *NO* need to use semaphores/mutexes/condition-variables. The potential for unwanted interleaving is dramatically reduced.

    So in the end, who are the people who still need to develop multithreaded algorithms? There are very few. I think they're just the people who write high-performance multithreaded libraries.

    • by blair1q (305137)

      The dirty secret of parallel programming is that it's *NOT* so widely needed.

      That's kind of begging the question, there.

      Those who need it know they need it. Those who think it's neato-keen and want to play with it try to come up with ways to use that that are maybe not obvious, and for which it is maybe not even necessary. I've known about this at least since the first Thinking Machines came out and our school got one of the bigger ones, solved the problem they used in the grant proposal that paid for it in about a week, then realized they had a multi-million-dollar computer with

      • by maraist (68387) *
        The problem is that people tend to focus on single-threaded designs for 3rd party libraries.. Then when those libraries get linked to larger libraries (or main apps) which are MT, then the whole world comes crashing down. Now you have to treat every function call of the ST-library as a critical region.

        Thus while YOU may not care about MT, you should strive to make all your code reentrant at the very least. To whatever degree this allows contention-free memory access (e.g. ZERO global variables). This fut
    • by Xyrus (755017)

      The dirty secret of parallel programming is that it's *NOT* so widely needed. I think a lot of academics got funding to study automatic parallelization or other parallel techniques, and they latch on to multicore as a justification for it, but it's not.

      It is widely needed, but perhaps not by you.

      There is only one GOOD reasons to use multithreading -- because your work is compute-bound.

      Ok, now you're confusing the issue. Are you talking about parallel programming or multi-threaded programming? Parallel programming is larger in scope the simple multi-threaded programming.

      So in the end, who are the people who still need to develop multithreaded algorithms? There are very few. I think they're just the people who write high-performance multithreaded libraries.

      No, there are quite a few and many of them make quite a lot of money to do so. Do you think the programmers at ILM, 3DS, Pixar, and NASA are just sitting around doing nothing? Does your MT algorithm library also know how to optimize for the GPU as well? Are you sure that a one si

    • by ihavnoid (749312)

      There is only one GOOD reasons to use multithreading -- because your work is compute-bound.

      ...and therefore, most of the people in the world won't need anything beyond an Intel Atom because their tasks aren't compute-bound.

      Seriously, I don't know what kind of code you write for a living, but the code I write is almost always has some portion of compute-bound submodules, even if what I do has nothing to do with video codecs or 3d or whatever field that there are convenient libraries.

    • Cooperative multitasking is like going back to Windows 3.1. That operating system totally relied on the trick you mentioned, and as we all know, it sucked.

      Why did it suck? Obviously, for many reasons but most of all because once such a "fake thread" of execution starts taking a long time, it blocks all other threads.

      Why do you think a modern webserver uses many different threads?

      Sigh... I guess you need to gain some more experience in the parallel programming world.

  • The basic problem with parallel programming is that, in most widely used languages, all data is by default shared by all threads. C, C++, and Python all work that way. The usual bug is race conditions.

    There have been many languages for parallel programming which don't have default sharing, but they've never taken over outside some narrow niches. Partly because most of them weren't that useful outside their niche.

    The other classic problem is that in most shared-data languages with locks, the language d

  • The Scala [scala-lang.org] community has tried to move the problem into a more practical realm by adding things like parallel collections [infoq.com], DSL's [scala-lang.org] to abstract out the problem for specific applications and the Akka Project [akka.io] for simpler concurrency.

    Most of the parallel programming discussion I've seen is very complicated and not likely to appeal to those who have to do practical day-to-day business projects. By pushing the abstractions up a level, I think the Scala folks have made parallel programming more accessible for the av

  • Parallelism is very easy, provided that you don't do it yourself.

    Use a pure functional programming language like Haskell that can be automatically parallelized.

    Or use a programming language that uses the Active Object Pattern (or the Actor model).

    Or do as I do: use C++ to implement message passing, then have one thread per object. Objects don't have public members, they simply communicate by exchanging messages.

    In all the above cases, the trick is to avoid state. In Haskell, state is avoided by design; in t

  • For certain types of problems, the Linda coordination primitives and shared tuple-space make parallel programming much easier. I used the original C-Linda many, many years ago, and IBM's TSpaces for Java more recently. If you're trying to do little bitty actions on lots of data with tight coordination, the overhead is pretty bad. Looking into PyLinda is on my list of things to do...

I have not yet begun to byte!

Working...