New Languages Vs. Old For Parallel Programming 321
joabj writes "Getting the most from multicore processors is becoming an increasingly difficult task for programmers. DARPA has commissioned a number of new programming languages, notably X10 and Chapel, written especially for developing programs that can be run across multiple processors, though others see them as too much of a departure to ever gain widespread usage among coders."
Parallel is here to stay but not for every app (Score:4, Insightful)
We need async frameworks too! (Score:5, Insightful)
Re:Parallel is here to stay but not for every app (Score:2, Insightful)
Parallel is not going to go anywhere but is only really valid for certain types if applications.
Exactly, some problems are inherently serial. These programs would run slower if you made them run in parallel.
Re:What's so hard? (Score:2, Insightful)
Awful example in the article (Score:4, Insightful)
The example in the article is atrocious.
Why would you want the withdrawal and balance check to run concurrently?
Re:Parallel programming is dead. No one uses it... (Score:4, Insightful)
Threading i don't count as parallel processing for the desktop. I don't even hear of any games or applications built for parallel.
Uhhhhhhhhhhh? Yes, well done with that...
Re:What's so hard? (Score:1, Insightful)
How many lines does it take you to parallelize this with pthreads in C?
for (i = 0; i < 1000; i++)
c[i] = a[i] + b[i];
If it takes you more than 2 lines, then your "language" is too hard to be used everywhere by everyone.
A view based on history... (Score:3, Insightful)
Rehash time...
Parallelism typically falls into two buckets: Data parallel and functional parallel. The first challenge for the general programming public is identifying what is what. The second challenge is synchronizing parallelism in as bug free way as possible while retaining the performance advantage of the parallelism.
Doing fine-grained parallelism - what the functional crowd is promising, is something that will take a *long* time to become mainstream (Other interesting examples are things like LLVM and K, but they tend to focus more on data parallel). Functional is too abstract for most people to deal with (yes, I understand it is easy for *you*).
Short term (i.e. ~5 years), the real benefit will be in threaded/parallel frameworks (my app logic can be serial, tasks that my app needs happen in the background).
Changing industry tool-chains to something entirely new takes many many years. What most likely will happen is transactional memory will make it into some level of hardware, enabling faster parallel constructs, a cool new language will pop up formalizing all of these features. Someone will tear that cool new language apart by removing the rigor and giving it C/C++ style syntax, then the industry will start using it
Re:Parallel is here to stay but not for every app (Score:3, Insightful)
For real time apps that do transactions Parallel is needed. What I was comparing them to is desktop apps where in many cases the benefit does not really exist. The main point I was trying to get across is that parallel programming is difficult and not needed for every application.
Re:Parallel programming is dead. No one uses it... (Score:3, Insightful)
If threading isn't parallelism, then what is? At what level of separation between separate streams of execution does an application become "parallel"?
Re:What's so hard? (Score:3, Insightful)
Making a threaded application in C isn't difficult. Testing and debugging said application is. Given that threads share memory, rigorously testing buffer overflow conditions becomes doubly important. In addition, adding threading introduces a whole new set of potential errors (such as race conditions, deadlocks, etc.) that need to be tested for.
Its easy enough to create a multi-threaded version of a program when its for personal use. However, there are a number of issues that arise whenever a threaded program interacts with the (potentially malicious) outside world, and these issues are not trivial to test for or fix. That's why I think that parallel programs are going to be increasingly written in functional programming languages (Common Lisp, Haskell, Scala, etc.). The limitations on side effects that functional languages impose reduces the amount of interaction between threads, and reduces the probability that a failure in a single thread will propagate through the entire application.
Re:What's so hard? (Score:1, Insightful)
That's what libraries are for. This is what you youngins continually forget. Design and implement properly libraries of useful code ONCE and use them many times.
You could easily write a "vector_add" function which spawns (or unlocks pre-spawned which is smarter) threads to perform a variety of tasks. Then from your application a single line of code would perform an optimized parallel vector addition or whatever.
In fact, a smart DSP lib today would do just that. Pre-spawn a bunch of threads then host a job server which unlocks threads to work on given tasks. That way you have single line functions like vector_add(in1, in2, out, size), etc...
So you could actualy write really easy to read parallel programs in C. You just have to know the first thing about software development.
In short, your rant is a product of not knowing what you are doing.
Re:Parallel is here to stay but not for every app (Score:5, Insightful)
Most computing intensive problems that a user will encounter at home are easily parallelizable, i.e. video encoding, gaming, photoshop filters, webbrowsing and so on. The amount of times where I maxed out a single CPU and the given problem would not have been to some large degree parallelizable are close to zero.
The trouble is that they are only "easy" parallelizable in concept, implementing parallelization on an exciting serial codebase is where it gets messy.
Re:Clojure (Score:3, Insightful)
Erlang is quite OK for non-distributed programming. Its model of threads exchanging messages is just a natural fit for it. As it is for multicore systems.
Re:What's so hard? (Score:3, Insightful)
AC here is right, but also missing something. I often hear about things being "easy". Often when people say something is "easy" they really mean, "it's easy after it's done." This is one of those things where it's only easy after it's done. The code might look easy after it's created and debugged. But getting to the point where it's created and validated and debugged is much harder in some languages and approaches (e.g. C) than it is in others (e.g. Erlang).
Take someone experienced with multithread, multi process, multi node programming in C. And put them up against someone experienced with same in a language designed for distributed systems. Have them drag race on producing code that expands to an arbitrary number of processes and computers and evenly distributes the load among them in a fault tolerant, smooth way. The person in a language designed for it is going to blow the doors off the person doing it in C in terms of productivity. And that's what these languages are all about.
If you've got a library already present that does exactly what you need, great, and AC is right on target there. But when you don't have such a library, and you almost never will, then it's great to use a tool that makes the job easy to do well and do quickly.
Re:Parallel is here to stay but not for every app (Score:5, Insightful)
And how many of those cores are above 2% utilization for 90% of the day? Parallelization on the desktop is a solution is search of a problem- we have in a single core dozens of times what the average user needs. My email, web browsing, word processor, etc aren't cpu limited. They're network limited, and after that they're user limited (a human can only read so many slashdot stories a minute). There's no point in anything other than servers having 4 or 8 cores. But if Intel doesn't fool people into thinking they need new computers their revenue will go down, so 16 core desktops next year it is.
Multi-threaded or Parallel? (Score:5, Insightful)
I have not read the article (par for the course here) but I think there is probably some confusion among the commenters regarding the difference between multi-threading programs and parallel algorithms. Database servers, asynchronous I/O, background tasks and web servers are all examples of multi-threaded applications, where each thread can run independently of every other thread with locks protecting access to shared objects. This is different from (and probably simpler than) parallel programs. Map-reduce is a great example of a parallel distributed algorithm, but it is only one parallel computing model: Multiple Instruction / Multiple Data (MIMD). Single Instruction / Multiple Data (SIMD) algorithms implemented on super-computers like Cray (more of a vector machine, but it's close enough to SIMD) and MasPar systems require different and far more complex algorithms. In addition, purpose-built supercomputers may have additional restrictions on their memory accesses, such as whether multiple CPUs can concurrently read or write from memory.
Of course, the Cray and Maspar systems are purpose-built machines, and, much like special-build processors have fallen in performance to general purpose CPUs, Cray and Maspar systems have fallen into disuse and virtual obscurity; therefore, one might argue that SIMD-type systems and their associated algorithms should be discounted. But, there is a large class of problems -- particularly sorting algorithms -- well suited to SIMD algorithms, so perhaps we shouldn't be so quick to dismiss them.
There is a book called An Introduction to Parallel Algorithms by Joseph JaJa (http://www.amazon.com/Introduction-Parallel-Algorithms-Joseph-JaJa/dp/0201548569 [amazon.com]) that shows some of the complexities of developing truly parallel algorithms.
(Disclaimer: I own a copy of that book but otherwise have no financial interests in it.)
Re:What's so hard? (Score:1, Insightful)
Simple answer: synchronized shared data access.
Technically creating the threads is the easy bit. Making sure that the threads don't fight over shared mutable data is so hard that most people give up. Frinstance, "serializable" isolation is the only really guaranteed isolation level for database transactions under all circumstances, but it can turn your DB into a uniprocessor; most people avoid it. Java's Swing UI gives up on data locking, and just says "only my thread touches the UI" and provides API to book tasks on it.
There exist a lots of cases where threaded parallelism is easy to implement. But many cases are hard, with subtle pathological difficulties. Once you bundle a few of these cases into the same system, correctness becomes impossible to estimate. Add to that the fact that many of the bugs are intermittent, debugger-resistant (schroedingbugs!) and potentially fatal (data corruption, when you encourage liveness; deadlocks, when you vote for safety), and you have some seriously difficult problems.
Re:Clojure (Score:3, Insightful)
Check out Clojure [clojure.org]. The only programming language around that really addresses the issue of programming in a multi-core environment.
That's a rather bold statement. You do realize that those neat features of Clojure like STM or actors weren't originally invented for it? In fact, you could do most (all?) of that in Haskell before Clojure even appeared.
On a side note, while STM sounds great in theory for care-free concurrent programming, the performance penalty that comes with it in existing implementations is hefty. It's definitely a prospective area, but it needs more research before the results are consistently usable in production.
Re:Chapel? (Score:4, Insightful)
No widely spoken human natural language was "invented," Modern English included. Where do people come up with these things? Modern English evolved out of Middle English, just as Spanish evolved out of Latin, etc. Modern English was not "invented" in any meaningful sense of "invernted."
reference for WikiWeenies. [wikipedia.org]
Re:How to Solve the Parallel Programming Crisis (Score:4, Insightful)
Sorry but, IMO, the cell is a perfect example of how not to design a multicore processor. Heterogenous processors introduce nothing new to the table of solutions that was not already there. We had systems with CPUs and GPUs before the Cell (or Intel's Larrabee and AMD's Fusion) showed up. Everybody knows that they're a pain in the ass to program. Neither CUDA nor OpenCL nor Microsoft's much ballyhooed TBB (threaded building blocks) will change that fact.
My point is that one does not design a parallel processor and then come up with a programming model to exploit it. It should be the other way around. The programming model should come first. One should design a model that makes parallel programming easy and the resulting apps rock-solid. Only then, after you have perfected your model, should you even consider designing a processor to support the model.
IOW, everybody's doing it wrong, and by everybody, I mean the all big players in the multicore hardware/software industry: Intel, Microsoft, IBM, Sun-Oracle, AMD, ARM, Apple, FreeScale, etc. The major computer science centers who are getting a lot of research money form the industry are not helping either since they have to kowtow to the likes of Intel and AMD whose main interest is to safeguard their installed base and preserve continuity.
It makes no difference. When the pain becomes unbearable (it's all about money), it will suddenly dawn on everybody that what is needed is to break away from the past.
Re:Parallel is here to stay but not for every app (Score:2, Insightful)
The 16 cores help because your operating system is doing hundreds of things all the time, not just waiting for you to type B into the currently active gui.
Re:Parallel is here to stay but not for every app (Score:3, Insightful)
While technically most servers are somewhat parallel in nature, it isn't really the same sort of thing that these sorts of languages are designed to achieve.
Servers, for the most part, are parallel because they have to be able to handle a lot of requests simultaneously, so they spin off a new thread or process(depending on the architecture) for each request to the server, do some relatively simple concurrency checking and then run each request, for the most part, in serial. They're parallel because the task being performed is parallel, a web server that could only return the page to one person at a time wouldn't work, not to make them faster or more efficient, or to take advantage of multiple processors. This kind of parallel architecture is relatively simple, because the architecture is defined by the requirements of the project and you just have to ensure that it works.
Taking something like a video encoder, PC game, compiler, etc and making it run in parallel so that it's faster and can take advantage of modern hardware is a totally different kettle of fish. You have to redesign your core idea so that it works in parallel, you have to turn one discrete task into two or more which can be run at the same time. It's a whole different challenge and one which very few programmers(myself included) seem to be prepared to meet.
Re:Parallel is here to stay but not for every app (Score:4, Insightful)
Web-browsing for instance, is not inherently paralizable.
Of course it is. Each Tab can be rendered in parallel, each JPEG can be decoded in parallel, each character can be antialiased in parallel, each download can happen in parallel, all the pixel that make up the final image can be alpha blended in parallel.
Yes, at some point you hit a wall where you will be left with a piece of code that you have to evaluate sequentially, you can't alpha blend half a pixel and your layouting engine might also require some sequential evalution. My point however is that none of those are CPU intensive enough to be a problem for normal use.
These days I don't have to wait for my computer because it has to solve some tiny non-parallelizable task, but because my computer is doing some heavy data crunching that could be done in parallel without much of a problem.
Or to put it another way: Half the for-loops out there just process each element in a list, the major reason why the don't do it in parallel is because the programming language makes it extremely hard to do so, not because there is some fundamental reason why it can't be done.
Re:How to Solve the Parallel Programming Crisis (Score:3, Insightful)
using threads, or even multiple processes, and nobody has found a good model that actually makes it easy to do parallel programming.
The reason is that threads and processes are the wrong tool for the task, they introduce a lot of additional complexity while still failing at giving you fine grained parallelization. They are patchworks that try to add the ability to do parallelization to languages that where build for sequential evaluation, instead of solving the problem from the ground up.
Functional languages look like a much better solution for parallel programming. Without side effects, a very large part of parallel programming problems disappears instantly. There will be a need for some relearning as functional languages require a different approach to tackle a programming problem, but in the end that is what it needs. Look at GPU programming, the reason why it is trivial to apply parallelization to it, is because the infrastructure forces you to write your program in a way that can be parallelized. You can't make your fragment shader go crazy and draw over other pixels and stuff, you have one pixel that you have to fill and no side-effects outside of that and as a result you can throw as much parallel processing power at it as you like.
Re:Parallel is here to stay but not for every app (Score:3, Insightful)
And how many of those cores are above 2% utilization for 90% of the day? Parallelization on the desktop is a solution is search of a problem-
The indication of the need to parallelization isn't the number of cores that are above 2% utilization for 90% of the day, but the number that are above 90% utilization for any part of the day.
Some of use our computers for more than that.
Re:Parallel is here to stay but not for every app (Score:3, Insightful)