Scaling Large Projects With Erlang 200
Delchanat points out a blog entry which notes,
"The two biggest computing-providers of today, Amazon and Google, are building their concurrent offerings on top of really concurrent programming languages and systems. Not only because they want to, but because they need to. If you want to build computing into a utility, you need large real-time systems running as efficiently as possible. You need your technology to be able to scale in a similar way as other, comparable utilities or large real-time systems are scaling — utilities like telephony and electricity. Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires. Amazon SimpleDB is built upon Erlang. IMDB (owned by Amazon) is switching from Perl to Erlang. Google Gears is using Erlang-style concurrency, and the list goes on."
Erlang: The Movie ! (Score:5, Funny)
Sufficiently? (Score:5, Interesting)
hard to read after (Score:2)
"you need large real-time systems running as sufficiently as possible."
Should that not be efficiently as possible?
Re:hard to read after (Score:5, Funny)
"you need large real-time systems running as sufficiently as possible."
Should that not be efficiently as possible?
You obviously haven't looked very closely at any of the "market leader" software lately.
Software from the Big Guys is more and more designed to sell (think forced upgrades) bigger, faster systems. You don't do this by making your software efficient.
The logic behind many software updates these days is "Will this release require sufficient resources that customers will be persuaded to upgrade to new hardware?"
Re: (Score:2, Funny)
You obviously don't read summaries, articles, or headlines. The logic behind your post is "rant about something for no reason at all."
Re: (Score:2)
Dude, if you threw out all those posts, each story would have what, maybe 10 comments per story at most? And yes, I'm guilty of the above sin, quite often actually
Re: (Score:2)
Re: (Score:2)
I work for one of the "Big Guys".
We spend a lot of time looking at efficiency and trying to do more with what we've got. This is because we're interested in selling software for everyone's hardware and not just our own. You can't get everyone to switch to exclusively our stuff, but we can still get into their operations by being the best.
Re: (Score:2)
Maybe efficiency is not the main goal.
We are talking about utility-like quality of service.
That is not necessarily efficient.
I am sure the power grid would be much more efficient if it didn't have to account for peak usage. But they do, because their core value is not efficiency, but high availability.
I don't know if the guy meant something like that, but "efficiency" is not the right word here, either.
Huh? (Score:5, Insightful)
"The two biggest computing providers of today"?
What the hell does that mean?
Also, is it just me or does the article intro sound like it was written by someone who has taken way too many marketing classes?
Re:Huh? (Score:5, Interesting)
Re: (Score:2, Insightful)
Also, is it just me or does the article intro sound like it was written by someone who has taken way too many marketing classes?
Too many marketing classes and not enough English classes.
Re:Huh? (Score:4, Insightful)
For instance, by dropping the imdb name, it is now my impression that this Erlang thing is best at destroying otherwise useful sites by making them less reliable and more annoying to users. Who in their right mind would want to do that. Oh, marketing people, thats who!
Re: (Score:2)
In the article the guy says he is a consultant. This explains everything
Proprietary? (Score:2, Insightful)
Who wrote the summary? GWB? (Score:5, Funny)
"running as sufficiently as possible"?
Sometimes as a nation we must ask ourselves, is our children learning?
Scala (Score:5, Informative)
People may also want to check out Scala at:
http://www.scala-lang.org/ [scala-lang.org]
It also uses the Erlang style concurrency approach and runs on the JVM with class compatibility with other JVM languages, ie Java, Groovy, etc.
Re:Scala (Score:4, Informative)
There is a significant difference between Scala and Erlang.
Erlang uses green threads. And green threads have advantages and disadvantages over native threads.
For instance Erlang is bad at IO but on the other hand it can spawn millions of threads, something that the JVM has a hard time doing because native threads are limited by the kernel.
Re:Scala (Score:4, Informative)
Scala has actors, which are allow you to do something _like_ green threads: http://lamp.epfl.ch/~phaller/doc/ActorsTutorial.html [lamp.epfl.ch]
Re:Scala (Score:4, Informative)
Modern JVMs on the modern Linux Kernel can spawn quite a hellacious amount of threads these days, actually.
The problem with Java is the shared-state synchronization that is often necessary, and the extra work required to distribute state to threads across different VMs. A functional language and programming style could work quite well on top of the JVM, though, and could leverage RMI and some kind of message port facility for the distribution.
Re:Scala (Score:4, Interesting)
Re:Scala (Score:5, Informative)
"Last time you checked" was some time last century in that case. Linux kernels have been able to support at least 100,000 threads [wikipedia.org] for ages.
That doesn't mean that using shared memory concurrency is a good idea though. When your computer comes with 10s or 100s of cores you'll realise that maybe SMP wasn't the best model of concurrency to choose. That's where models such as map-reduce, Erlang's shared nothing concurrency, message passing, and MPI come into their own. Even today they are useful because you'll be able to scale your program across multiple machines.
Rich.
Re:Scala (Score:4, Informative)
I quite agree that shared memory concurrency is a bad idea, however. Unfortunately, until you have message passing instructions in the hardware, you're stuck emulating message passing on top of shared memory, which leads to cache coherency issues and a host of other problems.
Re:Scala (Score:5, Informative)
Linux threads stopped using the LDT on x86 in 2002. This change went mainstream over subsequent years, and is nowadays always used on x86.
There was once a limit on the number of processes, too, due to each process having an entry in the GDT. That has long been removed too.
Mozart/Oz (Score:2, Informative)
http://www.mozart-oz.org/ [mozart-oz.org]
I'll just cite another "competitor":
"The Mozart Programming System is an advanced development platform for intelligent, distributed applications. The system is the result of a decade of research in programming language design and implementation, constraint-based inference, distributed computing, and human-computer interfaces. As a result, Mozart is unequalled in expressive power and functionality. Mozart has an interactive incremental development environment and a production-quality
Why Erlang Matters (Score:5, Insightful)
1. Multicore ready.
Erlang will use them. Write your application in Erlang and it's done for you.
2. Scales well.
As an example, http://yaws.hyber.org/ [hyber.org] scales very nicely when loads increase. Your basic LAMP/LYMP setup runs much better on vanilla hardware.
3. Designed for telecom
The architects designed the language to run in a telecom environment so things like upgrades can be done while the application is running.
Yaws in particular needs your help. Failover clustering inside the yaws server would be wonderful. Right now, it uses CGI to process other languages. It does it flawlessly, but a more direct solution might be a nice project.
Why Erlang doesn't matter (Score:5, Interesting)
1. Invariable variables.
This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.
2. Weird syntax.
Why, exactly, are there three different kinds of (required) line endings? It seems as though the syntax is designed to be as different from C as possible, while maintaining at least as many quirks. Moreso, even -- when constructing normal, trivial programs, you're going to hit most language features head-on and at their worst. Where's my 'print "hello\n"' that works most other places?
I don't believe the important features of Erlang are mutually-exclusive with the sane syntax of, say, Ruby or Python.
3. Not Unicode-ready.
Strings are defined as ASCII -- maybe latin1. But there's no direct unicode support in the language -- if you're lucky, there are functions you can pipe it through.
There are other things I haven't mentioned, mostly implementation-specific -- things like the fact that function-reloading cannot be done when you natively-compile (with hipe) for extra speed. My plan is to take the features I actually like from Erlang and implement them elsewhere, in a language I can actually stomach for its real tasks.
Re: (Score:3, Interesting)
Though I agree with you on 2 and 3, I'm not so sure about 1, but I might be wrong on that. As I understand it, you should look at variables in functional programming languages like Erlang more like those in a mathematical formula; such programs can be proven correct a lot easier, and since variables are effectively immutable, it facilitates forking the line of execution in a way that would not be possible without all kinds of semaphores and other concurrency stuff than if variables where not immutable. You
Re:Why Erlang doesn't matter (Score:5, Interesting)
As I understand it, you should look at variables in functional programming languages like Erlang more like those in a mathematical formula; such programs can be proven correct a lot easier, and since variables are effectively immutable
All of this is based on the premise that Erlang is a functional language. It's not purely-functional, and I just don't see the point of doing it half-assedly. Erlang is effectively an imperative language dressed up like a functional language.
And they're not immutable -- they can be unbound. As I understand it, this unboundedness is detected at runtime, not compiletime. If it was detected at compiletime, you'd have a valid point.
it facilitates forking the line of execution in a way that would not be possible without all kinds of semaphores and other concurrency stuff
Except that's not how Erlang does concurrency. It does concurrency with explicit "processes" (green threads) and message-passing.
Now, it does make these very easy, and you can get it to distribute processes among a few real OS threads (one per core) -- so it's still very cool. But you're thinking of languages like Haskell, which can be automagically threaded. Erlang is manually threaded, it's just much easier to think in threads (or "processes") -- they're effectively a language feature.
Re: (Score:2)
As I understand it, you should look at variables in functional programming languages like Erlang more like those in a mathematical formula; such programs can be proven correct a lot easier, and since variables are effectively immutable
All of this is based on the premise that Erlang is a functional language. It's not purely-functional, and I just don't see the point of doing it half-assedly. Erlang is effectively an imperative language dressed up like a functional language.
OH come on this is bullshit. Even lisp and scheme aren't "purely functional" the fact that the actor model breaks the functional model clearly makes almost no difference in practice.
And they're not immutable -- they can be unbound. As I understand it, this unboundedness is detected at runtime, not compiletime. If it was detected at compiletime, you'd have a valid point.
it facilitates forking the line of execution in a way that would not be possible without all kinds of semaphores and other concurrency stuff
Except that's not how Erlang does concurrency. It does concurrency with explicit "processes" (green threads) and message-passing.
Now, it does make these very easy, and you can get it to distribute processes among a few real OS threads (one per core) -- so it's still very cool. But you're thinking of languages like Haskell, which can be automagically threaded. Erlang is manually threaded, it's just much easier to think in threads (or "processes") -- they're effectively a language feature.
Re: (Score:2)
OH come on this is bullshit.
Which seems to be your way of saying "exactly right."
Even lisp and scheme aren't "purely functional"
Right. Haskell is.
Re: (Score:2)
OH come on this is bullshit.
Which seems to be your way of saying "exactly right."
It's my way of saying that even in languages that have pushed the boundaries of the functional paradigm, this doesn't matter.
The reality is that functional programing even when "impure" has proven to be a highly effective model, and pretending like impurities within the model discredit that, is just silly.
Even lisp and scheme aren't "purely functional"
Right. Haskell is.
Re: (Score:2)
Actually if you read the LISP documentation they agreed that being purely functional was a huge advantage. SICP for example spends a long time talking about how dangerous any kind of side effect is, and trying to justify why one would use an environmental evaluation (object oriented for example) vs a purely functional method. They also talk about the advantages of laziness vs. eagerness which is another big difference between Haskell and Erlang.
In the end LISP went with eager to simplify computation (its
Re:Why Erlang doesn't matter (Score:5, Informative)
1) Actually, there are quite a few good reasons for this, largely around the complete elimination of mutexing and locks. Just because you don't understand the purpose doesn't mean there wasn't one.
2) Oooooh, a language is faulty because it has a syntax with which you are not familiar. Immediately kill all non-Java clones!
3) They're just lists of numbers; they're neither ASCII nor Latin1. There is unicode parsing in the XMERL module.
Please wait until you know a language before criticizing it.
Re:Why Erlang doesn't matter (Score:4, Informative)
Actually, there are quite a few good reasons for this, largely around the complete elimination of mutexing and locks.
...What? No, the elimination of mutexing and locks is made possible by a shared-nothing architecture.
Oooooh, a language is faulty because it has a syntax with which you are not familiar.
Hey, I mentioned Ruby. I don't mind LISP, either.
The point is not that the language is unfamiliar, the point is that it's inconsistent (and unfamiliar) for no good reason. I use English, but I could make a lot of the same criticisms about it.
They're just lists of numbers;
In that case, the argument becomes, "Erlang has very poor text-processing, if any at all."
If Erlang has text-processing functions that are designed to operate on these "lists of numbers", then yeah, it's pretty much going to be ASCII. And how are Erlang source files read? Could be "neither ASCII nor Latin1" if you like, but they can't be Unicode unless the parser is actually Unicode-aware.
Re:Why Erlang doesn't matter (Score:4, Informative)
Actually, there are quite a few good reasons for this, largely around the complete elimination of mutexing and locks.
...What? No, the elimination of mutexing and locks is made possible by a shared-nothing architecture.
Oooooh, a language is faulty because it has a syntax with which you are not familiar.
Hey, I mentioned Ruby. I don't mind LISP, either.
The point is not that the language is unfamiliar, the point is that it's inconsistent (and unfamiliar) for no good reason. I use English, but I could make a lot of the same criticisms about it.
It's not that it's syntax is /inconsistent/ Erlang is actually incredibly consistent, it's just very different. Once you learn the 3 or 4 quirks that separate it from other languages those 3 or 4 quirks are very consistently applied.
Take for instance the punctuation (not line ending characters as is suggested).
Commas separated arguments in function calls, data constructors, and patterns. Periods separate functions.
Semi-Colons separate clauses. (this is the trickiest, but can be thought of as signifying the existence of multiple cases of pattern matching).
They're just lists of numbers;
In that case, the argument becomes, "Erlang has very poor text-processing, if any at all."
If Erlang has text-processing functions that are designed to operate on these "lists of numbers", then yeah, it's pretty much going to be ASCII. And how are Erlang source files read? Could be "neither ASCII nor Latin1" if you like, but they can't be Unicode unless the parser is actually Unicode-aware.
Re: (Score:2)
...What? No, the elimination of mutexing and locks is made possible by a shared-nothing architecture.
To eliminate locks, you need to satisfy the condition that no data is both aliased and mutable. Erlang does this by making sure that the only mutable data structure is the process dictionary, which is not aliased. This means that in a shared-memory system message passing is just pointer copying.
The other issue, that variables are single static assignment, is due to the fact that Erlang evolved from Prolog, where there are no variables, just unbound terms.
Re:Why Erlang doesn't matter (Score:5, Insightful)
Yes.
Where is Lisp today? Smalltalk?
On the other hand, languages that offered the same features with a familiar syntax have taken over the market.
Re: (Score:2)
Smalltalk: Objective-C, Squeak, Seaside, F-Script...
Re: (Score:3, Interesting)
http://www.paulgraham.com/avg.html [paulgraham.com]
Re: (Score:2)
Where is LISP. Slowly taking over all the other languages. More and more ideas are incorporated into mainstream languages with every generation (say a decade). The only thing missing from most scripting languages now is the interchangeability of code and data.
Re: (Score:3, Insightful)
Well no to use the advantages of these esoteric languages today and not 25 years from now. The people using LISP in the 1950s hard garbage collection, reentrant functions, complex data structures....
Re: (Score:2)
1) Actually, there are quite a few good reasons for this, largely around the complete elimination of mutexing and locks. Just because you don't understand the purpose doesn't mean there wasn't one.
The existence of mutable variables doesn't mean that you need locks. You can use Erlang-style message passing between threads, and still have the convenience of mutable variables where it is safe.
2) Oooooh, a language is faulty because it has a syntax with which you are not familiar. Immediately kill all non-Java
Re: (Score:3, Insightful)
But there's no direct unicode support in the language -- if you're lucky, there are functions you can pipe it through.
3) They're just lists of numbers; they're neither ASCII nor Latin1. There is unicode parsing in the XMERL module.
Which is exactly the problem that GP discussed. There is a huge difference between a true in-language support of Unicode (such as the one in Perl) and just "the Unicode parsing library". In Perl there is a difference between "string of bytes" and "string of characters", and this distinction is made when the string is created (i.e. in the I/O layer when it is read from the file, or in the source code pragma when it is a literal constant). And then all things work as expected (conversion between upper and low
Re: (Score:3, Informative)
1. Invariable variables.
This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.
On the contrary, there are very good reasons for having single-assignment variables. It makes the code more similar to plain mathematics, which makes it easier to reason about, and significantly reduces the number of programming errors. And you don't have to take that from me - there are some 20 years of experience at Ericsson and elsewhere with writing huge telecom applications in Erlang.
2. Weird syntax.
Why, exactly, are there three different kinds of (required) line endings? It seems as though the syntax is designed to be as different from C as possible, while maintaining at least as many quirks. Moreso, even -- when constructing normal, trivial programs, you're going to hit most language features head-on and at their worst. Where's my 'print "hello\n"' that works most other places?
I don't believe the important features of Erlang are mutually-exclusive with the sane syntax of, say, Ruby or Python.
The syntax is certainly different from C, Ruby, or Python, but this is because it is derived from the Prolog syntax. Fur
Re: (Score:2)
Furthermore, it is actually pretty systematic, once you get over those initial differences. It is a poor programmer who cannot master both worlds.
I've written a decent amount of Erlang. I can do it.
Point is, why would I want to?
That's simply wrong. Dynamic code upgrade still works, native or not. It's the unloading of older native code from memory that is not being done (this is safe, but could be a memory leak in a very long running server).
Ah, you're right.
I do consider it a fairly severe deficiency that you get to choose between interpreted bytecode and a memory leak.
Well, good luck, and see you in 20 years.
Done, in Ruby. Took a weekend. I'm polishing it now.
Re: (Score:2)
You wrote a SMP-capable micro-stack green thread scheduler in Ruby, over the weekend?
Nope. I used the existing Thread API. It is theoretically, but not actually, SMP-capable. It should be portable to JRuby, which uses Java threads.
What I did was port the message-passing to a Ruby paradigm -- so, much simpler than it sounds. Essentially, I wrote a proxy for objects -- every method call to such an object stacks up in a queue, and is sent (in the object's thread).
It's not so much a direct port of Erlang as a mapping of what I found valuable in the Erlang philosophy. A class can be written with
Re: (Score:2)
Your class can't be used concurrently, because Ruby is not actually SMP-capable.
But JRuby is.
I should note that, in a simple test, Erlang got about three times slower when I enabled SMP mode.
It can't be used on multiple systems, because the messaging can't be used across nodes.
Except it can, with DRb. Which also provides sloppy SMP support -- just run two nodes on the same machine.
Single-machine, non-concurrency queue-based message passing systems are what we usually call "linked lists".
Erm, what? I know what a linked list is, and I don't see the comparison.
DRb is just another RPC implementation.
...and Erlang's RPC isn't?
In erlang, processes and (immutable) messages are first class entities, whether they're operating locally or remotely.
Again, I have to go, "huh?"
DRb allows objects to be created and "passed" between the machines. I could do:
remote_object = DRB::whatever :arg1, :arg2 ...
remote_object.some_method
How is this differen
Re: (Score:2)
Since strings are just sequences of ordinary numbers, it would seem like you could develop Unicode by just using numbers in unicode character code range. A string could be a tuple with the encoding and the sequence, like this:
{ utf8, [ character1, character2 ... ] }
After reading through the documentation, my main gripe is that it's a totally different way of thinking, and so I have no clue of the first way to do what is accomplished with routine ease in other languages. The Erlang getting started guide ad
Re: (Score:2)
Re: (Score:2)
I was under the impression from reading the tutorial that guards were the only way to create an IF style conditional in the languge and so no user defined functions in guards meant no user defined function in the IF statement (which I did notice exists) as well. Did I get this wrong?
Oh, it's definitely easy to learn. I could certainly learn the language, in the sense of being able to write small programs with it, in a short time.
But the methods needed to create more complex software seem very alien to me
Re: (Score:2)
I don't know of any CGI or web form programs - there's a web server written in Erlang and the most popular (and most feature-complete) Jabber server is written in it, and this is probably a good place to look if you want some example code.
Re: (Score:2)
So what's the advantage of this language over, say, using threads in a language like Ruby or Perl? It seems to me that it's extremely difficult to master and lacks features that exist in virtually every interpreted language in existance today.
Because in practice if you don't stay pure you'll end making assumptions that are false in a multi execution system. It is very hard for people to picture all things that go wrong if the environment their code executes doesn't execute in the same order they expect.
The whole industry of relational databases exists because once we were out of batch: Read a record, modify, write was too difficult. The issue of what happens if the record has changed since you started, made VSAM type (COBOL) structures impossib
Get a clue! (Score:2, Insightful)
It's a functional language you dolt, of course the "variables" are invariant. The only way in which it was the designers preference is that he presumably wanted all the niceties that come with declarative languages; provability, implicit parallelism, etc.
1. Invariable variables.
This appears to have been done for no reason other than the designer's preference. In fact, it's not strictly true -- variables can be unbound, and later bound. They just can't be re-bound once bound.
Why is any syntax that's not filched from C weird? Frankly I'm not that fond of C's syntax it _can_ lead to very unreadable code (Speaking as someone with 20 years experience with C). If you'd done a modicum of research you'd realise that Erlang models its
Re: (Score:2)
Where's my 'print "hello\n"' that works most other places?
Well
#!/usr/bin/env escript
main(_) -> io:format("Hello World\n").
And for something less trivial here is 99 bottles of beer:
simple [99-bottles-of-beer.net] more standard erlang [99-bottles-of-beer.net]
Comparison of functional languages? (Score:5, Interesting)
I think the summary (and article) are somewhat poorly written, but that doesn't shadow the fact that functional languages are becoming more and more interesting these days with concurrency becoming so important.
I'd like to learn one, but there are several out there.. What I'd like to see is a good in-depth comparison of different concurrent functional languages: why would I choose Haskell, or Erlang, or OCaml, for example? Are they all interpreted? (Does one exist that compiles?) Which ones support concurrency? What language features do they boast, and what are the advantages and disadvantages of these features? Do they have a complete set of libraries?
Anyone know of an article like this? I've been searching for a while. Every article on functional languages I've found seems to concentrate on a particular one, but I can't find something helping me decide which one is most worth learning.
Re: (Score:3, Insightful)
Brief answer:
All three languages have both interpreters and compilers (ocaml is part of the base distribution, haskell has a number of compilers, and Erlang apparently has a compiler)
They all support concurrency, all in slightly different ways. They all have a lot of libraries.
Ocaml is sort of a functional language that includes object oriented features and also has very good performance numbers. It allows mutable updates, including arrays and references. For threading I believe it has the usual mutexes and
Re:Comparison of functional languages? (Score:5, Interesting)
OCaml compiles down to native code, which about 10-20% slower than C. Faster than C in a few (narrow) cases.
Haskell is also compiled to native code, but difficulties with the execution model mean it's pretty slow for any practical use.
Erlang is interpreted - the execution model is similar to Perl or Python - which means its slow on single cores, but of course the whole point of Erlang is to run in highly concurrent, distributed machines. There is a project [google.com] to use OCaml for the performance-critical, single threaded parts, and Erlang for coordinating the parallelism.
Of course, this is probably missing the point. Unless you're doing intensive numerical work, you probably don't need the performance. The real advantage of these languages is how your code will be much smaller, easier to understand, safer, and faster to write.
Rich.
Re: (Score:2, Informative)
Just a minor correction: Erlang has native code compilation on quite a few architectures -- try the "+native" flag. Most projects seem content with just using the VM interpreter, though.
Best,
Thomas Lindgren
Re: (Score:2)
Of course, this is probably missing the point. Unless you're doing intensive numerical work, you probably don't need the performance.
...or if you are a company with a large web presence. Using something that runs at half the speed of C++ is not a big deal if you only have 2 servers. However, if you have 1000, then its wasting a lot of money, and it's worth the extra programming time to tune the system. This is kind of why I like OCaml (though I don't get to use it at work), it's a very good balance of a mostly-functional language, speed, and pragmatic features.
Re: (Score:2)
Re: (Score:2, Informative)
Can't point you to a comparison article, but one language you should consider is Scala. It compiles to the Java platform, and thus can interact almost transparently with existing Java code and libraries, and uses Erlang's concurrency model. It can do both functional and imperitive, object-oriented tasks. It's statically-typed, but with features I didn't think were possible outside a dynamic language, such as duck-typing (only compile-time checked!)
It's very powerful, but sometimes hard to figure out. No
Re: (Score:2)
This [defmacro.org] is an excellent article about functional programming.
Ocaml has a very good reputation. It compiles to native code, and has performance comparable to C. It's agnostic about programming style. You can use it as if it was C with garbage collection, or you can us it in an fp style, or you can mix the two. There are good libraries. Because of the way the garbage collector is designed, it doesn't support SMP to the extent that erlang does. The design of ocaml includes some conscious trade-offs of elegance
Re: (Score:2)
Erlang = massively parallel systems
OCaml = most practical / mainstream
Haskell = best language to make sure you really "get it" because the constructs just don't exist to do it your old way.
LISP/Scheme = Most books by far. Best language for really seeing that code is data and data is code. Very old fashioned though.
Re: (Score:3, Informative)
Deceptive (Score:2)
>The two biggest computing-providers of today, Amazon as well as Google, are building their concurrent offerings on top of really concurrent programming languages and systems
Google is largely a C++ company, a language that doesn't include explicit support for concurrency (although the next version, C++0x, will).
They mention erlang only being used in a relatively small project that most of google's own software doesn't support yet.
Note, that google gears is used in the excellent google reader software (al
Re:Deceptive (Score:5, Insightful)
Actually, Gears doesn't use Erlang either. What he means is that Gears threading doesn't allow for shared state (is it really threading then?). Instead threads communicate back to the browser by message passing.
It's remarkably deceptive indeed to even imply that Gears and Erlang are connected. Message passing based concurrency isn't exactly new or limited to Erlang, and can be implemented in any language.
I'm not sure what the point of this piece is. I've looked at Erlang and didn't see much of anything to get me excited. It's a functional language, which like most of them have unnecessarily weird syntax and force immutable state. I don't really see what this buys you over a language like D 2 (or hell, even C++) in which you can write in a functional message passing style if you like, but then still use imperative shared state whenever useful, convenient or performant.
Re: (Score:2)
You're probably looking for an impure functional language like OCaml. You get to use purity when you want to (it helps in reasoning about code), but drop to impurity when you need the performance.
Rich.
Re: (Score:2)
Well, OK. Most functional languages I know require immutable state, but I'll look more closely at OCaml. How though, is it better than v2 of D, which appears to offer most things that functional languages offer (except perhaps an easy pattern matching syntax) but also a whole lot more?
Re: (Score:2)
I don't really know much about D, but OCaml nowadays has loads of libraries, excellent features like macros, type safety everywhere, pattern matching, a super-powerful object system (not used very much, mind you), functors, and it compiles down to fast, tight machine code. If you use Debian or Fedora you can just 'apt-get' or 'yum install' the whole system and almost all of the popular libraries [cocan.org].
Rich.
Re: (Score:2)
I should add that banning or confining mutable state is fairly important for efficient generational garbage collection.
This is often overlooked (for many programmers a good GC is "invisible"), but if you allow mutable state, then you allow for pointers which point from old heaps to young heaps. That complicates and slows down the garbage collection because you have to keep a special list of those pointers. With immutable data, no such pointers can ever exist. The OCaml compiler/runtime allows mutable da
Re: (Score:2)
But immutable data seem really inefficient. Rather than altering one value in the data structure or collection, you have to copy the entire thing to a new location with one value changed!
Re: (Score:2)
Seems really inefficient. but may not be. It really depends on how much data you need to copy versus how much data you can share. And how fast copying is on your processor (probably very fast indeed). And whether your GC strategy can be made globally more efficient because of the immutable data. There's no single answer to this and naturally it depends on the precise application.
Rich.
Re: (Score:2)
Huh? Real Mike, you got me here. I'm hard pressed to even see how D is remotely like a functional language:
-- order of execution is explicit
-- functions have side effects
-- there are explicit looping structures
etc...
Gibberish (Score:5, Insightful)
If you want to build computing into a utility, you need large real-time systems running as sufficiently as possible.
But if you want to build sprockets into a weasel you need small batch-mode systems running as necessarily as possible.
If the poster had anything interesting to say (I'd guess not, but who knows!), it was totally obscured by his lack of grasp of the English language.
Re: (Score:2)
Lol... (/English Major)
Wrong parenthesis added for humour
Too late (Score:4, Funny)
Given that this statement appears almost halfway through the blog post, I would say that it was already too late for that.
Stupid article (Score:5, Informative)
Wow, it's not often I strongly criticise articles around here, but that was total garbage.
For the smart ones that didn't RTFA, here's a quick summary:
For the record, I work for Google and we don't use Erlang anywhere in the codebase. Google Gears restricts you to message passing between threads because JavaScript interpreters are not thread-safe, so it's the only way that can work. Visual Basic threading works the same way for similar reasons. It's not because eliminating shared state is somehow noble and pure, regardless of what the article would have you believe, and in fact systems like BigTable use both shared-state concurrency and message passing based concurrency.
The article says this:
But in fact the Google search engine, which is one of the larger "industrial-grade, internet-grade" systems I know of, is written entirely in C++. A language which is much the same as it was 10-15 years ago. Thus the central point of his argument seems flawed to me.
Seeing as the article is merely an advert for Erlang, I'll engage in some advocacy myself. If you have an interest in programming languages, feel free to check out Erlang, but be aware that such languages are taking options away from you, not giving you more. A multi-paradigm language like version two of D [digitalmars.com] is a better way to go imho - it supports primitives needed to write in a functional style like transitive invariance, as well as a simple lambda syntax, easy closures and first class support for lazyness.
However it also compiles down to self-contained native code in an intuitive way, or at least, a way that's intuitive to the 99.9% of programmers used to imperative languages, unlike Erlang or Haskell. It provides garbage collection but doesn't force you to use it, unlike Java. It doesn't rely on a VM or JIT, unlike C#. It provides some measure of C and C++ interopability, unlike most other languages. And it has lots of time-saving and safety-enhancing features done in a clean way too.
Re:Stupid article (Score:4, Interesting)
I'm not going to disagree with most of your post, I think you're spot on. However, your suggestion of D is totally off. I like the D programming language quite a bit and version 2 is going to be really cool. However, even version 1 of D is not ready for prime-time [timburrell.net]. Version 2 of D is unstable and not recommended for production by even the author himself. All of the other languages you mentioned such as Erlang or Haskell are much more mature.
Also, "most other languages" have a foreign function interface for C, including Erlang, Haskell, Python, Java, Perl, Ruby, etc... In fact, I can't think of a well known programming language actually used by people other than the author that does not have an FFI... It is true that in most cases the FFI of other languages is more difficult to use than the one in D, but they are there.
Re:Stupid article (Score:5, Interesting)
Yes, D is very young and has problems. But then again, what language didn't? It's easy to forget but Python was first released in 1991. It took many years before it became mainstream (and some would say it's still not there yet).
The post-mortem is an interesting document, but I disagree with the authors conclusions. The compilers are buggy, well, C++ had exactly the same problem for a long time but still was a huge success. In particular, the trend seems to be basing new compilers on LLVM, which has a pretty robust optimization core. Frontend bugs are by comparison pretty trivial and easy to fix. Another few years and I think this problem will be licked - and besides, lots of C++ code has workarounds for compiler issues. Same thing for class libraries.
You're right about C-level FFIs. However D provides a simple C++ FFI which as far as I know is unique. Such a thing would be very useful for a company like Google which has a lot of C++ code, as it'd simplify binding considerably (I don't mean to imply anything about the future direction of the codebase, by the way).
The argument about parallelism is a more interesting one. But I disagree with that too :) D provides exactly what is needed for automatic sharding of work across cores (or machines). Specifically the combination of transitive invariance, reflection and purity enforcement is a very powerful one.
Essentially, if you can write your code to consist of non-trivial trees of pure functions, then it's perfectly safe to parallelise something like this:
foreach (item; list) {
fooResults[item] = someTransform(item);
barResults[item] = anotherTransform(item);
}
If someTransform and anotherTransform are both pure, by implication their parameters are transitively invariant, and thus they can both be invoked in parallel (because the compiler knows "item" can't be changed). What's more both calls can be invoked simultaneously as well.
Once the compiler knows these things, making this code run in parallel is simply another compiler optimization. That's the whole theory behind how functional languages can be super easy to parallelize. But in fact the key concepts can be applied to imperative languages as well, with the advantage that you can still have temporary mutable state within the function scopes - you just can't modify the heap, or anything reachable through your arguments.
D has keywords that let the compiler know and enforce function purity.
Now as it happens I doubt that any D compiler today implements this optimisation - it's sophisticated and transitive invariance is newly introduced in D2. But all the pieces of the puzzle are there. This also lets the compiler do calculations on data structures available at compile time.
Re: (Score:2)
Right but what I can't do is is be sure in looking at your code block that someTransform didn't have a side effect that pollutes item when I want to run anotherTransform. Nor can I be sure that one of these two doesn't have some something like a "last" so I can't go through list in whatever order I want.
Re: (Score:2)
Yes you can, that's the point of the "pure" keyword and transitive invariance. If the someTransform function is defined pure you know it will not change item, and the compiler can enforce that.
Re: (Score:2)
OK that's useful. Does it also prevent all side effects (like a file write)? What about the issue of one of them having a last statement?
Re: (Score:3, Insightful)
Sounds great in theory but in the real world you don't get m
Re: (Score:2)
Not to mention Sabre and the VISA credit card processing system -- both use TPF, which is a lot older than Erlang. Meanwhile, many libraries have used the Tandem/Compaq/HP Non-stop system for years -- in fact, I'm pretty sure some have been
Learning it (Score:2)
However, I got stuck a: By the lack of documentation (perhaps I could have downloaded it, but over a mobile phone it would have been painful), and b: one of the tutorials in his book seemed to include a library that he had written, to which there was no source code, and the others then built upon that
Re: (Score:2)
Might have another crack at it.
no new language needed (Score:5, Insightful)
Erlang is a language that has all the right properties and mechanisms in place to do what utility computing requires.
Well, except that it's darned inconvenient to actually write the applications in it.
Google Gears is using Erlang-style concurrency, and the list goes on."
Yup, and it makes more sense to add "Erlang-style concurrency" to existing languages than to throw out everything and switch to Erlang.
Wings 3D is written in Erlang... (Score:3, Interesting)
Andy
"Cloud computing" is an Xmas artifact (Score:5, Interesting)
The enthusiasm for "cloud computing" may evaporate when Xmas rolls around.
I went to a talk at Stanford by the architect of Amazon's web services. It came out in questioning that the real motivation between Amazon's low-priced web services is that their load in the Xmas shopping season is about 4x the load for the rest of the year. Their infrastructure is sized for the November-December peak, so for ten months of the year they have vast excess capacity. That's why Amazon's web services are so cheap.
Don't expect good response time during the shopping season. Although this Xmas might be OK, due to the recession.
Re: (Score:2, Funny)
Re:"Cloud computing" is an Xmas artifact (Score:4, Insightful)
While the origin of EC2 in 2006 is certainly related to peak capacity requirements at Amazon, it is certainly way beyond that point now.
Two Christmas seasons have come and gone without major capacity problems on EC2.
The reality is that EC2 has grown far beyond its roots as a way for Amazon to amortize their peak capacity by reselling it and it has turned into a small but growing profit center and publicity success for Amazon.
Erland textbook: read and adapt (Score:3, Interesting)
Dunno if Google uses Erlang, but Facebook does (Score:2, Interesting)
They built the Facebook Chat backend using Erlang. Scaling something from 0 users one day to tens of millions of active users the next day is a challenge, and they decided Erlang was the right tool for the job. http://www.facebook.com/note.php?note_id=14218138919&id=9445547199
sufficient efficiency? (Score:2)
>> you need large real-time systems running as sufficiently as possible.
lol. Are they running vista then? oh wait.. that would be INsufficient...
Going from Perl to Erlang, eh? (Score:3, Informative)
TFA more or less says that IMDB is switching from Perl to Erlang. So I looked at the link and here's what I got:
(From here [computerjobs.com]
We are looking for developers with experience building web scale distributed systems. We are currently working in Perl but have plans to use Java, Erlang and any other language that we think will suit our purposes. We aren't looking for expertise in any of those, particularly, but we expect that you will be an expert in the systems you know. We do require that you be passionate about testing (unit, integration, fault-injection) and code quality. Experience with relational databases (Oracle, MySQL, etc), embedded databases (BerkeleyDB, CDB, MonetDB, etc) and Linux are a big plus.
I'll leave anyone to draw his own conclusions.
Concurrency Fad? (Score:2)
People are turning to currency-friendly languages as kind of a fad. A solid need has not been identified for it yet except maybe action gaming. Outside of that, a smart use of the database takes care of most if not all of the alleged "concurrency problems" that are being generated by new chip sets.
I agree that database architecture may need an adjustment to take advantage of more RAM, but that shouldn't require a significant change in the way applications are written. DB's already insulate most apps from im
Re: (Score:2)
Since this sounds practiced thought I'd comment that Software Transactional Memory [wikipedia.org] essentially uses the approach you are suggesting. Then just certain parts of the compiler need to be functional not the apps themselves.
Facebook chat (Score:3, Interesting)
Re: (Score:2)
Google wrote their own language for parallelism called Sawzall [wikipedia.org]. I'm not sure where they're using it in particular (possibly as a front-end to MapReduce), but they've published some research papers about it.