Grid Computing at a Glance 96
An anonymous reader writes "Grid computing is the "next big thing," and this article's goal is to provide a "10,000-foot view" of key concepts. This article relates many Grid computing concepts to known quantities for developers, such as object-oriented programming, XML, and Web services. The author offers a reading list of white papers, articles, and books where you can find out more about Grid computing."
Selling your cycles (Score:5, Interesting)
it's not all about the cycles (Score:5, Insightful)
see, that's the major fallacy of the hype behind "The Grid". yes, one of the benefits can be seen in the supercomputing realm, where you can link up many different machines (we haven't gotten to doing this between architectures yet, mind you) to make a gianto-machine.
however, the key in *all* of this is the technologies that allow for that to happen, along with the data transfer, authentication, and authorization, et al, that have to happen.
as far as cycles go, no, we probably won't see a dynamically created, scheduled, and allocated meta-supercomputer anytime soon. most companies will use these technologies to make static or mostly-static links between a few select sites and partners for now.
however, these protocols (GridFTP, ack), standards (OGSA,
to wit: the killer app in "The Grid" is not to make a giant supercomputer. it's to develop a lot of different ideas and technologies which allow for resource sharing (at the general level, among other things) to occur in a standardized, efficient, and logical fashion in the future. noone will use all of them, but the key is to use what you need from what "The Grid" encompasses. that's why it's referred to as "The Third Wave of Computing"!
Re:it's not all about the cycles (Score:4, Informative)
Grid computing is not about making a giant computing farm out of a bunch of distributed machines.
Make that "not" a "not only", and I totally agree with you. See, I work with EDG (european data grid, based e.g. on Globus authentication) on a daly basis. And for us it is merely a tool to make exactly that, namely a giant computing farm out of our computing farms in USA, UK, France, and Germany. It really sucks to log into all our datacenters and see where the batch queues are least utilizised. With the grid, all our batch farms look like a single farm and I just submit my job and don't need to care where in the world it is running. That is exactly the small part of the "Grid" cloud which we are picking for us.
Now to the cycle based selling of our spare time. You would be surprized to hear how many hours I spend a week to implement exactly this. The finance department calculated the prize of lost cycles our farm had the last quartal and it will probably pay out for us to spent 0.5 FullTimeEmployees to work on trying to sell those on a three years timescale.
There are many aspects of "Grid Computing", as you say, but most if not all of them are based on large scale science projects (me) or on big business. I am most curious to see if Grid computing will eventually find it's way to a home user. I heard that Sony is using Grid tech. to connect computing centers which are supposed to host multi-player games. The home user will most likely not get in touch with the Grid soon though.
Cheers
Re:it's not all about the cycles (Score:1)
yup. i don't expect mr. home user to use "the grid" any more than I expect him
Re:it's not all about the cycles (Score:2)
Re:it's not all about the cycles (Score:1)
Re:it's not all about the cycles (Score:1)
Mabye you havn't heard of PVM [ornl.gov] or MPI [anl.gov].
Re:it's not all about the cycles (Score:1)
neither of these allow for an autonomous, dynamic, automatic architecture-spanning system. yet.
Re:it's not all about the cycles (Score:1, Insightful)
Re:it's not all about the cycles (Score:2)
This is much like a lot of the distributed computing systems out there these days. I don't think Folding@Home cares whether you ran their work unit on a
Re:it's not all about the cycles (Score:3, Interesting)
Re:Selling your cycles (Score:1)
Re:Selling your cycles (Score:1)
Sigh, anyway I will try again next week, and if they dont bother with it I will try some other outfit.
Re:Selling your cycles (Score:2)
Re:Selling your cycles (Score:1)
Is their product in C or Java?
Re:Selling your cycles (Score:2)
Next big thing? Again? (Score:4, Funny)
But I thought that this [slashdot.org] was the next "killer app"?
Re:Next big thing? Again? (Score:1)
Social Software (Score:1, Funny)
I can see it now... (Score:4, Interesting)
2. spyware that not only spies but also hijacks your CPU cycles for remote computation
3. dubious companies selling "grid computing" service pop up all over the place
4.
5. Profit?
It may look funny, but what if the next version of Windows comes embedded with this kind of thing? All it would take would be some marketing genius to convince enough people. (disclaimer: yes this is slightly paranoid, it's not intended to be MS bashing, just an example on how this technology could be misused).
Re:I can see it now... (Score:2)
It already comes with an enabling technology - the Outlook Express Scripting Engine.
Possibly one day it'll be more lucrative to exploit OE for grid computing than for opening a SMTP relay - then we will know that it has really arrived as a mature technology.
Patriot@Home (Score:1)
The projects that the grid is best at are pretty much the areas that already have 'grid' projects, biochemistry [grid.org], genetics, SETI [berkeley.edu] and some maths problems. In which I include one of the most appropriate maths problems for the grid, is brute force password attack. How long before the US Gov. starts a Patriot@home grid to brute force any encrypted files it wants to see, in the name of homeland security...of course.
It Could be (Score:1, Interesting)
With The realitivly recent move of object oriented Programming, you could think of this as just the next level of abstraction, abstracting your objects out to a broader system level as apposed to an implemenatation level.
In Any event it would be a good scheme for many things that need distributed systems....such as crypto
Re:It Could be (Score:2, Interesting)
Basically the toolkit approach implements a low level set of common grid functionality, security, job monitoring, brokering etc, which is then leveraged by other apps.
Of course the toolkit can to some extent be wrapped in OO methods and abstracted away, but its not pure OO.
That's what happens when Computational Scientists are allowed to design things.
Re:It Could be (Score:3, Insightful)
The fact that desktop pc's are 5-20% utilized is why you can just claim another layer of abstraction won't hurt you.
--- now please go and find me a list of things that "needs distributed".
-- next from your list remove any jobs that do not parallelize in to chunks of data that can fit in common machines --- yes the grid will have some big boxen, but do you think you are going to reliably get farmed onto one of those?
-- next from the remainder that you have
Re:It Could be (Score:2)
The fact that desktop pc's are 5-20% utilized is why you can just claim another layer of abstraction won't hurt you.
Amzaing how some people can be so passionate yet know so little
we are talking about the concept here...not the individual implementations...ther are OO implementations (ie kde) that abstract out without preformance hits
Now go back to your single threaded world
Re:It Could be (Score:2)
What you say has been said in an amazingly similar wording to WWW back in the days when we (High Energy physicists) developed the stuff at CERN.
HTML would be nice to excange scientific documentation and news, but no company would ever benefit from it, let alone home users.
Nowerdays everybody is using WWW. Industries rely on it. Maybe this will not happen to Grid t
Re:It Could be (Score:1)
I agree, the people that need massive computing power now and in the near future are pretty much all running finite element analyses or similar (aero/hydrodynamics,nuclear explosion,climate, quantum physics,galaxy models). This method doesn't scale too badly on a supercomputer,but it relys on rapid and regular communication between processors in order to work efficiently, something which the grid (in a global sense) is unlikely to provide in the near future
sourceforge? (Score:1)
hrm..... (Score:1, Troll)
xao
NOT the next big thing (Score:5, Funny)
I happen to know that beowulf clusters of quantum iPods, built by nanobots, running social software, using a Post-OOP paradigm and a journaled filesystem over a wireless IPv6 network to make profit with a subscription-based publishing model will be the next big thing.
Re:NOT the next big thing (Score:2)
so, yes.
More grid info (Score:5, Interesting)
They also host an open source project Grid Engine [sunsource.net] for the software. The software used to be commercial, but Sun bought it and open sourced it, like they did with Open Office.
grid computing... (Score:2, Interesting)
Re:grid computing... (Score:1)
It's more like "distributed computing." The granularity of parallelism is much, much larger than you'd get on an SMP architecture.
You're all wrong (Score:1)
Never mainstream (Score:5, Insightful)
1) Still need a central server for storage/backup
2) One server needs one UPS, 1000 workstations...
3) Worsktations are flaky: They reboot, crash, play video games, etc. The distributed software can handle this, but the inefficiency involved is painstaking. I hope everybody doesn't run Windows Update all at once, or all the PCs could go down.
4) The corporate network is now a bottleneck.
I rattled off this list in about 30 seconds, so I'm sure there are lots more. Since these are physical limitations, not technology limitations, they aren't going away.
No, this can be practical for many applications (Score:2)
Well, this scenario would not be appropriate, since there's hardly any processing involved in web orders. Mostly that is just database queries. But you could easily imagine that you'd see a useful speedup if you had your advertising firm's 3D animations rendering on every computer in the office, or your software development company's nightly build/regression suite. Fault tolerance (not trivial, but not impossible either) takes care of 2 and 3, so you just need to find an application that's a ppropriate to t
Re:Never mainstream (Score:1)
What is mainstream? Even if it doesn't become mainstream, does that mean it can't be "the next big thing" or not be involved in useful science? The NEESgrid project [neesgrid.org] relies on creating a grid infrastructure, and the system architecture of that grid involves storage at each equipment site that is a part of that virtual organization, not (only) some central storage server. Standing up a NEESpop, a TelePresence server, or a data storage server does not require 1000 workstations, although it may require a UPS
Re:Never mainstream (Score:5, Insightful)
_Your_ workstation may be flakey, but real workstations are not:
peu@elrsr-4 peu $ uptime 19:33:50 up 140 days, 2:01, 3 users, load average: 0.26, 0.26, 0.14
So grid computing gives you just one more reason to move your company desktops to AIX, Linux, BSD, IRIX, or other competent operating system of your choice.
Re:Never mainstream (Score:3, Insightful)
1) Still need a central server for storage/backup
There might be interesting applications of grid computing for distributed, redundant storage, but the classic applications wo
Re:Never mainstream (Score:1)
1: The Network is reliable
2: Latency is zero
3: Bandwidth is infinite
4: The Network is secure
5: Topology doesn't change
6: There is one administrator
7: Transport cost is zero
Grid computing addresses 1, 4, and 7 IMHO, yet leaves 2 and 3 unsolved. Since you can't even solve the "infinite bandwidth" issue with Grid computing, I submit that "Grid Computing" isn't The Last Word (tm) on computing...
Maybe a bit long in the tooth already... (Score:1, Interesting)
10,000-foot view? (Score:1, Offtopic)
Software Architectures for Grid Computing (Score:4, Interesting)
Currently for distributed computing we have Thin-Client/Fat-Server, Client/Server, N-Tier and Shared-Node architectures. I think most people are expecting a Shared-Node or Client/Server for Grid Computing because that is how existing implementations work. The issue with either of those is the size of the work unit. If the work unit is small than the nodes/clients must sychronize often. If the work unit is large then you are more likley to have nodes/clients in a wait state because required processing is not completed.
Using a network style architecture (distributed Shared-Node) raises more issues because of message routing. Interestingly, this is the 'web-service' model! For example a web site must verify a customer, charge her credit card, initiate a shipping action and order from a factory in a single transaction. So you get four sub-transactions. Let's say that each of those initiates two sub-transactions of its own and each of those initiates one sub transaction of its own. We now have a total of twenty transactions in a hierarchy that is three deep. Let's also assume that we only have one dependancy (the verification) before launching all other transactions asychronously.
The problem here is response times, they add up. if the average response time is 500 ms, then three transactions deep gives us 1500 ms. The dependacy, at a minimum, doubles this. So it takes three full seconds to commit the transaction. Something a user might be willing to live with until a netstorm occurs and the response time drops to thirty seconds or more. (Note: Isn't it funny how you never see this math done in the whitepapers pusing web services?) But three seconds is far too long for sychronizing between nodes of a distributed computing grid unless you only have to do it every once in a great while, pushing us towards large work units and idle nodes!
So the Internet itself imposes costs on a distributed model that wouldn't exist on, say, a Beowulf cluster because that cluster would have a dedicated high-speed network. Client/Server architectures work better for the Internet, but require dedicated servers and a lot of bandwidth to and from them.
I believe the real answer lies in what I call a Cell architecture. This would require servers, but their job would be to hook up nodes into computing 'cells' consisting of one to N (where N is less than 256?) nodes. Each node would download a work-unit from the server appropriately sized to the cell, along with net addresses of the other nodes in the cell. Communication would occur between the nodes until the computation is complete and then the result would be sent back to the server. When a node completes its work unit (even if all computation for the cell is not complete) it detaches and contacts the server for another cell assignment.
By reducing cross-talk to direct contact between nodes within the cell we allow smaller work units. By using a server to coordinate nodes into cells we are allowed to treat the cells as larger virtual work units.
Comments?
This is how things like Globus and Condor work (Score:1)
If you have an app which is Grid-enabled, a hydrology simulation for instance, you would get accounts on the various NCSA Grid nodes. Then you would use Globus or Condor, or th
Just 10000 feet? Bah! (Score:5, Funny)
what kind of distance is that? about 2 miles? (Score:2)
Can you give that in metric for us euros ;-) ?
in other words... (Score:1)
Some companies have "a not invented here" problem with stuff but not IBM : Java, Linux, Cell, J2EE. Is there anything more substantial to IBM than a marketing department and two factories (one to make models of factories and the other churning out hard-disks designed to fry after 24hrs continuous use).
Why doesn't IBM just "show Sun the money" so they can get it on
Some problems. (Score:3, Interesting)
The entire GRID standard actually only covers the data transfer and login. Becasue that's the only thing standard about the different types of hardware. You still need to write the software specific to the hardware. Even with tools like MPI programming for Sun big iron is nothing at all like IBM big iron. And you dont exactly use Java. The value is not in the software - that's why it's getting standardized and is given away for free. The value, as always, is in owning a huge pool of computing power and renting it out, or even better, selling it in racks full.
The only people benefiting financially are the people that make the hardware - IBM, HP, Sun, Fujitsu, etc. Just like 30 years ago. Open Source has completely devalued the software - why pay for that, money is better spent on more hardware.
Then there is the cost of transporting the terabytes of data involved in the types of problems you do with these systems. Transport costs are more then the computing costs in many cases - another reason that part got standardized.
Hardware costs are falling FAST. Blade mounted and racked CPU are running about $500/Ghz ($7k for the same from IBM). That means for about 1 million you can get something like 2K CPUs and 2Thz of power, running Linux and all the tools you need. Thats a lot of FLOPS.
For those kinds of costs, outsourcing it at seems silly. You still have to do all software development, data transport, post-processing, and research yourself anyway, and those costs DWARF the hardware/electricity/HVAC costs of owning the hardware and having exclusive access 24/7 until the next updgrade.
How about the 10-inch view? (Score:3, Interesting)
I've seen a few articles giving the 10-micron view, describing CPU architectures making use of a grid topology.
I've seen a few small demos of massively distributed clusters. I've heard hype about the idea of a service provider and service consumer oriented topology. I've heard about self-healing networks. I've heard about the PS3 making use of a grid-based system.
I have not heard any of the "step 2s", the means by which we transition from individual PCs accessing a network, to a single shared "grid computer" actually composed of the network. At least, nothing that would make the resulting network noticeably different than the current internet.
For individual systems (ala the PS3), grid computing seems like possibly the next big thing, sort-of an evolution of SMP to a scale larger than dual/quad CPU systems. The rest of it, the over-hyped massive "revolution" in how people use computing resources in general? Pure marketing hot-air, and nothing more. The closest we'll get to the idea of a worldwide "grid" will consist of an XBox-like home media console with anything-on-demand (for a fee).
Good for CPU bound processes only (Score:5, Informative)
As we discovered early on in MIMD parallel computing, MIMD (aka grid computing) parallelism can only really help processes that are CPU bound in the first place.
Most of the processes that require 'big iron' are memory bound and I/O bound--e.g. databases that are hundreds of gigabytes to terabytes in size. This is why so many CPUs are '90% idle' in the first place, and this is why system designers devote more attention to bit-striping their disks, a good RAID controller, bus speeds, disk seek time and so forth.
Problems that require brute-force computation on small amounts of data, and produce small results, are simply few and far between -- and the people addressing those problems have been onto MIMD for decades. For instance, my first publication, in 1987 to the USENIX UNIX on Supercomputers proceedings, involved putting ODE solvers wrapped in Sun RPC, so that hundreds of servers could work on a different part of initial condition and boundary condition space, to provide a complete picture of the properties of certain nonlinear ordinary differential equations. Cryptanalysis and protein folding problems are already being addressed in a similar manner, and the tools to distribute these services as well as the required communications standards have been around for more than a decade.
Furthermore, if you've already got a marginally communications-bound domain decomposition of a parallel problem, and you want to cut down the communications overhead in order to take advantage of MIMD parallelism, the last communications protocol you're going to use is a high-overhead one such as CORBA, or a text-based message protocol such as XML. Both XDR and MPI are faster, more stable and better established in the scientific computing community than Yet Another layer of MIMD middleware--which is all Grid Computing is.
clustering daydream (Score:1)
TeraGrid (Score:3, Interesting)
They did this at squaresoft during FF the movie (Score:1, Interesting)
Re:They did this at squaresoft during FF the movie (Score:1)
User moderation!! (Score:3, Interesting)
Hell yes I want that. (Score:1)
Only problem with distributing internal stuff to external machines is trust and better is denyability.
So ripping and compressing 1000 DVDs or 1million MP3s with better quality is probably not a good idea unless there is some method to cloak what is happening.
GridShell Expert System User Interface (Score:2, Interesting)
Oh, and it's all 100% Object-Oriented Perl, for those of you who care about clean code.
More really crazy GridS
Re:GridShell Expert System User Interface (Score:2)
Oh, and it's all 100% Object-Oriented Perl, for those of you who care about clean code.
I want to say something here, but I can't quite find the words.
*Wanders off, muttering...*
"Perl
"..."
"Perl?
"..."
*...shakes head in confused amazement*
(It does sound like a fascinating project and all, it's just that I've never heard those two terms associated that way before.)
Service based computing. (Score:2)
User says I want service blah, service broker manages where to run the blah application. You can kill loads of machines and the service continues to exist on the network.
Oh, not again.... (Score:2)
First it was Globus... now companies have latched on that whole idea, hoping it'll be "the thing".
And you know what? It's not.
There are two points to all this:
The people who are already involved with this have already declared victory, and are going to work towards this, no matter if it's going to work or not. After five years of pushing it, you'd think they'd get the idea that people just aren't buying into it... at leas
more detailed info (Score:2, Interesting)
OpenMosix and COWs (Score:3, Informative)
I do software and sysadmin for scientists. Those with simulation or data analysis needs usually work either:
- connecting remotely to a main computer (say SGI in many cases) to run their jobs, at a high price for hardware and support and at the risk of saturating the machine when everyone wants in;
- or, with the more recent increase in computer power in PCs, running directly on their own PCs.
In both cases the PCs are underutilized most of the time. OpenMosix is a patch to the Linux Kernel allowing you to transform your workstations into a cluster. No software modification is necessary. OpenMosix balances the load automagically. No more expensive mainframe. No more powerful but underutilized PCs.OpenMosix has been featured on /. before: here [slashdot.org], here [slashdot.org], here [slashdot.org], and here [slashdot.org]