MySQL Creator Contemplates RAM-only Databases 290
Aavidwriter writes "Peter Wayner asks Michael 'Monty' Widenius of MySQL, 'When will RAM prices make disk drives obsolete for database developers?' From Monty's answers, it sounds like hard drives may be nothing but backups before long." From experience, I'd wager that RAM failure rates are less than hard drive failure rates, so it might also mean more stability from that perspective.
Not only need oodles of RAM... (Score:4, Insightful)
Re:Not only need oodles of RAM... (Score:2, Informative)
Depending on the application it may not be feasible to have a in memory DB for a long time.
I currently work on DB's that are many hundreds of gigabytes as well as terabytes, some are even pentabytes(sp?). It will be a long time before a we can afford to buy a terabyte of memory to hold our DB's.
Eventually it could be a possibility but its still cheaper to buy hard drives and let the DB just cache in memory the active store
Umm...now? (Score:2, Funny)
Re:Umm...now? (Score:3, Informative)
The real speed improvements, according to the guys working on projects like Bamboo on sourceforge come not from the fact that it's in RAM, they test against SQL in RAM and show that most of the performance improvements come from keeping the data in the same process space as the application operating on the data. If they're ri
Re:Umm...now? (Score:2)
Re:Umm...now? (Score:3, Interesting)
And they wonder why I'm crazy.
Re:Umm...now? (Score:2, Interesting)
ECC RAM? (Score:4, Interesting)
Re:ECC RAM? (Score:2)
Re: (Score:3, Informative)
Re:ECC RAM? (Score:2)
Re:ECC RAM? (Score:2)
Re:ECC RAM? (Score:5, Informative)
Normal RAM: no errors detected
Parity RAM: an extra bit is used to store the XOR of all the othe bits on the line. If the math doesn't work, the OS can be informed the data at a particular address is corrupt, and let the software decide how to best cope with the condition. Only odd numbers of bit errors can be detected.
ECC RAM: Uses a more complicated method and even more bits to provide single-bit error *correction* and double-bit error detection. The odds of two bits on the same line being corrupt is slim unless the ram is really bad. Single bit errors cause no problems. Double bit errors are at *least* detected. More than that and chances are the memory module is so screwed that even if the OS doesn't get it reported, the problem will be obvious to the user.
Re:ECC RAM? (Score:3, Informative)
Re:ECC RAM? (Score:2)
Another idea, have m x n banks of effective storage by using (m+1) x (n+1) actual banks of memory and do the XOR
Re:ECC RAM? (Score:5, Interesting)
Correct me if I'm wrong (Score:3, Insightful)
Re:Correct me if I'm wrong (Score:5, Informative)
RAM only needs a trickle to keep refreshed...
Re:Correct me if I'm wrong (Score:2, Informative)
If you ask me, RAM is still too expensive to make this feasible. The article seems to assume that the examples listed are typical. But for so many purposes the idea of keeping the whole database in memory is a huge waste unless RAM gets a lot cheaper. For $100 you can get 120GB of HD or 256MB of RAM. That's not a comparable expense. And depending on the database, you are
Re:Correct me if I'm wrong (Score:5, Interesting)
time for some napkin math:
1 512MB ecc reg pc2100 dim -> $ 78 or $156GB
1 70GB Fibre Channel Drive -> $700 or $ 10GB
Now lets factor in raid (for access speed and redundancy)
we typically put 8 drives in a bundle which tends to give us 36% of the total drive capacity (mirrored raid 5 aka raid 6 remember teh ram is ecc reg so this factoring is already in place for it)
8 * $700 -> $5600 for
36% * 8 * 70 = 200GB
This give me approximately 1GB for $28
now thats a factor of 5.6 (call it 6) in price from ram only. AND i still get a prolly 4 fold increase in throughput. Not bad at all in my book.
True, however.... (Score:5, Insightful)
UPS would prevent the data loss, the database could be written to disk when the power fails.
Re:True, however.... (Score:2)
EMC and similar storage arrays have done this for decades; the array cabinet itself contains enough backup power to flush the caches to disk and power down cleanly if the main power supply fails (assuming you didn't already have a UPS there, of course, and if you can afford EMC then you can a
write-back scheme. but what about heavy load? (Score:2)
Re:write-back scheme. but what about heavy load? (Score:2)
Re:True, however.... (Score:2)
Re:Correct me if I'm wrong (Score:5, Informative)
UPS is old technology, the battery needs constant replacement, and very few have multiple redundant batteries and/or transistors to deal with wear and tear. Yes even a simple MOSFET transistor is not 100% reliable [glary.com]. Usually the only way to tell a battery is dead is your UPS fails when you need it (this happened to us when my MD was demonstrating our service live to customers, afterwards was the only time he's taken less than 9 months to sign off a purchase order on new equipment). A UPS also has a power cord to pull out when you recoil after burning your fingers on a Seagate Cheetah 15000RPM HD in the server room. A UPS also trips if you overload it, which again means the UPS fails when you most need it.
Other posts mention cosmic radiation at high altitude makes RAM fail. Last time I checked there were no Quad-Xeon Oracle databases on Concorde, although if the International Space Station were to use one this might pose a problem for non-ECC RAM. Anyway, somebody could always write a driver to do software-ECC with Reed-Solomon for RAM if it becomes necessary.
Huge databases (>500 Gigabyes) would benefit most from this as running a simple OUTER JOIN query on the biggest tables will require most of the database to be called into RAM.
- Small databases become slow due to HD latency problems if they do a lot of WRITE operations (the database is stored in RAM, the transaction log is appended to, COMMIT TRANS). This would benefit least FROM RAMdisk because a HD append operation is cheap, however it would benefit database speed in mid-backup
- Mid-size databases become HD-intensive due to aggregate queries/triggered operations over large '>RAM' datasets. For instance enforced cascading deletes where millions of tuples are being deleted cascaded to hundreds of other unindexed tables (in my job I go to the toilet whenever I run a query like this).
- Huge databases where 'Index size' > 'RAM size' - the simplest query would benefit hugely from more RAM or faster storage or RAM-storage. With current databases this would be a 10Gig Eth connection to a Terabyte RAMSan solid-state disk [superssd.com].
In the future, who knows, maybe a FPGA/ASIC DPU (Database Processing Unit) for INSTANT COMMIT like NVidia's GPU?Re:Correct me if I'm wrong (Score:4, Funny)
Don't have to what?.
What is this strange new word you use? Some new jargon? Don't you geeks have enough incomprehensible terms already without making up new ones? Why can't you just keep things simple and not use all this "technical talk".
Re:Correct me if I'm wrong (Score:2)
I already do this in SQL... (Score:3, Interesting)
Re:I already do this in SQL... (Score:5, Informative)
It's also interesting to note that Microsoft was going to release what they called In Memory Database(IMDB) support in Windows 2000. However, this feature was removed after Windows 2000 RC2 due to technical issues.
Re:I already do this in SQL... (Score:2)
Not so much technical as priorities. They discovered that they were essentially rewriting all the code in SQL Server and that it wasn't signifigantly faster than SQL Server. It was decided it was simply a waste of time.
Like others have pointed out SQL Server, Oracle, etc. already maintain their own caches of data in memory. So on SQL that accesses the same chunks of data, the response is really quite fast.
Re:I already do this in SQL... (Score:2)
Wow - now that must be something to see, a techincal issue that would stop _Microsoft_ releasing a product.
Re:I already do this in SQL... (Score:2)
> widespread adoption.
Wow! Whats funny is someone just like you said the same thing about harddrives.
I mean, $25,000,000 for a 10 gig drive?!? Thats a tad high to EVER take off!
Except for the fact we can now get a drive 25 times as large for 100,000 times less the price (250gb for $250) and not only that but 10gig drives and much larger ARE common place, so i'd have to say you have very high chances of being wrong
The same can b
bad idea (Score:2, Insightful)
Re:bad idea (Score:3, Funny)
I've heard about comic radiation..
It tickles and is really funny.
Already exists? (Score:4, Insightful)
Nobody in their right mind would have a busy database server which accesses the hard disk like crazy. A few years back I saw Oracle servers running NT with 4GB of RAM, so I guess they're using even more now.
Re:Already exists? (Score:3, Insightful)
I few years back, I saw a Sun box running Oracle with 64 Gig of RAM... They're already using quite a bit more. I can't even begin to fathom how much RAM a DB stored in RAM would take. It would be absolutely astronomical for any reasonable sized database. Sysadmins would spend all day swapping out RAM sticks as they died.
Re:Already exists? (Score:5, Insightful)
Well, a professional database like Oracle manages its own cache, but MySQL really only relies only on the OS-level cache. The problem with that approach is that the database knows a lot more about what you're doing, so it can make much smarter decisions about what to cache, what to age out, what to prefetch, etc. On an Oracle server, you want to lock the buffer in memory, and leave as little as possible over for the OS filesystem cache. You see, if a block doesn't exist in the cache, it has to be fetched from disk into the database cache, and if it does, the db will go straight to its own cache. Another caching layer inbetween provided by the OS is just wasted.
I don't think Monty understands any of this; in the article he seems to say that ACID, rather than being a fundamental principle of relational databases, is just something you need to do because disks are slower than RAM. The only reason that you might not want full ACID and use semaphores instead, as he suggests, is because you are only updating one record in one table in a single transaction!
Further, if he is thinking in terms of a few Gb of data, then he is a little out of touch with modern database usage. SouthWest airlines do not have a database that stores 10 bytes of data for every seat on every flight, and I have a hard time figuring out why they would want to - the database of seats would be tied into the customer records, the freight/baggage handling database, the billing records, the accounting system. That is the point of relational database, that you can query data based on data that exists elsewhere. Monty thinks in terms of databases that have only a few tables, because that's all you can do in MySQL. He says that database programmers are forced to be very careful about the consistency of their data - well those using MySQL are, but those using Oracle (or any other database with real transactions and real integrity constraints) find it's all taken care of transparently.
Re:Already exists? (Score:2)
Re:Already exists? (Score:5, Interesting)
Impossible to put a figure on the total amount of data that exists within an organization, but a typical SAN in a major financial institution has terabytes online. UBS Warburg has 2 Tb [oracle.com] in just its general ledger database. Acxion has 25 Tb [oracle.com] in its data warehouse, which will mainly be used for queries, whereas the GL database will be more transaction heavy. SouthWest [oracle.com] is an Oracle customer, but it doesn't say here how much data they have.
Re:Already exists? (Score:3, Interesting)
In fairness to Monty, it's his interviewer, Peter Wayner, who suggests that ACID is just for keeping RAM and disk synchronized. Monty at one point cautions, "You still need commit/rollback, as this functionality is still very useful on the application level."
Re:Already exists? (Score:3)
Uh, something very very different from a cache as one might normally use the term in the context of a database implementation.
I find it staggering that MySQL doesn't have a cache. It's not like it's a difficult thing to write -- you need to have a page in memory to look at it, so it costs almost nothing to keep it there.
Dipshit.
On the other hand, why am I bothering to tell you anything?
Re:inaccuracies... (Score:3, Insightful)
Well, Monty started it by talking about SouthWest's booking system. Airlines are among the heaviest IT users in the world. If you want to drop names, be prepared to be called on your assertions by someone who does know. Someone
Re:Already exists? (Score:4, Insightful)
The whole thing was about not writing to disk *ever* (in the normal running of the database anyway).
No matter how much memory buffer cache the database is using, in order to be ACID it has to be *writing* to the disk and waiting for those writes to be *written to disk*.
Removing the disk completely removes any need for all that...
Re:Already exists? (Score:2)
Writes go to the RAM buffer which flushes immediately to a cache disk, which never bothers seeking to a new track. During idle time this stuff is written back to the actual data disk.
Re:Already exists? (Score:2)
And the Computing Darwin Awards... (Score:2, Interesting)
Now if the RAM was non-volatile and was static with the power off that would rock, but volitile RAM - are you crazy?!!
Been there Done that got the T-shirt.... (Score:2, Interesting)
Re:And the Computing Darwin Awards... (Score:2)
So what about all this "flash ram" stuff we see for our mp3 players, portable pr0n viewers, and whatnot?
Maybe someone could design a new kind of high-speed drive fitting into a standard drive bay, that takes a bunch of MultiMediaCard's in an (m+1) x (n+1) matrix to provide a total storage of m x n cards, using RAID principals in two dimensions such that the memory controller could correct any number of bit errors in a s
unique problems (Score:3, Interesting)
Re:unique problems (Score:2)
Or are you talking about the stability of RAM-resident databases (which is NOT what the line you quoted was talking about, it was purely about RAM failure rates).
Considering the non-existent ACID support (Score:5, Funny)
It works! (Score:4, Interesting)
Fundamental concerns... (Score:5, Insightful)
I guess the RAM vs disk debate is on similar lines - but there are some vital differences:
1. Disks (esp. IDE) have become a commodity item and can be accessed by different system architectures easily.
2. IDE and SCSI standards have stood the test of time - 13 and 20 years respectively, unlike RAM wihch has evolved from Parity, non-EDO, EDO, DRAM, SDRAM, DDR-RAM, RAMBUS RAM etc., and suffers several patent and copyright encumberances.
3. Although RAM prices are driving down, the h/w to interface speciality RAM banks is proprietary and hence cost-prohibitive, and comes with attendant long-term supportability risks - think Palladium, or even Server mobos over the last 10 years. TCO for RAM based systems could thus be much higher than disk-based systems.
Overall, except for apps that need super-high speeds, and users that can risk proprietary stuff, disk-based databases shall remain.
My 0.02
Some thoughts on RAM (Score:5, Interesting)
Recent discussions about disks vs. CPU's have ignored the massive decreases in the cost of RAM. For a very long time, the secret bottleneck in PC's (in that it wasn't advertised heavily) was RAM. That's starting to disappear -- there's a gig in my laptop, and there's no discernable improvement in all but the most intense applications if I were to go beyond that.
Virtual Memory is already on the chopping block; any time it's imaginable that a system might need another gig of storage, it's probably worth going to the store and spending the hundred dollars.
But what if more RAM is indeed needed? One of the most interesting developments in this department has involved RDMA [infinibandta.org]: Remote DMA over Ethernet. Effectively, with RAM several orders of magnitude faster than disk, and with Ethernet achieving disk-interface speeds of 120MB/s, we can either a) use other machines as our "VM" failover, or more interestingly, b) Directly treat remote RAM as a local resource -- a whole new class of zero copy networking. This Is Cool, though there are security issues as internal system architectures get exposed to the rough and tumble world outside the box. It'll be interesting to see how they're addressed (firewalls don't count).
What next, for the RAM itself? I don't think there's much that value in further doublings...either of capacity, or soon, of speed. What I'm convinced we're going to start seeing is some capacity for distributed computation in the RAM logic itself -- load in a couple hundred meg in one bank, a couple hundred meg in another, and XOR them together _in RAM_. It'd just be another type of read -- a "computational read". Some work's been done on this, though apparently there's massive issues integrating logic into what's some very dumb, very dense circuitry. But the logic's already done to some degree; ECC verifiers need to include adders for parity checking.
My guess...we'll probably see it in a 3D Accelerator first.
*yawns* Anyway, just some thoughts to spur discussion. I go sleep now
Yours Truly,
Dan Kaminsky
DoxPara Research
http://www.doxpara.com
Re:Some thoughts on RAM (Score:2, Troll)
I don't recollect a disk vs CPU debate!
RAM prices might have decreased, but implementing databases over RAM need proprietary architectures over and above RAM, which srives the price up. Let me explain. A commodity $200 PC can support 500GB of disk space (4 * 120GB + USB drives). OTOH a mobo supporting even 4GB of RAM could cost over $2000, and it's likely a proprietary design.
"For a very long time, the secret b
Re:Some thoughts on RAM (Score:2, Informative)
Re:Some thoughts on RAM (Score:3, Insightful)
Clearly you haven't used XP much. I've got an XP Video Server hooked up to a TV; it has uptime of around four months right now. Good luck getting a Win9x machine to do that -- 95 literally could never stay up more than 47 days, due to a clock related overflow. They've done ALOT to fix stability, and it's nothing but ignorance to claim otherwise.
It's nice to be able to finally change your IP address without rebooting, too.
95->98 was a huge
Re:Some thoughts on RAM (Score:5, Interesting)
Re:Some thoughts on RAM (Score:3, Interesting)
Your characterization is still correct, in that my transactions from last week cannot disappear once they've been posted, but Google has this problem as well. Google solves it with massive redundancy. I don't know if that would be cost effective for my bank.
Re:Some thoughts on RAM (Score:4, Insightful)
A gig is nothing in the enterprise space. What happens when a terabyte is the unit you allocate between applications or departments, and a petabyte is still big but no longer guaranteed to be the biggest on the block? Gonna walk down to the store and buy a terabyte of RAM, plus a CPU and chipset and OS capable of addressing it? This whole discussion is based on a faulty concept of what "big" is nowadays. For a truly big database, RAM isn't going to cut it. RAM and disk sizes have grown rapidly, but database sizes have (at least) kept pace. That will probably always be the case. If something came along that was even bigger than disks, but even more cumbersome to access, databases would interface to it anyway. General-purpose OS access won't be far behind, either. VM is far from dead; if anything, the change that's needed is to get rid of bogus 2x/4x physical-to-virtual ratios in broken operating systems like Linux and Windows so they can address even more virtual memory.
I worked at a company several years ago (Dolphin) that allowed just this sort of remote memory access at the hardware level. Even then, there were so many issues around consistency, varying latency (this is NUMA where access is *really* non-uniform), and system isolation (it sucks taking a bus fault because something outside your box hiccuped) that the market resisted. InfiniBand HCAs don't even do that; access to remote memory is explicit via a library, not just simple memory accesses. RDMA over Ethernet is even less transparent, and has a host of other problems to solve before it's even where IB is today; it's a step backwards functionally from software DSM, which has been around for at least a decade without overcoming the same sort of acceptance issues mentioned above.
You could start with IRAM [berkeley.edu] at Berkeley. There are links to other projects as well, and some of the papers mention still more that don't seem to be in the links. A lot of what you talk about is possible, and being thought about, but a lot further off than you seem to think.
Re:Some thoughts on RAM (Score:2)
Been there, done that - I once worked on a flight simulator that used a shared memory area accross many machines to distribute data as things progressed.
It's not as cool as it sounds and was eventually ditched for ethernet (not TCP/IP, just raw network messages) for real time information exchange.
In the final analysis however, it's probably faster for high bandwidth applications to build a dedicated high speed
shows MySQL != "real" database (Score:4, Insightful)
Most modern databases also make very effective use of RAM as a cache in order to speed up queries. I don't know about MySQL since I don't use it. My guess, however, is that it does not, since that would eliminate the need for this stupid measure.
As far as reliability, RAM is more vulnerable to transient things like cosmic radiation. ECC memory will take care of most single-bit problems (there are lots of them...), but all it can do for multi-bit failures is determine that yes, your data is screwed.
Also, swapping out a bad hard disk in a RAID configuration is relatively simple and has a recovery process. Suppose your RAM stick fails; what is your recourse? You've permanently lost that data, and systems with hot-swappable RAM are much more costly than ones with similar capabilities for hard drives.
Finally, consider the problem of catastrophic system failure. If the power goes out, your RAM dies but your hard disk is still safe. If it is worse (say your facility burns down) then it is much easier to recover data from the charred remnants of a hard disk than from the charred remnants of a DRAM chip.
The idea of replacing disks with DRAMs has been around for quite a while now. But disks continue to get (a bit) faster and (much) larger. Every time the morons want to replace it they get shot down. More sensible people focus on using the resources available in ways such as caches that make systems faster and more reliable.
Re:shows MySQL != "real" database (Score:5, Insightful)
Seriously, can we all just get over size comparisons? MySQL runs a lot of very useful databases from my dinky little statistics systems that are less than 10MB to giant multi-TB DBs. When you talk about the latter being RAM-resident, you're usually not talking about ALL of the data, but rather indexes and as much of the indexed columns as possible. In that sense a database can be fully RAM-resident on a 4-16GB machine and still have many more TB on disk.
Re:shows MySQL != "real" database (Score:2)
Re:shows MySQL != "real" database (Score:3, Informative)
a) Of course MySQL has RAM cache. Here is one part of it:
http://www.mysql.com/documentation/mysql/byc h apter
b) More than a bit here or there and your disk data is probably toast also. Where do you think the data on the disk is computed from?
c) There are hot-swap/raid type RAM motherboards available also. But that's not really the point.
I run a master MySQL/innodb database from disks and I replicate to 20 slaves ea
Re:shows MySQL != "real" database (Score:3, Informative)
It is fairly clear you are the one who is a bit confused here. Because, typically, disk is the primary data storage mechanism and main memory capacity is less than the total size of a database the enterprise DBMS vendors (this does NOT include MySQL) have what is commonly referred to as a 'data cache' (vendors may call it something else) which stores data pages in main memory (there are other caches for other data structures, but we're only concern
Niche applications (Score:3, Insightful)
This means that in any situation where data is unchanging except for periodic updates this could be a good idea.
ebay uses solid-state ram drives for performance (Score:2, Interesting)
The basic idea: use solid state ram drives (with separate power supply) for your busy tablespaces and your redo logs.
This leverages 'cheap ram' technology with existing (and proven and scalable) db architecture.
For ebay, for example, they might store 'active items' in 'ram-drive-backed' tablespace and 'old items' in the 'hard-drive-backed tablespace'.
These solid-state drives are expensive, but additional Oracle licenses (or moving from 'standard' to 'enterpr
This is nothing new (Score:2, Informative)
Importance of ACID Properties in a DB System... (Score:4, Insightful)
Re:Importance of ACID Properties in a DB System... (Score:5, Insightful)
From the article:
Umm, no. There is no difference in the ACID algorithm whether the database is stored in memory or on disk. The only thing that is easier to do in memory is to fake it, because for low levels of concurrency you can serialize everything without anyone noticing. But that strategy will collapse under load. Far better to do it properly the first time. Yeah, it slows you don't with a single user, but when you have tens or hundreds of users connected, it'll still work.
Actually, the algorithms are the bottleneck on disk too. Monty would know this if he had a query optimizer like the one in Oracle (or had even looked at an explain plan).
Nonsense - you need ACID if there is any conceivable case in which two users might want to access the same data at the same time, or if there is any conceivable case that a write could fail, or if you want to support commit/rollback at all. In other words, if you're running a database and not a simple SQL interface to flat files. Hell, there's an ODBC driver for CSV that does all that MySQL does.
Already in Use (Score:4, Informative)
There are some cool ideas there. They use two copies on disk for backup in case of system failure. Because of this they don't have to do page-latching.
In some configurations, though, this is irrelevant, because write transactions lock the whole database! Because they know all transactions will be extremely short, this is faster than locking at page or row level.
Just wait for a crash .. (Score:3, Informative)
A recent hardware write up I read from HP / Compaq has ram partitioning / raid'ing on some of the higher end x86 servers
Another little burb was that with ram
I know memory can develop stuck bits without any warning
Re:Just wait for a crash .. (Score:2)
Another little burb was that with ram .. as the number of individual ram components increases the risk of a single bit non ecc correctable fault scales up accordingly .. such that with 8 gig + arrays the chance of uncorrectable error approches 50% per time interval
So what? Most high-end systems scrub the ram every so often, correcting ECC faults as they go. Hell, some of the Opteron chipsets do this (go AMD!).
Praedictus (Score:2)
ACID? (Score:5, Interesting)
I'm confused. I actually haven't used MySQL much, and someone else can clarify its current ACID compliance. My application involves multiuser financial transactions. When making my DB selection a couple of years ago, at that time it was claimed that MySQL had some ACID deficiencies that made me nervous. I settled on PostgreSQL, which I'm very happy with.
But there's a lot more to ACID than just keeping RAM and disk in sync, and I don't see how RAM would make ACID that much easier, and certainly not "almost trivial". You still have all the transactional semaphores, record locking, potential deadlocks, rollbacks, etc. to worry about. In fact I don't see why you wouldn't just have the RAM pretend to be a disk and be done with it, since the disk version already has stable software. Then, if it is important to increase performance further, RAM-specific code optimization could be done over time, but slowly and carefully.
I'm sorry - I really don't want to get into a religious war here, but the interview didn't do much to bolster my confidence in MySQL for mission-critical financial stuff. Educate me.
Yer education here ... (Score:3, Interesting)
Cancel that financial adjective. You're thinking too narrowly.
Its called a flat file and mmap... (Score:3, Interesting)
Yet more proof of MySQL's irrelevance (Score:2, Funny)
Here's a tip kids, when you stop playing around with the toy databases, give us a call and we'll show you why you still need hard drives.
DIV (Score:3, Interesting)
Is there a similiar kind of instruction on Intel? It's probably too late though - indexed arrays have become less useful since associative array patterns have become better defined. A hardware implementation (RAM) of JDO would be interesting.
Re:DIV (Score:2, Interesting)
Best of both worlds (Score:2)
Plus with new standards like fiber channel and varies SCSI you wouldn't lose much if any speed.
Re:Best of both worlds (Score:2)
What Monty probably meant (Score:5, Interesting)
For example, disks have incredibly slow seek times and much better bandwidth; therefore it's far cheaper to write things to disk in big chunks. The purpose of write-ahead logging (or "redo logging") is to mitigate the performance impact of slow seek times by blasting all the transactions to disk at once, in the redo log, thereby avoiding the slow seeks that would be required by putting each transaction in its proper place. Putting the transaction data in its proper place is deferred until after the load has died down somewhat. This could be seen as exchanging seek times for bandwidth.
This redo log mechanism would be unnecessary for ram-persistent databases. It's a significant source of complexity that would be obviated by the removal of disks. And that's just one example of complexity required to get adequate performance from disk, a medium that has disastrously slow seek times.
New Sparc? (Score:2)
Hopefully this will be a solved problem soon.
Replication of a database (Score:2)
As long as the DB is replicated onto a slower HD based DB. This would have other advantages, IE duplicating a DB to a remote site for disaster recover purposes.
What's at stake (Score:2)
These hundreds of disks often have very little on them. The purpose of having lots of disks isn't for more storage, but for more drive heads, because lots of heads can be seeking in paralle
Re:What's at stake (Score:3, Interesting)
Re:What's at stake (Score:2, Informative)
Then again, bank dataservers are presumably distributed clusters, and taking out all of them at once is pretty damn hard. Guaranteeing power isn't all that hard
Re:What's at stake (Score:2)
No you won't. There's a difference between a transaction not happening, and a transaction being lost - a lost transaction is one that happened, and application and the user think it happened, but in fact the data was never stored. That's the difference between depositing $100 and it never appears in your bank account, and turning up to the bank with $100 but walking away because the bank was closed.
If a database does proper ACID tr
Google does that already (Score:3, Interesting)
I'd much rather see replication and clustering (Score:3, Insightful)
What MySQL and PostgreSQL really lack is the ability to replicate on-the-fly and to support running on clusters for *real* failover and fault tolerance.
For Postgres, this means multiple 'postmaster' processes being able to access the same database concurrently, and probably something similar for MySQL.
Being able to run a database on an OpenMOSIX cluster, for example, would make it massively scalable, and being able to run multiple independent machines with an existing HA (High Availability) monitoring system would provide a truly fault-tolerant database.
There are of course major technical difficulties involved in making databases work this way, but an Open Source DB that can compete with Oracle's 'Unbreakable' claims would be a huge shot in the arm for OSS in the business world.
Power failures anyone? (Score:2)
With a dynamic RAM system (DRAM also isn't all that reliable...SRAM is better, and SRAM is very expensive) you are highly vulnerable to this.
I suppose you could implement a kind of write-back system to the disk where you pile up things in some kind of buffer, but under heavy load, you're going to overwhelm it. Or at the very least cause the thrashing that this suppose
DBD::RAM (Score:3, Informative)
Non volatility and so forth (Score:5, Insightful)
First, as other have said, a properly designed RAM subsystem can be battery backed up. In terms of getting the data out, loss of power to the RAM is no more catastrophic than loss of power to the CPU, the router, the computer running the middleware, or whatever. Because RAM is a purely semiconductor approach, any battery backup system can be simple and reliable.
In fact, it should not be too difficult to design a system which, in the event of power fail, dumps data to backup disk drives. To get to that state, the main system has already failed to do a clean shutdown, so this is a last resort issue.
The next thing is error detection and correction. It's true that single bit ECC is limited, but it also takes only limited resources (7 additional bits for a 32-bit subsystem, 8 for 64). Memory subsystems could have extra columns so that if bit errors start to multiply in one column, it can be switched out for a new one. Just as with any error detection and correction strategy, single bit detection in columns can be combined with further error correction down rows, using dedicated hardware to encode and decode on the fly. Just good old basic electronics.
In the worst case, it should be possible to build an extremely reliable memory system for a bit penalty of 50% - no worse than mirroring two hard drives. It won't be quite as fast as writing direct to motherboard RAM, but we don't want to do that anyway (we want to be able to break the link on power fail, save to disk, then later on restore from disk. And we want the subsystem in its own cabinet along with the batteries. No one in their right minds is suggesting having a couple of C cells taped to this thing and held on with croc clips.)
I'd even venture to suggest that most MySQL databases are not in the terabyte range, and that most databases aren't in the gigabyte range even if they are mission critical in SMEs.
Conclusion? As usual we have the people trying to boast "My database is far too big and complicated for MySQL! So MySQL sucks! My database is too (etc.) to run in RAM! So running DBs from RAM sucks!" and ignoring the fact that there are many web databases where transactional integrity is not an issue, and the market for a RAM store for databases in the low Gbyte range might actually be rather substantial.
Needed: much more reliable disks (Score:5, Interesting)
We need archival storage devices that won't lose data unless physically destroyed. We don't have them. Tapes don't hold enough data any more. Disk drives don't have enough shelf life.
DVD-sized optical media in caddies for protection, maybe.
(It's annoying that CDs and DVDs went caddyless. Early CDs drives use caddies to protect the CDs, but for some idiotic reason, the caddies cost about $12 each. We should have had CDs and DVDs in caddies, with the caddy also being the storage box and the retail packaging for prerecorded media. There's no reason caddies have to be expensive. 3.5" floppies, after all, are in caddies.)
He just doesn't realize that... (Score:2, Informative)
The Prevayler Project [prevayler.org] is a RAM-only Java persistence project that works and is so simple not a single bug has been found in the production release.
3000 times faster that MySQL (9000 times faster than Oracle) even with the database in caches entirely in RAM simply because of the JDBC overhead that is eliminated .
The only sticking points I've seen are:
1. Normal PC's boards generally will only take 1GB of RAM. Sure there are th
Ram DBs can be faster (Score:3, Informative)
If you know that your data will always fit into available ram then there are a number of performance optimizations that can be done. I'm not sure about ACID becoming "trivial"; You still need most of the same db components: indexes, lock managers, operation journaling, etc. But many of these could be greatly simplified:
1. Page/Buffer Manager Eliminated. Since no disk IO will be required for the normal running of the db, there will be no need for a page manager. This eliminates complexity such as block prefetch and marking and replacement strategies. In fact, the data will probably not be stored on pages at all. Details such as block checksum, flip flop, log position, page latches etc can all be removed. The values in the rows would be sitting in memory in native language formats rather than packed making retrieval much faster. There would be no need for block chaining.
2. More flexible indexing. Since it is not necessary to store data in pages, traditional B-Trees are not absolutely required. Other index structures like AVL trees would be faster and might allow better concurrency. These trees would also be easier to keep balanced
3. Lock Manager Simplified. Row level locking (and MVC) are still desired features, but keeping the locks all in memory simplifies implementation. Oracle and InnoDB store lock information in the blocks (associated with transaction) to allow update transactions larger than memory.
4. Log manager simplified. You will still need journaling capability for rollback, replication, recovery from backup etc. But the implementation of the log need not be traditional. Any structure that maintains information about transactions and contains causal ordering will do. Techniques such as keeping old versions of rows adjacted to current versions that are unacceptable for disk based databases (ahem, Postgres) could be used.
Although these may seem like small things, they can add up: less code to run is faster code. A company called TimesTen offered a product that they claimed was 10x faster than Oracle using an all memory DB. Generally the corporate world doesn't care to split hairs. They want something that works, and they are willing to throw some money and iron at it. Thats why battery backed ram in the disk controller to buffer writes is probably going to be fine for now.
A last note: modern databases already know to not bother with indexes when a table is sufficiently small.
JJ
Re:MySQL embarasses themselves again (Score:5, Interesting)