Object Prevalence: Get Rid of Your Database?

A reader writes:" Persistence for object-oriented systems is an incredibly cumbersome task to deal with when building many kinds of applications: mapping objects to tables, XML, flat files or use some other non-OO way to represent data destroys encapsulation completely, and is generally slow, both at development and at runtime. The Object Prevalence concept, developed by the Prevayler team, and implemented in Java, C#, Smalltalk, Python, Perl, PHP, Ruby and Delphi, can be a great a solution to this mess. The concept is pretty simple: keep all the objects in RAM and serialize the commands that change those objects, optionally saving the whole system to disk every now and then (late at night, for example). This architecture results in query speeds that many people won't believe until they see for themselves: some benchmarks point out that it's 9000 times faster than a fully-cached-in-RAM Oracle database, for example. Good thing is: they can see it for themselves. Here's an article about it, in case you want to learn more."

Re:RAM ?

No more than any other database. Perhapse you missed the part where they said they would serialize the commands that change the objects. In this context they are talking about saving the commands.

Re:RAM ?

This all depends on what data you're trying to preserve. Some data, such as say a users UI preference changes might be deemed "not that important" and thus you can risk storing these in ram until convenient to commit it somewhere.

Conversely, some data such as a financial transaction really needs to be commited straight away.

But commited means *you must* write it out to non-volatile storage (i.e. a disk) otherwise the transaction may be lost. So (I believe) most DB's write the update out to their transaction log very quickly and deal with updating the DB tables/indexes at a latter stage. Obviously, this all depends on if you need to allow other processes to access this data immediatly or not.

Personally, I don't think this represents anything new (**in true /. fashion, I have not read the article!!**) and I doubt this is faster than a well designed system anyway.

What it might offer however is:
1). A nicer interface for managing object persistence; 'cos it is ugly managing mapping objects to DB columns.
2). A clear guide to help people manage which objects need persisting to disk and which are less important.
But thats about all.

---
I'll now go and read the article - you can catch me later contradicting myself!

Re:RAM ?

It's a good idea because relational databases, XML files, and flat files are not really all that similar to objects. [...]

Sorry, you're wrong. With a little forethought it's pretty simple to map an object model to an RDBMS schema. The trick is to design the database schema first, then build your objects on top of it. Doing it in reverse is a bitch. The reason many OO programmers have problems using databases is that they treat the database as an afterthought instead of as the foundation of the application. Here's the basic method I use:

• Tables map to objects; fields map to the properties of that object on a 1:1 basis
• Object methods map to stored procedures where appropriate

A row in a table is an instance of an object

Foreign-keyed child tables map to collections within the parent object.

using Oracle for storing objects means that you've already got a badly-designed system

You illustrate my point perfectly about putting the cart before the horse. You don't build a database to store your objects -- you build objects to manipulate your database. A badly-designed system is one where the database was tacked on after the object model was complete. Your database schema should be the first thing you write, before you even start thinking about the classes.

Unfortunately, Comp Sci cirriculums are heavy on OOP concepts but pathetically light on database theory, which is why you wind up with otherwise talented programmers who don't understand the basic fundamentals of designing solid client-server applications.

Re:RAM ?

As someone else also posted, applying transactions that have occurred since the last time the db was saved to disk avoids this problem. A small company in WA years ago, Raima, had this transaction log concept implemented nicely to support their network database, dbVista (later called RDM). Basically a transaction log is started for every sequence of updates. All records and pointers are saved in a transaction file first. If any problems or system abends occured the entire sequence would be flushed, avoiding a half-updated sequence of records (for example an invoice is posted but the customer record is not updated). It worked pretty well. The big problem with the RAM scheme is that for very large databases the capacity of the computer or the times required to save to disk are prohibitive.

Re:RAM ?

The client commands are immediately written to a log file for later execution. Even if the power fails, the system may be brought back up and the commands exectuted from the logfile regardless of the power failure. While the commands are written to the log, the command the user has decided to invoke is executed immediately in parallel on the live data in RAM. Pretty ingenious, if you ask me!

Re:RAM ?

Reminds me of something that I heard about year ago- one of the DB players (I think IBM) built a fully OO DB in C. Used to store the relations in RAM.
Blazing fast, and easy as hell to fuck up beyond replair- you could do both a read and a write to the same memory area at the same time, or something like that.

This sounds just as bad.
For example, let's say that we're doing a transaction of a few million dollars. In mid process the power dies and the machine goes dark. Outside of shouting 'redunant this that and the other', what state would the machine be in when it comes back online, were is the money, and could we back out of and rerun the transaction?

Re:RAM ?

Blazing fast, and easy as hell to fuck up beyond replair- you could do both a read and a write to the same memory area at the same time, or something like that.

Well, the system is there to stop you from doing that - just like in the traditional DBMSes. Synchronizing memory access isn't harder than synchronizing disk access, it might even be easier if you decide to serialize all access.

This sounds just as bad.

For example, let's say that we're doing a transaction of a few million dollars. In mid process the power dies and the machine goes dark. Outside of shouting 'redunant this that and the other', what state would the machine be in when it comes back online, were is the money, and could we back out of and rerun the transaction?

It can be implemented exactly like it is in traditional DB systems, after all they handle similar problems pretty well. After the failure, you have:
1) The last full image dump
2) all successful transactions (the DB meaning) serialized in the log, from the last dump to the power failure.
Since your transaction (both DB & business meaning) hasn't been successful, it has not yet been written into the log, so the money stays in the ordering party account. Of course the power failure could have occured just after a transaction has been written to log and before the client software got the message that it was successful, but traditional DBs have this problem too. To sum it all up: the synchronization problems are there, but they are no worse than in traditional DBMSes.

Re:RAM ?

What does 64-bit computing have anything to do with how much RAM one needs

its not how much you need that he's talking about. only with 64 bit computing can one have more than the current limit of RAM (which i believe is 2GB right now). it has to do with the maximum possible number of 32 bit addresses can exist in the RAM. so with a 64 bit processor, you can have enough ram to hold that database all at once time.

Re:RAM ?

To license Oracle with similar features to Prevalent, you would only be looking at a 5 figure pricetag.

Don't forget the price tag for all the extra hardware; since a Prevaylent system is thousands of times faster, you can get by with a lot less hardware. And add in all the programmer time spent dealing with SQL. Oh, what about the DBA's salary?

How well does Prevalent do on 30TB+ datasets?

One doesn't use Prevayler for systems like that. Prevayler makes sense if your data can fit in RAM. If it doesn't, you should do something else.

But note that "something else" doesn't have to mean some SQL thingy. Google has a metric shitload of data, and you can bet they don't keep it in an Oracle server.

gigabytes?

At first, I had a problem understanding object oriented methodology because I kept thinking of objects in terms of a database -- they seemed so much alike. But...

Who uses a database small enough to fit in RAM?

Re:gigabytes?

Who uses a database small enough to fit in RAM?

Not every solution is for every problem. This isn't for huge data warehousing systems. My impression is that this is for smaller databases where there is a lot of interactions with fewer objects.

I have also seen object databases used as the data entry point for huge projects, where the database is then periodicaly dumped into a large relational database for warehousing and reports.

Re:gigabytes?

And that goes for OO as well. Not every database (or a collection of data) needs to be accessed in Object-Oriented way. Most (or should I say all) data I store to small tables would not benefit from being objects.

And how does this differ from storing non-object-oriented data structures in RAM? You'd still need to implement searches, and how do you search an collection of objects without placing them on the relational line.

Re:gigabytes?

Who uses a database small enough to fit in RAM?

I do, but I'll thank my SQL server for doing it for me. Most aggressively cache data and databases - if Database A is used constantly, it'll be kept in RAM, whereas less-frequently Databases will either stay on the hard disk, or certain tables of that database will be put in memory. It lets you make the most of your RAM.

Slashdotted

For a scalability test, it sure fails the Slashdotting Test.

It's about 9000 times slower right now :)

Neat concept...

but it doesn't really provide any compelling reasons NOT to use a database. Besides the fact that their home page, Prevayler.org [prevayler.org] seems to be non-existant I think it's more of just a "neat idea" type thing rather than a compelling reason for any prodcut/project to drop relational DB support.

You can always have a caching system as the author states, but even then what systems use this? The countless PHP/MySQL sites out there seem to perform just fine. This may be desirable for some very strict real time communications systems, but for just about every other form of app, I don't see it.

What are you going to tell your 3rd party integrators? Drop their XML/ODBC report and surf on over to prevayler.org?

Re:Neat concept...

The countless PHP/MySQL sites out there seem to perform just fine.

Object-oriented programming and data persistance is about a lot more than public web sites. Private, corporate data warehouses with terabytes of persisted objects squeeze every bit of processing power available. For example, I used to work on Mastercard's Oracle data warehouse. An average of 14 million Mastercard transactions occur per day. That's 14 million new records to one table each day, with reporting needing hundreds of other related tables to look up other information. To get something of that scale to run efficiently for a client app (internal to the company) costs millions of dollars. Object persistance on a large scale is tough to get right and is far from perfected, and there's a lot more going on that public web site development. Every new idea helps. Consider the article written on IBM's developerWorks. It's readers are mostly corporate developers.

What about existing data ?

Their solution really seems to rock, and may finally be the OO to DB paradigm everyone was waiting for.

That said, I wonder what their position is towards the import of existing data. Many projects would only benefit from the solution if and existing data (usually object-oriented but saved in a roughly flat database as the article points out) can be ported seemlessly to the new environment.

My point is, this solution solves a known problem by introducing a new technology, however this new techno will have to be bent towards the older systems in order to retrieve what was already saved. Same old story : in the database world existing data is paramount.

FInally OO? I think and hope not!

OODBMSes have been thoroughly and handily debunked. For the best opinions on relational database technology, visit these hardcore guys: http://www.dbdebunk.com/ [dbdebunk.com]

The problems with OODBMSes can be summarized so (OTOMHRN - on top of my head right now):
1) Proper relational technology can model OO-hierarchies, but the other way around is unnatural and cumbersome, if not impossible. Proper relational technology is a step up on the ladder in generalization from OO-technology. It's simply a generation or two ahead, while OODBMS is several steps backwards.

2) Proper relational technology is proven concepts from mathematics and logics, while OODBMSes are just a hack to store application data "quick'n dirty". Everything can be modelled as general relations, while OO-technology lacks the fundamentals to model *ANYTHING* and is limited and impeded by having an obligatory and *meaningless* top-to-bottom hierarchy. (You cannot have *meaning* without relations of differing types to other entities.)

3) Proper relational technology allows you to extract, convert and manipulate data in standardized methods (using query languages like SQL), in ways not thought of at the time of design. OODBMSes can only be used properly in the context of the OO-application layer, often relying on runtime data. If you need flexible solutions, you will have to spend extra time programming a specialized solution, instead of having the benefit of a fully relational query language (which unlike SQL, can express almost any problem to be solved).

4) The future is relational. Current RDBMSes do not implement true relational technology, which if they did, nothing else would be needed. The matemathics in the theories behind it would be at the programmers disposal during programming, reducing time and potential errors. Yes, it requires understanding the theory, but wouldn't you like a true DBA to do that anyways?

Don't buy into the hype, look into true relational technology and educating yourself. As for storing everything in RAM, and "saving it for the night", I wouldn't risk to have my bank-account in such a DB. Such solutions are only usable for storing non-volatile data. For non-commercial game-servers, it maybe perfect.

Re:What about existing data ?

Their solution really seems to rock, and may finally be the OO to DB paradigm everyone was waiting for.

Not likely. The REAL problem with OO databases isn't that RDBMs might be more mature or whatever else you might read, it is that the data is almost always more important to companies than the behaviors that operate on that data. For example, if the company has a database of customers, they might want to use that database in dozens of different ways, and they might want to grow it for years, if not decades. The OO-database view tends to look at things too much from the view of one single application of the data and the data gets entangled with code behavior based on that specific application. With a clean RDBMs you can hit the same database from many different applications (assuming the database has a well thought-out schema to begin with)... the data isn't so tightly wound up with a specific bit of application code.

This 'solution' doesn't fix that aspect of OO databases. In fact, it makes it worse. I will grant that it is a neat technology, but I wouldn't expect to see it take over the place of RDBMs systems any more than OO-databases of the past have.

Re:Data integrity?

Aha.

Someone's been reading DBDebunk.com [dbdebunk.com] again.

Yes, data integrity is one of the major considerations here. I'm willing to bet that by the time you implemented the equivalent of constraints, triggers, etc... in a system like this, you would be running no faster than a typical SQL DBMS, and you would have thousands of bugs as you reinvent the wheel. But there are even more considerations than integrity. This is language-specific, or application-specific. What do you do when you need to access your data from another application? Even if it is possible, that means you have to implement all your integrity checks again in that application.

Essentially, what this looks like is just another OO method of heirarchical (or perhaps "multi-valued") data storage. This is nothing new. It will suffer all of the historical problems the industry has had with hierarchical storage (there is a reason the relational data model was invented: the problems IBM had with hierarchical data). For example, what happens to existing data when you need to change your logical schema or business rules? The cost of re-ordering or reformatting _every_ single stored object since the beginning of your application would be ridiculous, and in some cases even impossible. How do you track dependencies? In theory, these kinds of systems will work fine, if your application stays exactly as created, and if the nature of the data doesn't change, and if no other applications are involved. In other words, NOT in the real world.

I have a nick-name for hierarchical data storaqe: "headache-ical".

Two words...

Enterprise JavaBeans.

Here's the definition of an EJB from the http://java.sun.com [sun.com] site.

A component architecture for the development and deployment of object-oriented, distributed, enterprise-level applications. Applications written using the Enterprise JavaBeans architecture are scalable, transactional, and multi-user and secure.

And more specifically, here's the definition of an Entity EJB:

An enterprise bean that represents persistent data maintained in a database. An entity bean can manage its own persistence or it can delegate this function to its container. An entity bean is identified by a primary key. If the container in which an entity bean is hosted crashes, the entity bean, its primary key, and any remote references survive the crash.

Re:Two words...

Entity beans are all about transactions. You've got a transaction context that can propogate over several beans. The EJB container doesn't do this on it's own, however. It uses the ACID properties of the database along with the database's commitment control mechanisms to accomplish the properties you mentioned. Entity beans are usually mapped to tables, and could represent a join in the BMP case. That said I'm not sure if you're saying EJB will benefit from using this as a backend, or that EJB did this first? The latter is false, but the former... I'm not sure this technology will benefit entity beans, but may benefit STATEFUL SESSION beans because they're less RDBMS-centric.

Ever looked at object-oriented databases?

Have you have looked at object-oriented databases? They give you ACID transactions, and also take care of mapping the data into your main memory so that you as a programmer only have to deal with in-memory objects. The leading OODBs are Objectstore (www.exln.com), Versant (www.versant.com) and Poet (www.poet.com).

3 issues I see

First off, I like the concept, but speaking as a former Oracle DBA, I have several issues:

1) You're limited by how much RAM you have on your server, not how much disk space you have

2) If you're making a lot of data changes and have a crash or power outage, I'd imagine that it can take a while to replay the log to get things back to the most recent point in time (you can have the same problem with Oracle, but your checkpoints would be a lot closer together than "once a day")

3) There are millions of people that already know SQL and can write a decent query with it. How does this help them? Never underestimate the power of SQL.

On the other hand, for projects dealing with small amounts of data I can see how implementing this would be far easier than integrating with Mysql, Postgresql or Oracle.

Re:3 issues I see

1) What about Swapping? I know that you would by limited by physical ram (which IS getting cheaper) but couldn't you also get a really large virutal memory space and utilize that?

2) You can probably set up your own checkpoints to be more than once a day.

3) I agree. Lack of SQL would cause people to.... GASP.... learn a new system. SQL is very cool. And I admit that I have a system I am thinking of porting away from JDBC and into Prevalence just to see how it goes (No, it isn't mission critical) and one of the first things I realized is that I would have to design a new method of querying. But you know what... That can lead to new thinking and more powerful software in the future.

Buggy whips

3) There are millions of people that already know SQL and can write a decent query with it. How does this help them? Never underestimate the power of SQL.

There are millions of people that already know how saddle and ride a horse. How do these new fangled automobile help them? Never underestimate the power of a horse.

While I agree with your other points... number 3 is never a reason to keep from embracing something new. People are suprisingly trainable.

3 More Issues for the Do-It-Yourself Database

Personally, I still think it sounds a lot easier to just map objects to a database.

4) Concurrency - If you haven't implemented locks for an object model, then you haven't lived. Seriously, I can see a lot of people screwing this up with deadlocks galore. Locking up concurrent systems can be a nightmare.

5) Ad Hoc Support - Goodbye Crystal Reports, Goodbye English Query, Goodbye ANY Ad Hoc query support, because if you need anything different, you're going to have to write a lot more code to enumerate throughout your objects. Have fun.

6) Indexing - I hope you have a good B-Tree library and are familiar with Indexing/Searching algorithms when implementing HARDCODED indexing. Oh yeah, have fun rewriting all of your query procedures when you decide to change your hardcoded indexing.

Nothing says flexible like HARDCODING! Yay!

In all seriousness, this is a bad idea for 99% of projects out there. It's inflexible, unscalable, severely error prone, and timely to implement.

(sarcasm) All this just to avoid the "cumbersome" process of mapping objects to tables?

Seriously people, it's not that hard (3 magnitudes easier than this) and there are a lot of tools that help doing it.

If you're REALLY hung up on not using a relational database, try an Object Database, XML Database, or an Associative Model Database.

Interfacing

This may be a great way to snapshot the state of a Java application but how on earth would you query anything out of it with a non-Java/non-OO language?

A SOAP interface could go some ways towards accomplishing this but what about the traditional ACID properties of a DBMS? Durability is obviously guaranteed... Consistency? That would depend on programmers following the practices... Atomicity? Not sure about that one. For simple commands it seems to work. What about compound commands? If no rollback occurs how can I assert that I changed both objects not just one? Isolation? Not sura about this one either.

Something about this doesn't sit right with me

I think this would work well for most web-server DB backends as the data isn't changing on the fly that much. But what about even /. where the content of a discussion thread is changing possibly several times a second (with new posts and mods)? I'd think then you'd want to use the strong atomic operators of the DB to pull directly from the tables instead of relying on serial operators to try and refresh.

Since the benchmark page was slashdotted I might be speaking out of my ass. But I never trust "9000 times faster!". It sounds too "2 extra inches to your penis, guaranteed!"

It's not a simple question of speed

The advantage of putting data into a database isn't just speed! Just think about sharing data between application, between many computers, exporting data into another format, or simply making a query to change some values! You simply don't want to write code that will change value of data with some specific conditions : you prefer make a single query that any database manager or simply a sql newby could write, not just the 2-3 programmers that have done the work on that code some years ago! You also sometime need to visualise data, make reports, sort data. You simply don't want to code that. I think most serious database can also put data in RAM if you have enough, and is able to do some commit/rollback when it's necessary. So your point that RAM data with serialize in-out is ok, as long as you absolutly need 100% speed, don't need to do complex query on your data and is in used only on one computer.

Blazing fast

This architecture results in query speeds that many people won't believe until they see for themselves: some benchmarks point out that it's 9000 times faster than a fully-cached-in-RAM Oracle database, for example. Good thing is: they can see it for themselves.

Yes, I've seen it. The page on www.prevayler.org only took about 30 seconds to load. Does that mean that a fully-cached-in-RAM Oracle database would spend 75 hours loading that page...?

no queries

Queries are run against pure Java language objects, giving developers all the flexibility of the Collections API and other APIs, such as the Jakarta Commons Collections and Jutil.org.

In other words, "it doesn't have queries". What real project doesn't (eventually) need queries? And even if writing your queries "by hand" in Java is good enough for now, what real project doesn't eventually need indices, transactions, or other features of a real database system?

Re:no queries

In other words, "it doesn't have queries". What real project doesn't (eventually) need queries? And even if writing your queries "by hand" in Java is good enough for now, what real project doesn't eventually need indices, transactions, or other features of a real database system?

Indeed. It looks like a high-level, language-neutral API for traversing linked lists of structs. Yes, you can rip through such a structure far faster than Oracle can process a relational table, but they are two different solutions to two different problems. I wouldn't use an RDBMS for storing vertex data for a scene rendering application, and I wouldn't use an in-memory linked list for storing bank transactions!

Get best of both worlds...

If you need performance for persistant data, this "new" system doesn't seem to be much different at all to what you can do today. Using JDO (Java Data Objects) with a file-system backend would be about identical, though easier to use and have more features.

Of course, you can always write your own persistance layer. I've done this a few times - very easy in Java. Map a row in the DB to an object, and cache the object in memory. If need to fetch that data again, check the cache first. When doing a write, write to the DB and update/flush your cache as necessary.

That's just the basics - what's most optimal depends on how your data is accessed and changed (and also your programming language and capability as a programmer). Java has nice really nice stuff for caching built-in, like SoftReference wrapper objects, and of course threading and shared memory that you can use in production.

I'm currently working on a super optimised threaded message board system. Almost all pages (data fetch/change + HTML generation) complete in about 0.001s.

Umm what about multiple servers?

Reading through the article it seems to lack a rather small but important item - multiple systems interacting read/write with the same database. This is not a very robust or scalable way of doing things. I wonder how this stacks up to one of the normal ways of improving performance by have one read/write database with lots of read only repicas.

Sourceforge Link

http://sourceforge.net/projects/prevayler [sourceforge.net]

OO databases are an evolutionary step...backward

In 1999, I worked for a company that used an OO database (ObjectStore) to develop an e-commerce shopping portal. It was a disaster.

OO advocates point to extremely fast (extremely special-case, in practice) queries, and natural persistent object mapping as reasons to why OO is superior.

However, this is very misleading.

Some of the MAJOR problems we ran into in using ObjectStore were:

• It is very difficult to "see" an OO database. By nature, the data isn't tabular. It's a persistent object heap. There's no "SELECT * FROM USERS". So tracking down data-related problems involves exporting data to an XML file and sifting through it.
• Reporting tools don't exist for OODB. Try hooking up Crystal or another reporting tool to this. You end up writing every report from scratch.
• DB Performance when querying outside the normal object hierarchy (aggregate queries grouping on object attributes, etc.) is orders of magnitude SLOWER on an OODB! This was a huge problem when allowing users to do a product search on an e-commerce portal.
• 32-bit memory limited our max customer size dramatically.

When developers first consider OO databases, their first assumption is that OODBMS is to RDBMS as OOP is to Procedural Programming. This is a FALSE analogy! Migrating to OODBMS offers precious little to support better software design while introducing significant maintenance and design issues that should be considered prior to using this technology.

Unless I had a product that had an extremely specialized use case that matched OODB strengths, I would NEVER develop on this kind of platform again.

OODB are very different from RDBMS

and like in every object system, it is very important you get the design (and objects) right before you start coding.

If you are thinking of accessing your objects like you are doing with SQL, then you haven't understood how OODB work. As for accessing your objects and doing your queries, there are tools (like Inspector for ObjectStore) than enable you to do just that.

In term of performance, Oracle and co are nowhere near what you can reach with ObjectStore, provided you designed your application well.
The 2 main problems with OODB, are:
- schema evolution
- reporting
But these can easily be solved by a good design of your application.

OODB is a skill that needs time mastering. After 4 years, seeing ObjectStore application from various companies, I can tell the difference between the ones where people knew what they were doing, and those from people who didn't have a clue...

Re:OODB are very different from RDBMS

The 2 main problems with OODB, are:
- schema evolution
- reporting
But these can easily be solved by a good design of your application.

In other words, OODB technology is doomed.

Re:OO databases are an evolutionary step...backwar

Although you certainly have a point, there are some remarks I have to make here:

There's no "SELECT * FROM USERS".

That's just like saying Latin is a bad language because it does not have equivalents for 'the', 'le/la', 'de/het', 'der/die/das', whatever. An rdbms is *fundamentally* different from an oodbms

DB Performance when querying outside the normal object hierarchy (...) is orders of magnitude SLOWER on an OODB!

That's right: you are trying to use a oodbms as a rdbms. Ever tried to drive a car like you ride a bicycle?

Oodbms are relatively new, and they have their 'problems', just like rdbms-es have theirs. But the biggest problems arise when one approaches an oodbms like one would an rdbms. Just like you run into problems using an oo language when you have only used a proc. language

Speed is not the only factor

The first problem I see with this method is the lack of a powerful and flexible querying method. One of the most powerful features of SQL databases is their capability for searching. No where in the article did I see anthing about advanced querying of the objects. Even if there is, I'm sure its no where near as fast as a MySQL or Oracle. The author states that it is several orders of magnitude faster, but I bet it is this much faster only on fetch routines where you already know what object you are looking for.

Here's the issue they are trying to solve: mapping object to records. That's it. Now the problem with removing the records / database is you lose all of the searching power that is inherit in relational databases. The author states that the codebase is 350 lines of code. How can any complex search engine be implemented in 350 lines of code that also covers the persistance?

Memory is CHEAP?

how much is 25Gig Hard disk as opposed to 25gig RAM? is that you call cheap? :)

This concept is not new

In fact, this concept actually predates SQL-based databases! The first one I am aware of is MUMPS (Massachusetts General Hospital Utility Multi-Programming System) which goes back to 1966. One company that continues this legacy is Sanchez [sanchez-gtm.com]. Another commercial version is Caché [e-dbms.com]. This makes sense, really - the most obvious solution to serializing an object is to store all properties of a single object together (the OO solution), rather than store a single property of all objects togther (the RDBMS solution)

Old News: Main Memory Databases

Main Memory Databases have been researched for nearly 10 years now and there are a number of commercial products. For details you can check out:
TimesTen [timesten.com]
Polyhedra [ployhedra.com]
DataBlitz [bell-labs.com]

etc..
The idea it to have enough RAM to be able to store all the database in memory. This gives higher performance than a fully cached Oracle for two primary reasons:
- there is no buffer manager so data can be directly accessed.
- the index structures use smart pointers to access the data in memory.

Typically the data is mapped using mmap or shared memory. Each application can have the databae directly mapped into its memory space.
For providing persistence, typically main memory databases provide transaction logging and checkingpoint to be able to recover the data. Various techniques have been developed to be able to do this without affecting performance.

The Electric Database ACID Test

Things I want in a persistent datastore:
- Atomicity of transactions (commit/rollback),
- Consistency in the enforcement of my data integrity rules,
- Isolation of each transaction from other competing transactions (locking)
- Durable storage that can survive a crash without losing transactions (e.g., journaling)

My experience with RAM-centeric disk-backed object storage is that you, the developer, often have to implement the ACID fetures yourself, from scratch. And from-scratch implementations of complex data-integrity mechanisms tend to be time-consuming to develop and test and often take much, much longer than you think to "get right".

Call me old-fashioned, but I really like using data storage (database) engines that pass the ACID test and have already been debugged and debugged and debugged and debugged and debugged.

-Mark

Not a "database" but a persistence mechanism

Some people seem to be missing the point: this is not a "database" it is a persistence mechanism. What they are saying is that persisting objects is difficult (er, tend to disagree but I'll bite) and so they are solving this. Whether a RDBMS offers better searching is completely irrelevent as this, in their architecture, is handled by the application.

What they seemed to gloss over is that you need to take snapshots of the actual data. If you didn't you'd have to keep every single "log" in order to safely playback the actions and know you have the same data in the same state. Loose one log, say the very first one, and you're pretty much screwed.

Problems with Object databases

This is not a new idea. There are all sorts of object databases out there. (Versant springs to mind).

The main problems I see with object databases:

1) SQL is incredibly powerful. You give up *a lot* of power when you go from sql semantics to object semantics. Sub-selects, group bys and optimized stored procedures, to name just a few things. All the object language query constructs I've seen fall far short of these. (As a side note, most O/R tools make a hash of it as well.)

2) You immedately make a massive reduction in the number of database administrators who will be willing and/or capable of helping you out in your project.

3) Scaling is always a question. With oracle, it just isn't.

4) Backup, redundancy, monitoring, management, etc. Most mature relational databases have very good tools for doing these infrastructure activities. Developers often forget about banal things like this, but they are crucial for the long term health of IT systems.

Don't get me wrong. Every time I construct some nasty query and go through the mind-numbing process of moving the results into an object, I think to myself "There has to be a better way!", but I've looked at the O/R tools and the object database out there and, sadly, I don't feel they are worth the trade off.

Just my opinion,
prat

Interoperability, Scalability

The persistent store is quite language-specific. It doesn't allow for a Python application to access a Java store, for instance. It also doesn't seem to allow concurrent access to data, which would require significantly more than 350 lines of code.

Both of these issues make this solution unusable in an enterprise environment. The RAM size issue has already been mentioned by others and is another very real limitation.

In general, object caching mechanisms are not terribily difficult to create. This generic solution proves the point by only requiring 350 lines of Java code.

I am sure that there is something worthy in this project, I just cannot see it used for anything other than very small-scale development efforts.

Database System vs Database Management System

This OO scheme is a database system, but it leaves out much of the management element. (1) Things like changing the database structure without bringing the whole company down probably won't work. (2) You lose all the enforcement of the rules of relational integrity that an RDBMS gives you right out of the box. (3) And you lose Crystal Reports. (1) and (2) kill it technically in many situations, and (3) kills it management-wise.

Gadfly, a Python package, gives you an in-memory DB and SQL. If you want to trade SQL for extra speed and do more programming, you can run the ISAM-like engines of Btrieve or Berkeley DB without the SQL layer on top. We have SQL RDBMS's because the conventional wisdom is that such a trade is not a good idea.

BS

Sorry this just won't cut it in most enterprise systems.

1) Doesn't scale. Most enterprise databases don't fit in RAM. Data volumes grow with the capacities of hard disks which outpace RAM. If your database fits in memory now and you use this architecture, what do you do when it grows larger than your RAM capacity? You fire the guy that proposed this and switch to an RDBMS.

2) Performance claims are BS. Good databases already serialize net changes to redo logs via a sort of binary diff of the data block. Redo logs are usually the limiting factor on transaction throughput, since they require IO to disk. Serializing the actual commands is more inefficient than using a data block diff. You simply cannot minimize the space any better than an RDBMS does, therefore you cannot minimize the IO for this serialization any better, and therefore you cannot do it faster without sacrificing ACIDity. If your performance is too good to be true, then you gave up an essentail feature of the RDBMS.

3) Consistancy. If there is only one object in memory for each record, then you'll be writing a tremendous amount of custom thread-safety code and even then, either A) writers block readers and readers block writers or B) read consistacny isn't guaranteed. Either is usually unacceptable. One alternative is to clone objects at every write (sounds slow and horribly inefficient). Of course, this too has to be serialized, or you don't have ACIDity. If you are serializing these, then you aren't really different than an RDBMS which uses rollback/undo, except you are wasting disk IO and are slower.

4) Reliability. A hardware failure, software hang/crash, or system administration mistake would force recovery from the last full backup. Replaying a full day's transactions could take hours. Sure you could be continually making a disk image, except for read consistancy issues like above. Its not clear what you do even for a daily backup. Are all sessions simply blocked during backup? Ouch.

Every few years object fanatics try to come up with some way to get rid of RDMBS's. The methods invariably rely on sacrificing some of the core capabilities of the RDBMS: data integrity, performance, consistency, ACID, reliability etc... These "innovations" are really only of interest to OO fanatics. In the real world, OO gets sacrificed way before RDBMS's do. This is not going to change.

OO is a tool that is good for writing maintainable code. It is not good for performance critical uses like OS's, device drivers, and real time systems. It is not good for data intensive systems. These things are not likely to change. If all you can accept is OO, then you are a niche player.

This Won't Replace A Database

As has been mentioned, it fails the ACI portion of ACID (it's not Atomic - all or nothing, not Consistent - data is left in a consistent state, doesn't provide Isolation - you appear to be the only transaction running; other processes don't affect your data in mid-transaction). Passes Durable, I suppose.

I've read a few posts that say that the performance claims (vs a relational database) are not true. I think this will be much faster than a database. This is an in-memory cache. It will be very fast. Our Oracle databases have a cache-hit ratio of 98 and 99+ percent, but will be slower. Why?

First, databases (especially Oracle) do alot of stuff behind the scenes, logging all sorts of stuff from a user connecting to the SQL being run.

Second, this sort of thing offers nearly direct access to the data. SQL usually needs to be parsed before it is executed. The database needs to come up with the optimal query plan before it actually executes the statement. A database offers different ways of joining data, and accessing data. Find me all managers that make more than $50,000 per year and have a last name that start with K. You will have to decide the best way to get the data yourself. A database will do all the work for you.

This is a great, idea, though for a middle-tier cache. Say you want to do some fast searching on a small amount of data. You can use this in the middle tier to save yourself the trip to the database.

A good object oriented database that has not been mentioned yet is Matisse [fresher.com]

WOW! After twenty years...

somebody has figured out that things in memory are faster than disk...

After twenty years, we finally get to...

the in-memory database!

Oh wait, didn't my Atari ST have that?

OK

How is this any different from using a journaling filesystem and mmap?

MOO

MOO has been doing this very thing for years, and it actually draws a lot of criticism for it. Keeping a persistant image of objects around and making checkpoints at determined intervals doesn't really seem to be that big of a deal, though it is cool to have bindings to all of those languages. But really, what's the big deal? (an honest question, not a flame)

race conditions?

Is anyone else bothered by the complete lack of the synchronized keyword in his example code? So the ChangeUser Command can apparently be in between these 2 lines:

usersMap.remove(user.getLogin());
usersMap.put(user.getLogin(), user);

Meanwhile someone else can run an AddUser Command with the same username. Guess what happens when ChangeUser gets to that 2nd line?

Maybe when this radical new concept in databases can be presented in a way that avoids race conditions I'll pay a little more attention...

Congratulations!

You've invented an Object-Oriented database! Wowee zowie! Wait, what's that? You say this is nothing new? Well, you're [sleepycat.com] right [odbmsfacts.com]. Of course it's faster than an Oracle database stored in RAM. Oracle is not designed for the purpose of storing objects. It's a relational database, which is something else entirely.

The bad old days

Why really did DBMS come about? It was not because of a need for secure transactions or to store a lot of data, although obviously those are necessary qualities of a DBMS.

Before dbms applications stored their data in very efficient data stores designed just for that application but were worthless for anything else and hard to upgrade or extend without breaking or rewriting the existing application.

DBMS were developed so that data could be stored in an application independent store that could be used and extended for new applications without breaking everything that went before.

DBMS were never designed to be more efficient than the application specific data stores that they replaced, so that somebody saying that they can build a custom data store just for a particular application that is faster is missing the point entirely.