Evolutionary Database Design 171
Andre Mermegas writes "Check out this article by everybody's favorite object mentor Martin Fowler on database design. Be sure to take a peek at his wonderful books as well."
This discussion has been archived.
No new comments can be posted.
Evolutionary Database Design
Related Links Top of the: day, week, month.
Ocean: A body of water occupying about two-thirds of a world made for man -- who has no gills. -- Ambrose Bierce
relational databases, woo hoo (Score:5, Insightful)
I think it's nice that people are starting to get interested in relational databases again. They really are the backbone of information systems in business, despite what the industry rags will have you believe.
The "hype" of object-oriented and XML-driven "databases", although aesthetically prettier, have adverse effects on performance and design. Programmers get lazy, applications become sloppy and performance goes into the toilet.
Re:relational databases, woo hoo (Score:1)
I agree with your observations except for XML-driven "databases", sloppy applications, toilets and relational databases in general.
Other than that I feel you present valid points.
Re:relational databases, woo hoo (Score:2)
"XML-driven" databases suck, however, so I'll give you that.
Re:relational databases, woo hoo (Score:2)
Only for a specific, hand-tuned application, and if the database never changes. For multiple applications, automatic optimising and changing databases, you need a relational system.
But obviously when you said relational you were thinking SQL, not relational.
Re:relational databases, woo hoo (Score:2)
Re:relational databases, woo hoo (Score:2)
The relational model is a powerful, high-level general theory of data, while OO data management is just a mess on all accounts. It can't even decide if objects are values or variables. I dare anyone show something OO database systems can do better than RDBMSs.
Re:relational databases, woo hoo (Score:3, Informative)
I agree that relational databases are the best solution for most problems -- that's why they're the backbones of most apps these days.
That said, there are some cases where they fall down. One example that I'm working on right now is organizing a million or so smallish documents. The relational design to store the documents with the same degree of specificity as the XML format they are in is ridiculously complicated. But storing them in an XML database (we're using Xindice, but have looked at Tamino, and a few others) is a lot simpler.
Another downside of relational dbs is that it's generally pretty difficult to change your schema. A lot of XML databases, on the other hand, can be configured to not enforce a schema at all. So if you're working on a problem that requires experimentation on the basic schema, it can be a lot easier to use an XML database (or even just files on disk) instead.
-Esme
Re:relational databases, woo hoo (Score:2)
More correctly, they're the least worst solution for most problems. The reasons why they're the backbones of most apps these days are far more likely to be some combination of:
Re:relational databases, woo hoo (Score:2)
They aren't. SQL is not relational.
No one does. Even universities tend to teach SQL instead of the relational model.
You can use Oracle as a back-end for Dataphor Alphora, and thus have a truly relational application without endangering your data.
Re:relational databases, woo hoo (Score:2)
SQL not being relational, the only apps using relational databases nowadays are the ones based on either QUEL (are there any yet?), IBM BS12 (probably none?) and Dataphor Alphora (those by SoftWise, and some few inhouse ones up to now).
All the cases you mention are specific failures of SQL, not of the relational model.
Re:relational databases, woo hoo (Score:2)
This is the kind of argument that gives IT folks a bad rep. Everybody knows that the popular database products that use SQL or some subset of it are commonly referred to as relational databases, regardless of whether they perfectly implement the relational model. The fact is that you can take example queries from Date's Introduction to Database Systems and run them with almost no change on any number of SQL databases. These systems store data in basically the way described and follow most of the guidelines for being relational. Of course, they also let you do all kinds of other wacky stuff, too. But that doesn't alter the fact that the features that are common across database products are the core of a relational system.
Actually, no, breaking up documents into many discrete units is a requirement of the normalization required by relational theory. It is this fragmentation which makes relational databases awkward for storing documents where retrieving the entire document is usually the desired operation.
Now, if you want to say that these are implementation problems, then I'd agree with you. A relational database could assemble a schema on the fly (just like XML databases parse their contents on the fly when processing XPath or XQuery queries). But they don't, and it's a well-known property of relational databases that they don't.
-Esme
Re:relational databases, woo hoo (Score:2)
So we should call relational databases something else, like X, and leave the relational name for SQL?
Now, what happens when someone comes calling his product X even if it is not conforming to the relational model Y?
Perhaps we should not allow vendors to implement their Newspeak in the first place.
These examples were never intended to be a compliance test, and many are in SQL anyway for convenience.
Not if document, or a reasonable subdivision of it (section?) is a data type.
Re:relational databases, woo hoo (Score:2)
Whether you agree or like it, the term relational database is used by virtually everyone to mean databases that use SQL. That's the way language works: what people mean when they say something is what the word means, period.
These databases are implementations of the relational model. Not perfect ones, not complete ones. They also have a lot of other junk thrown in to address other needs. If you want to talk about a database that does a better job of implementing the relational model, call it a "True Relational" or "Pure Relational" or something like that. Or just call it a RDBMS and say that Oracle has a crappy impl.
Software vendors are a big source of terminology getting blurred, but I don't think that's what happened here. I think a couple of vendors implemented the relational model, and people started calling them relational databases. Over time the databases added new stuff, and proprietary extensions, and stopped being "just" relational databases. The recent fad of adding XML translation to RDBMS is just the most recent example of this.
In my particular application, the data is mostly semi-structured content (names, dates, titles of paintings, names of countries, etc.) that aren't formatted consistently (b/c they come from several different source institutions, some of which have local heterogenity as well). There are about 15 top-level categories, with most categories having several all-optional, all-repeatable subcategories, with a different type of data for each sub-category. So a traditional relational design would quickly wind up with dozens of tables and some pretty nasty joins to get the whole doc back.
My solution was to use one system for storage (a native XML database - Xindice) and different system for querying (a fulltext search engine - Lucene). Most of the XML-enabled RDBMS could do basically the same thing, but it seemed like a bad fit to use a RDBMS with a totally non-relational approach.
-Esme
Re:relational databases, woo hoo (Score:2)
Even when clearly it is vendor Newspeak? Scary your definition of language.
They aren't, because they don't comply with the basic principles of the relational model. If you disagree, either you don't know the relational model principles, or you don't know SQL, or you're dishonest.
I understand your point, but you forget that then I would have to explain what's "True Relational" and why SQL isn't it. So there is more trouble, not less.
You should not "think" if you do not know History. What happened was that SQL was created against Codd's orientation, so it was never relational in the first place. Codd even has quitted IBM because of that.
Again, a relational system (like Dataphor) makes it simple to either decompose the tags and contents in a recursive structure, or to define suitable domains. There is nothing non-relational to that. The problem again is SQL not being a RDBMS.
Re:relational databases, woo hoo (Score:2)
First, about language. If a vendor comes out with some terminology that's completely stupid, they'll usually get mocked or ignored, and they'll drop it. On the other hand, if they come out with something that is a mutation of what has come before, people might adopt it. Language changes like this all the time, and there's nothing anyone can do about it. The fact that most people go along with the terminology is a strong sign that they agree with it.
For the theory, my understanding is that the heart of relational theory is organizing data as n-tuples in a system that abstracts data access away from the details of data storage. I'm not dishonest, I do know SQL, and I'm quite sure that relational databases do this. Even if Codd doesn't like the implementation (or consider it to be faithful). Personally, I'm not terribly interested in what Codd thinks of an implementation, since the people who originate theories are notoriously inclined to be poor judges of their implementation, extension and interpretation.
And you shouldn't be pedantic if you honestly want have a discussion.
-Esme
Re:relational databases, woo hoo (Score:2)
So they can implement Newspeak as they want, provided they do it gradually. Remember telling the Party that before 1.984.
Yet this might be confusing and self-defeating, as it is in this case.
Relational databases are supposed to, but SQL doesn't: it has no proper distinction of physical, logical and user schemas, and updateability of derived relations is arbitrarily limited. Also by way of pointers, row ids (Oracle), OIDs (PostgreSQL), undifferentiated NULLs and the like, it violates the Information Principle, and thus its rows aren't n-tuples and its tables aren't relations. You can work around some of these limitations, but not all.
Let me be clear and honest: I want to inform you, because you are misinformed. Now if you try to make do for information with imagination, as you tried to do and thus elicited my pedantic answer, there is no point in trying to discuss anything.
Re:relational databases, woo hoo (Score:2)
Melodrama aside -- language is constantly evolving, and draws new words and new uses from many different places. Despite its effective use in dystopic fiction, totalitarian regimes are generally not any more effective at changing language than other sources. Add to that the fact that the idea that our thoughts are controlled by our vocabulary has been utterly discredited. One look at the parade of newly-coined euphamisms for being disabled or black should be enough to convince you of the futility of trying to control people's thoughts by controlling their vocabulary.
I think it's presumptious of you to assume that I'm misinformed. We disagree. I see a flawed implementation of relational theory (like any instatiation of an Aristotelian ideal is bound to be). You clearly see something different.
But my original point stands: everybody uses the term relational database the way I do, and it's disingenious of you to insist otherwise. There hasn't been a massive conspiracy to control people's minds, there has been a natural evolution of language. Loudly denouncing people for using terms the way they are consistently used in the industry is not constructive.
-Esme
Re:relational databases, woo hoo (Score:2)
About language, you are exaggerating. We don't need any new-fangled theories to know that big organisations or charismatic leaders do change words' meanings to suit themselves, and I would say this should be resisted.
I guess you meant Platonic instead of Aristotelian. Anyway, is set theory and predicate logic an ideal that one can dispose of as he wishes? Either you get the operations right or not. SQL gets them wrong.
Who needs conspiracies, when intellectual laziness and greed cooperate?
Relations are a Mathematical concept in set theory. Predicate logic is also scientifical. So if people start saying 2 + 2 = 5 we should assume that's just an evolution too?
C'mon.
Re:relational databases, woo hoo (Score:2)
>>Actually, no, breaking up documents into many discrete units is a requirement of the normalization required by relational theory.
Actually, as Date himself would say, that all just depends on what the logical requirements are. Breaking up data is for logical reasons, to remove redundancy. But, if your database design is to handle documents as discrete entities, then why not?
Date himself argues for the existence of a native XML datatype. This would mean you could have an XML document stored in a single column. The big difference is that this datatype would not be just a blob. It would require a valid XML document, and it would have operators that allow you to interact with the DOM. it would also mean you can change your schema, using a combination of view, for example, to present your data any other way desired.
The whole point that the (serious) relational guys keep saying is that there's no need to throw away the relational model in order to get certain additional advantages. Literally any sort of logical operation on data should be able to exist inside the relational model. And, even more importantly, physical storage can be implemented any way desired for performance. The only difference is that the relational model requires complete logical control over data, which no other model truly handles.
You don't have to throw away one to get the other.
Re:relational databases, woo hoo (Score:2)
Simple enough:
Re:relational databases, woo hoo (Score:3, Interesting)
A. It can't be done.
Q. What do you lose by putting a document in as a BLOB?
A. Granularity. The ability to have the database sort and extract parts of files at the tag level. For example, take a site that has an essay spread over 10 pages. Do you store each section as a database record? That's not clean, as what if we want to break that essay up over 5 pages? It seems rather strange to hardcode into the database presentation logic, so webpage = database record is a workable but inelegant model. Do you store the essay as one BLOB and extract substrings? Extracting substrings is certainly not as faster as an XML database, though it is smarter than 'webpage = database record'. And here we have a scenario where an XML database might suit, and would easily outperform a relational database (yes, Tamino or Excelon do outperform relational databases some of the time! ;).
#######
Here's a general rule of thumb, kids. When you read a post bashing another database model, bashing another operating system, or bashing another programming language, just realise that the poster is a jock who refuses to see where the alternative suits, and where the alternative doesn't suit. They're not about creating understanding, they're bashers.. They're not informative.
Re:relational databases, woo hoo (Score:3, Informative)
A. Granularity. The ability to have the database sort and extract parts of files at the tag level.
That's not true. If you've the Oracle documentation to hand, read about interMedia (formerly known as ConText). It gives you extensions to SQL to use XPath-like statements to select from within an XML document in a CLOB.
Re:relational databases, woo hoo (Score:1)
I don't see how this make what I say not true. I am just saying that XML databases have their place. They can be misused, but so can relational databases.
Re:relational databases, woo hoo (Score:1)
mmmmmmm...... clobs....
Re:relational databases, woo hoo (Score:1)
Re:relational databases, woo hoo (Score:2)
You can either decompose the document in an ordered tree (even SQL, being subrelational, can make recursive queries to put the document together), or make XML document a supported data type.
Obviously you do not know the relational model to understand its capabilities and how different it is from what you know about SQL, or SQL itself.
Re:relational databases, woo hoo (Score:5, Informative)
"Not only do most practitioners think that SQL DBMSs are relational, but they actually blame the problems due to SQL's violations of, or lack of adherence to relational principles on the relational model itself!"
In my opinion, there is no reason that an object-oriented environment cannot implement the relational model, and thus be a true relational database.
Re:relational databases, woo hoo (Score:2)
You can have an engine that actually stores the information in a fixed length field text file, then uses SQL to extract information from that file. Slow? Yup, but SQL can still be used.
Case in point is FOXPRO. You can use either SQL or the xBase language to get at the information. In the same code file.
Re:relational databases, woo hoo (Score:2)
Not quite. SQL adds many arbitrary restrictions to the relational model, and thus fails to implement any significant data independence capabilities. So you are quite limited on how to store SQL data, and on changing the physical schema without changing the logical and user ones.
That is actually a bad example. Not only SQL itself prevents real data independence, if one mixed a relational language (that SQL is not) to any non-relational language to access a particular database, one looses the relational data independence because the non-relational access will have assumptions on the database physical schema and access plans.
Perhaps one would be able to present a stable user schema to the non-relational parts of the application, but even then performance would suffer mightly because of access path assumptions.
Re:relational databases, woo hoo (Score:2)
Well yes, but to be able to change the relationship of the data (ie: table dependancies, column names, keys) WITHOUT the application even knowing that the change had been made means that you need to add an abstraction layer between the application and the data. This already exists in multi-tier models.
But you still need to be able to converse with the data base engine at some level. So your data access layer must have an understanding of the data storage schema. Whether you are using an RDBMS, flat file system, or OO system, somewhere along the line the data in the file becomes data in RAM, and becomes organized in such a way that the application can use it.
Using SQL means that you are using a standards based language to converse with the data base engine. Because computers are stupid, care must be taken to ensure that data integrity is not lost. Whether this happens at the application level, data level, or through some RDBMS engine is only relevant to whomever is responsible for the integrity.
It would be nice to be able to specify ALL of this in the data base engine, but it still needs to be retrieved. And the retrieval system must have an understanding of what it is retrieving so it can package it for the application (and vice versa).
That is actually a bad example. Not only SQL itself prevents real data independence, if one mixed a relational language (that SQL is not) to any non-relational language to access a particular database, one looses the relational data independence because the non-relational access will have assumptions on the database physical schema and access plans.
I was pointing out that SQL is only one way to get at the data. Yes, extreme care must be taken to ensure that data dependancies are not violated (and probably in more places). But then a DBA must do the same when they set up dependancies in the data schema. The main difference is the level at which data dependancy is conserved. I agree that it is MUCH better to set up the dependancy checking lower down. All higher level code is then forced to comply with it.
I jumped into this discussion because the real problem is the way that we store information, not the particular way we get at it. Columns, rows (collections of columns), tables (collections of rows and columns) are all terribly limiting. You must carefully partition the information, try to figure out the owner of each piece of information, then apply strict rules so someone does not break your view of the information. This is known as a data schema.
Then someone comes along and tells you that you forgot something. Now your schema is wrong, and you must change it, and the application which acts on the information must be changed. Oops!
Until we can store, retrieve, and process information in a homogenous fashion we will have these problems.
Re:relational databases, woo hoo (Score:2, Informative)
You seem to have read the site, but missed some important points... Especially, their vehement opposition to objects as they are understood in most OO settings, and inheritance as it is understood in C++ and Java.
In particular, object Id's (=pointers, references) violate the Information Principle, one of the basic tenets of the relational model.
Re:relational databases, woo hoo (Score:2)
Maybe it's just not possible to build a truly relational database or perhaps building one has severe cnsequences that people are not willing to accept.
I think it's a question worth asking and answering don't you think.
Re:relational databases, woo hoo (Score:2)
You are assuming they have no historical knowledge about how the market came to be like it is now.
In fact Hugh Darwen is a current, and EF Codd and Chris J Date are former, employee(s) of IBM, involved in the creation (Codd), refinement (Codd, Date, McGoveran & Darwen) and publicising (Date, Darwen & Pascal) of the relational model and engines.
There are at least three implementations of a relational system: QUEL (as in old Ingres), BS12 (an IBM retired product by Darwen, that IBM didn't push because internal politics favoured SQL as a part of the failed F/S project) and Dataphor Alphora (a .Net translation layer over SQL).
The reason almost everyone uses SQL is herd instincts: SQL is popular, is backed by the database bullies IBM, Oracle (sorta), MS (sorta) & Sybase, and everyone will stick to that no matter what just because everyone else does. Kinda like MS-W32 vs POSIX, or procedural vs functional, or imperative vs declarative.
Re:relational databases, woo hoo (Score:2)
Re:relational databases, woo hoo (Score:2)
I actually know of three relational systems: Ingres QUEL (obsolete), IBM BS12 (not available), and Dataphor Alphora.
Just look at Intel vs RISC, or MS-WXP vs Unix, or C vs Lisp.
Sleepycat does a library, not a full DBMS.
Re:relational databases, woo hoo (Score:2)
I know. You said that. Two of those are no longer around and one is brand spanking new (and according to their web site it's a "Automated Application Development foundation." whatever that means.
"Just look at Intel vs RISC, or MS-WXP vs Unix, or C vs Lisp."
Again that proves my point. Although RISC, Unix, and Lisp are not popular they do exist in niche areas. I guess it depends on how you look at it but RISC systems are very prevalent in high end workstations and embedded systems, unix is gaining popularity every day (especially if you add linux, freebsd and MacOSX) and lisp is still being used and actively developed. That brings up my question again. If Lisp, unix, and RISC have manged to survive so far why hasn't the "truly relational database".
"Sleepycat does a library, not a full DBMS."
Yea OK whatever.
Re:relational databases, woo hoo (Score:2)
Yet that they do exist shows that it is not a technological, but a political -- as in market herd instincts -- issue.
That means that by declaring relational integrity constraints one can implement all business rules without procedural coding. Also that the interface is stored in the database.
But the reason why they are not promoting it as a RDBMS yet is that they are still working on an integrated storage engine. For now their product works as an interface to SQL DBMSs, so they don't get the full performance benefits of which the relational model is capable.
Because all these survived in higly technical environments, or at least under the sponsorship of technically-oriented SysAdmins. OTOH DBMSs are more commercially oriented, so they fell under the spell of IBM.
I am sure you realise that a data access library is just a fraction of the effort to make a RDBMS. Alphora for one all but dismissed Sleepycat as a storage engine; what they need to make Dataphor a full, integrated RDBMS is more in the line of InnoDB than Sleepycat.
Re:relational databases, woo hoo (Score:2)
Short answer is performance. Not the small differences shown in benchmarks but differences of several orders of magnitude.
Long answer probably involves representation of data and the nature of theoretical versus real-world. Theoretical, regardless of how complicated, is always a vast oversimplification of the real-world.
Take something simple like a date. Everybody knows what a date is, right?
You have a table of people including dates of birth and death.
Guestimated dates and partial information cause trouble.
Bad example, but real data is not as clean and exact as one would like. Putting that data into a system that demands that everything be clean and exact doesn't really work.
You're hungry. You need food. Do you starve because not everything is done to perfection?
Relational databases, grrr (Score:2)
Flamebait time!
Summarising several other replies and adding my own biasses...
Summary: All data storage solutions suck. For your specific application, there will be one or two which suck the least. That's why they pay you.
Re:Relational databases, grrr (Score:2)
Re:Relational databases, grrr (Score:2)
I know you meant it as a joke, but for some applications (squid springs to mind, plus a few specialist web cache-like applications that I've written), it's precisely what you need.
Re:Relational databases, grrr (Score:2)
No SQL system is relational, because SQL in and by itself violates the relational principles.
All data can fit neatly into the relational model, provided one defines the domains and normalise.
XML DBMSs won't scale, because they are too complex. In fact, they are just a throw away to the hierarchical DBs that were tried and dumped thirty years ago because of complexity.
Why? Just make text a supported data type. In fact, even SQL databases can do that.
And that's one of the big relational advantages. It can hold any data, optimising access for itself, under a consistent model. Only the DBA has to worry about the physical schema.
The problem is actually worse. There is no OO data model to adhere to, so creating a standard would have to cater to too many different goals. Just look at CODASYL, and weep. An OO standard would probably be much more complex than CODASYL ever was.
Re:Relational databases, grrr (Score:2)
Say what?
Any data can be shoehorned into a relational model if you use enough IDs. For any sufficiently complex model, there comes a point when it goes way beyond "neat".
A few examples that I have worked with spring to mind, most of which would take too long to explain. Consider, however, manipulating a directed acyclic (or, indeed, cyclic) graph where you need to query on "reachability" and propagate information around the network. A relational setup for this would got pretty unwieldy pretty fast.
I'm yet to see an SQL database which can handle multiple terabytes of textual data, pulling in documents and full-text indexing them in "real time". (Disclaimer: I get paid to work on a product which can do precisely this, amongst many other things.) For this, you need a database optimised for storing and indexing text (e.g. SGML, XML).
I think we might be talking about different things when we say "XML databases", BTW. I'm talking about databases which store and index XML data (i.e. XML is a basic data type), possibly in addition to other kinds of data. I think you're talking about databases where XML is the record model too. Sorry about the confusion.
I do take your point about DBAs worrying about the physical model(s) and application developers working on a more abstract model of the data, but unfortunately I've never worked under those conditions. Small to medium-sized organisations working under budgetary constraints can't afford enough DBAs who have domain-specific knowledge about all of the individual problems that the enterprise applications are trying to solve. A physical database which follows the conceptual model of the data closely is a big boon here, as there are far fewer surprises.
Guess you haven't heard of ODMG [odbmsfacts.com]. It very much exists, however most OODBMS vendors support it incompletely [barryandassociates.com].
Re:Relational databases, grrr (Score:2)
Again you are thinking SQL instead of relational, and working around its arbitrary limitations. What you need isn't enough IDs, but a normalised database model.
Please do yourself a favor and get Fabian Pascal's "Practical Issues in Database Management", if you don't want to explain what was your problem and how you tried to solve it. He has some pretty neat graph examples which are simple and powerful in the relational model but nearly impossible in SQL.
As I said, SQL is too complex and full of arbitrary restrictions. These wouldn't be the case with a relational system.
Relational datatypes are arbitrary, so there is no reason to deviate from the relational model to store whatsoever data.
That is just because SQL is so complex. Relational systems would be to SQL as Unix is to MS-Windows: far less administration required.
You are assuming the DBMS to be defective, and lots of procedural code to be needed. You are assuming DBAs' work to be needed in addition to, instead of to replace coders' work.
How it should work is that DBAs work with SysAnalysts to design the types and the logical model. Then the DBA creates the physical model according to expected load. Programmers create the declarative integrity constraints as expressions of the business rules, the DBA tunes again for performance. See how much easier it should be?
Yes, I have, but ODMG does not a data model makes. It is a data access standard, sure enough, but based on OO ideas, not on any formal data model. At most, it is an ad-hoc data model, and thus without the power and simplicity of the relational one.
Moreover, ODMG is so disputed that probably either it will never be universal, or it will grow to be as complex as SQL or even as much as CODASYL, or worse.
Re:relational databases, woo hoo (Score:2)
For procedural code, you are very right. For object-oriented code, though, relational databases are dangerous. Why?
- One problem is that rectangular tables don't map particularly well to hierachies of objects. This results in either distorted OO models or database layouts that look weird to DBAs (and are therefore hard to optimize).
- Another big problem is that they encourage procedural thinking. There is an awful lot of Java code out there that isn't in any sense object oriented; it is so procedural that it might as well be written in COBOL.
- A third is that they encourage leakage between layers. I don't know how much code I've seen where SQL is scattered fucking everywhere, rather than isolated to a persistence tier. Sure, you can say that programmers shouldn't do that, but when faced with a deadline, an awful lot of developers will cough up some SQL hairball rather than fixing the underlying design issue.
- A fourth is that since the schema duplicates information in your code, this acts as a brake on refactoring. If you use a Martin Fowler's tricks for keeping the database in sync (or if you use a good O/R mapper) you can survive this, but it's still a pain.
- And a fifth is that once the database exists, no matter how much the original designers warn against it, people start using the database as an integration layer. Suddenly 14 different apps are munging the same data, making it impossible to change the schema, and nearly as hard to track down a bug. The whole point of OO programming is that data should always be wrapped by the code that goes with it.
- But my biggest gripe about them is that for 90% of the OO apps out there, an SQL engine just isn't necessary, and only serves to slow things at development and again at runtime.
That's not to say that SQL databases aren't sometimes useful, just that they aren't a magic bullet, and they lead an awful lot of people astray.For those OO developers out there, check out Prevayler [prevayler.org]. As long as your dataset fits in RAM (and really, how much does RAM cost these days?) you can simplify your code and improve performance by thousands of times [prevayler.org].
Even for those who can't use something like Prevayler because the dataset is too large, it's a valuable thought exercise to demonstrate that databases need not be objects of worship.
Re:relational databases, woo hoo (Score:2)
Not true. Relational does do hierarchies well. The problem here is SQL.
OO is also procedural. Relational OTOH enables one to think in sets and declarations, thus saving much more programming than OO could.
This is a fault of SQL, not of the relational model.
?!?
On the contrary, the relational model provides data independence that both SQL and OO deny.
Then go read The Third Manifesto to learn how much better the relational model is than SQL.
Re:relational databases, woo hoo (Score:2)
Ah, yes, When I said "relational", I meant the thing that 99% of people think of as a relational database. Sorry for the confusion.
Are there existing products that you feel provide the full power of the relational model? I've never had the opportunity to use a relational DB other than via SQL.
Re:relational databases, woo hoo (Score:2)
Unfortunately, that is vendor Newspeak, not reality.
Alphora Dataphor does not provide the full power, but at least it does not violate any principles. Thus it offers, say, something like 50% of the relational model instead of the 20% SQL does. Obviously the numbers are just feelings, but the reality behind these feelings is solid.
Even if it is a proprietary and expensive product, you still can try a fully functioning demo of Dataphor.
Re:on sharing data (Score:2)
In the OO perspective, no, that's not good. The OO notion is that you should think of data and behavior together, not separately. So it's good to provide access to the object (data + behavior), but dangerous to provide universal access to the raw data itself.
But OO wants to wrap the data or state within a single language interface. If that is done, then how is the data shared with multiple applications, languages, and paradigms?
You expose it via some sort of interface, as needed by the situation. There are lots of good solutions to this, from low-level OS stuff to the venerable CORBA to the latest web services fads. The right choice depends on your circumstances.
Also, what kind of 'bug' are you referring to? A bug in the RDBMS engine, or a shop app?
Exactly my point. Since the responsibility for managing a particular datum is not cohesive, it's hard to say where to look. Is it in one of the apps? Is it in some of the code that's been pushed into the database? Is it a bug in the database itself? Or perhaps there are bugs in all of those, interacting in some subtle way. Who knows?
Whereas in good OO designs, all of the manipulation of a given datum goes through the object that contains that datum. If it becomes corrupt, you know where to look.
Working Together... (Score:5, Interesting)
Case in point, when I read in an Oracle PL/SQL book about Nested Tables, the light bulb in my head went off (or lit up, or whatever). Basically, these nested tables were objects with methods (code behind them), however, could be queried like tables. So, instead of selecting say a person's name, birthdate, and calculating an age, I could select name, birthdate, and age (the age column had code behind it automatically calculating the age). Now the beauty of this is for derived quantities that are only used once, but would be burdensome to store, this was a godsend. However, my DBA completely rejected this idea as too untried and new-fangeled.
This may sound very arrogant, but I think the developer should manage the DBA, often the DBA is a lone-wolf with too much power. Often the poor programmer has to submit changes with about as much hope they'll get done as one might have submitting universe changes to God Almighty.
Re:Working Together... (Score:1)
Re:Working Together... (Score:5, Insightful)
So instead you'd end up with a developer with too much power...
The thing about the separation of database design from "front-end" design (which could be middleware or front-end applications) is that in most cases the database scheme and I/O design has a shelf life far longer than most front-ends. i.e. Don't see that new web interface as a new system with a new database, but rather as a "one of many" front ends to the back-end database: i.e. the database is of much greater long term importance than any front-end.
Regarding your particular scenario: Is it possible that you're looking to shoehorn functionality for your particular front-end into a universal back-end where it might not be appropriate? Will every query on the person table suddenly have have the overhead of calculating the persons age because one page in one obscure part of one front end needs it?
Re:Working Together... (Score:2)
Re:Working Together... (Score:3)
create view view_name as
select name, birthdate, age_from_bday( birthdate ) from base_table_name;
Where age_from_bday is the function used to calculate the number of years.
Oracle traditionally has problems with new stuff. Okay, Oracle only has problems with esoteric corner cases with new stuff, but I've run across some of them w/ partitioning. So for production stuff, your DBA might be right on the money. Good DBA's and good SA's get paid big bucks to be ultra-conservative, and say "No". That's because they get paid big bucks so when they say "This will work in a production environment for the next ten years with acceptable downtime", they are correct. This is coming from a developer whose had to do his own DBA, SA and production support work because we can't afford a DBA or an SA. I dream of having another team member to ensure the stability of my production system.
Kirby
Re:Working Together... (Score:5, Interesting)
I guess you haven't been around the industry too long. You see, this is an industry totally driven by fashions and fads (far more so than even the clothing or entertainment industries). Every year there's a slew of new buzzwords and technologies, each of which promises to be the "silver bullet" and a whole new "paradigm" and none of them ever are. So when some bright-eyed bushed-tailed young hotshot announces that he's discovered the solution to the organization's IT ills, all the "old geezers" just roll their eyes, 'cos they've seen it a dozen times before.
However, my DBA completely rejected this idea as too untried and new-fangeled.
The problem with many developers is that they see a shiny new feature, can't wait to use it, and you end up with an application in which a dozen different people have solved the same problem a dozen different ways.
My attitude is usually that a developer can do anything they want... so long as they're willing to carry a pager that might go off at 3AM, and take responsibility for fixing it before the next business day. Amazing how many times they just wanted to try out a new feature without any real need for it.
In your specific case, you could have done exactly what you wanted to do with a view.
This may sound very arrogant, but I think the developer should manage the DBA, often the DBA is a lone-wolf with too much power. Often the poor programmer has to submit changes with about as much hope they'll get done as one might have submitting universe changes to God Almighty.
Yeah, and the accountants use software, so the developers should manage the accountants! And the salesmen! And the canteen staff! After all, a developer wrote the program that prints their paychecks!
I personally have spent half an hour rewriting a developer's SQL that took the run time down from 15 hours to 9 seconds. Having said that, I don't know all that much about writing, say, MT-safe C++. That's why we have specialists in the first place. I'll bet dollars to donuts that your DBA knows far more about databases than you do, even if you know many more trendy buzzwords than he does.
Re:Working Together... (Score:2, Interesting)
Sounds interesting, would you care to explain what you did?
L.
Re:Working Together... (Score:1)
1) Non-use of indexes
2) Improper joins
3) Procedural mindset
Non-use of indexes is easy. Generally they'll run queries with functions on column names, or type-mismatches, etc. (e.g. WHERE colname * 2 = 12000, etc.). Or they'll not request an index when it is necessary.
Improper joins -- joining on non-indexed columns, non-key columns, etc. Or leaving out a join condition and causing a cartesian product.
Procedural logic should not be in the database. Often you'll see a query such as this:
SELECT keycol FROM table1
-- loop
SELECT *
FROM table2
WHERE foreign_key = $keycol
-- end loop
Obviously a JOIN is needed.
Re:Working Together... (Score:2)
Well, a modern relational database isn't designed to be used as a Von Neumann machine - it is massively inefficient to retrieve and process one record at a time. If you can express your query in terms of sets, and let the database worry about how to actually execute it, performance can be orders of magnitude better. And joins are almost always faster than subqueries. And if you want to use a compound index, the query predicates have to be in the same order as the columns were when the index was created.
I'm sure a programmer could tell you plenty of techniques for making a GUI run faster, and as someone with a DBA background, I wouldn't presume to tell him that since it's a GUI onto a database I could tell him how to write it.
Re:Working Together... (Score:2)
The slow run time is due to the way the work is organized.
This is a contrived example to show what is possible.
You have a document on a screen on one computer that you want to copy to a different computer. To save time and preserve accuracy you will copy this letter by letter. Only thing is the computers are in different buildings and you have to walk back and forth.
What you do is to knock out the incredibly vast amount of (re)wasted motion.
Re:Working Together... (Score:1)
Yes, overall a DBA should know and probably does know more about databases than the developers. However, they can be very resistant to use new features simply because they don't understand them, not because they have knowledge of it and have weighed the pros and cons before deciding against it. Sometimes I do know more about some aspects of the database, because I've actually used the feature(s) successfully in prior projects at another client, while the current DBA hasn't used it or even read up about it.
Re:Working Together... (Score:1)
I've been in this business too long (30+ yrs) to stop learning. The day I do that; make me a manager. I've seen developers balk at using new database features because they refuse to RTFM. The good ones listen and learn. The really good ones use what they learned again. The things I've seen developers do in designs make me laugh out loud.
When I started at my company, the developers did all the work a DBA should do. The systems had no security, ran slow, and the database would get corrupted every 3-4 months. I had to recover from two of these major fuck-ups. It's been over two years and the database designs by the DBA team I'm on run smooth with no corruption. This included moving up a database version for more stability (M$SQL 6.5 sucked wind out a dead cat's Ass). The developers couldn't even figure out on how to convert the monster (2 - 50 Gig DBs). It's now split between a dozen servers with no single database problem bringing down every system we have. A developer may know features, but they never know stability, recovery or security. That's what's on the line for the DBA. We're the ones that accountable for database systems and it's our ass that upper management goes for when systems are down.
There was a comment in this thread on how a DBA rewrote code to speed up a process. That's done in my team every day. I've heard the old Oracle/Sybase/Informix/MSSQL debate for years. What I've seen is that a developer will write code that sucks for any database they use.
As for the mature crack, I'm 51 and you can byte me. I've already seen the fuck-ups that you're going to make, years ago. They never change. Just the size of the systems and the languages developers use. Just remember to learn from them so you don't hurt yourself again. You only have two feet and one ass.
This article is short and common sense... (Score:5, Interesting)
A big issue with iterative development is that the QA folks will quickly fall behind and become very anxious. What's the solution to that? Either embrace the QA person to get closer to the real development environment, or if that is impossible, get a new QA person. That's the only way to succeed.
Quality assurance (Score:4, Interesting)
Just like the actual users the QA folks should be heavily involved in the actual development cycles. Extreme programming states that every development cycles starts with a functional design and the development of tests for each deliverable. Having these available means that QA is able to keep track of the quality of the deliverables very well.
IMHO if QA cannot keep track of the big picture they fail as QA, because that is just an important part of their job. On the other hand perhaps extreme programming should involve relatively more QA people than 'regular' development methods.
Re:Quality assurance (Score:2)
On an XP project, much more QA work certainly gets done than on your average project. One important component of XP are the customer tests, which are automated tests that are used to decide whether or not programmers are done with a particular feature.
Whether there are special QA people who write those tests is a local decision, but good QA people have the right orientation to write really good customer tests.
Re:This article is short and common sense... (Score:1)
The big picture (Score:4, Interesting)
When you have a lot of people all making little changed everyone starts to loose sight of the Big picture and you run into a Too many cooks spoil the broth.
I'm sure a lot of people who read this site have seen a lot of code and design, and probably a lot of horrific code and design, well enough said.
Re:The big picture (Score:3, Insightful)
Why?
He's big on automated testing. He's big on frequent, small integrations. Ideally, every developer integrates with the bleeding-edge source code at least once a day. That's why the changes are so little! No one strays too long from the work of everyone else.
It may seem like anarchy at first glance, but if you read a little closer, you'll see that there are strong behaviors in place to prevent that chaos.
Re:The big picture (Score:2, Interesting)
Assuming that the planning meetings give everyone a Big Picture overview and assign tasks to the developers, and that the cycles are short so that the mettings happen regularly, then you are calling what most people consider normal work "chaos".
That was a crazy sentence. Basiclly it seeems the people on his team are all on the same page. That means that they can each write their parts without fear of stepping on anyone's toes.
Too many cooks only spoil the broth if they don't know what it is supposed to taste like, and/or fail to taste it after each small change.
A developer perspective of the world. (Score:5, Insightful)
As a general rule, the developers think that the database is there to support their application, which is really the piece that solves the problem. In the other hand DBAs think that the developers are there to support their data model, by supplying an interface with validation and some simple pieces of logic that their store procedures don't cover.
I have worked much longer as a developer than as a DBA, but I still find it funny that the article assumes that the developer should be able to add a column to a table freely and the incorporate the changes to the main database. This is the equivalent of saying the DBA should be able to freely change a class or an interface and then add the changes to the source control repository.
While not wrong in itself, it clearly shows that many developers consider the DBA role secondary to the developer. It goes something like this: I can somehow do some DBA tasks that impact the development like adding tables to the schema, I just don't want a get involved in the boring parts (backups, recovery or replicating schemas).
I think that creating a good data model is as difficult as creating a good application design and doing a decent store procedure as hard as doing an efficient method. While some DBAs can write very good C++/Java code and some developers can design very good data models, no one should be doing each other job unless they really, really, really know what they are doing.
As a general rule of thumb, if you consider that mySQL is a better database for large complex applications as PostgreSQL or Oracle, you should not be doing any database work.
Re:A developer perspective of the world. (Score:3, Insightful)
True but perhaps irrelevent. Sometimes the most important difference between DBAs and developers is not their technical skills but their attitude to change. The nature of the job is that a DBA tends to be a bit like a soccer goalkeeper - he's not rewarded for scoring goals (adding new features that responding to user requirements more rapidly than anyone else) - all he gets is the blame if allows goals in (lets someone break the database). The nature of the job tends to reward defensiveness.
The result - semantic corruption, with any amount of database re-use, however dirty, prefered to re-factoring. Like my insurance client in 1999, who were using 2000-01-01 as the null value in some of their date columns...
It's a really good article. We're doing a fair amount of the recommendations already, I can confirm the value of the tight DB layer, and having good test data packs from the start. In fact I'd go further - it doesn't matter whether you think you're doing waterfall or iterative, you will have to change the DB and you might as well work out how to do it efficiently.
Really good comment (Score:2)
What I really am not crazy about in the article is that it fails to realize that databases live on. Applications come and go. Databases just keep on living. Hence doing a bad database design comes back and haunts you forever....
The project Atlas that has worked for the past two years is a CPU cycle in terms of databases. I know at large corporations that they have databases that have been running for thirty years non stop. It is called a mainframe!
Re:Really good comment (Score:2)
You're close, but you're not quite there yet.
What really should live on is the data, not a particular database or a particular schema. The reasons that applications have to come and go is that they aren't being maintained properly. And one of the big brakes on proper refactoring is database schemas that aren't allowed to change.
If you force your database schemas to evolve along with the application (which in turn is forced to evolve along with your business) then everybody wins. Well, everybody except the Oracle marketroid trying to sell you yet more magic bullets.
Re:Really good comment (Score:2)
Now about applications coming and going because they are not maintained properly? I doubt that one. Some may fall into this category but not all.
The other problem that is not being remembered and I think this is extremely important is that you cannot change databases as simple as developers would like to have it.
For example lets say that you want to keep things flexible and change databases. Can you ensure that all of the data in database A will equal 100% database B? Answer no! I remember at a client once that they wanted to move data from a mainframe to an Oracle database. About 2 terrabytes. Oracle would not guarrenttee data correctness and hence the client did not make the move.
The problem is that developers look at software using a one off design issue, which is correct. But data is a production issue. It is sort of like the engineer who designs the car that cannot be built because of production constraints. In the industry the production constraints determine which car is built. Likewise in the software world the database needs to define what programs can be built, not the other way around.
And I am not a DBA...
Re:Really good comment (Score:2)
If you don't maintain things properly, that's true. If you don't have good system tests, then moving to a new system is hard, because you are never quite sure that the new system works like the old system.
Fear, as they say, is the mind-killer. For the project I'm working on right now, I'm confident I could migrate it not just to a different database, but to an entirely different database technology (e.g., to LDAP) in short order. Why? Because I have good tests, and I made sure to keep my persistence logic in a single place.
The problem is that developers look at software using a one off design issue, which is correct. But data is a production issue. It is sort of like the engineer who designs the car that cannot be built because of production constraints. In the industry the production constraints determine which car is built. Likewise in the software world the database needs to define what programs can be built, not the other way around.
All production issues should be development issues. Neither the database nor the code should define what gets built; business needs must determine that.
It sounds like you've worked an environment where the developers can escape consequences of their actions. Try doing weekly iterations and monthly releases; the developers will soon be cured of there ignorance about what it takes to put things into production.
Re:Really good comment (Score:2)
Then you break things into tiers and have them make requests of one another rather than rummaging around in the raw data.
For example, suppose a previously uncomputerized business decides that the first thing they need is a intranet where they can track the names and addresses of their customers. So you build one.
So then they want to add something to invoice the customers. Since you wrote the contact manager in a clean way, the UI stuff and the domain model were already separate, right? So you just pull the domain model out into its own layer, and now you have two apps talking to the same customer data.
Sure, you can use the database as an integration layer. But that's only one option, and rarely the best one, as you are then forced to either a) have duplicate code all over the place for the same sorts of things, or b) start putting code into your database via stored procedures, triggers, and hatnot.
Now we all know that duplicate code is the way to madness and woe. But putting logic in the database isn't much better. Now your database isn't a database, it's an application server in disguise, one with weird proprietary languages, poor tool support, and limited options for scalability.
If you're going to build an application server (which will happen in some form when you go from singular app to plural apps), you might as well do it in a language that all your developers already know, not one that is a mystery to all but the DBA. Then you get the advantages of portability, a wide variety of tools, and a panopoly of ways to scale up as requirements grow.
Re:A developer perspective of the world. (Score:2, Insightful)
We seem to forget that database(-s) are there only to serve business and business requirements. Nice if you can build a fast, reliable and maintainable database just to find out that your company can't afford it - what then ? Or, the company is planning to announce new services and the current information structure can't support it because some DBA or developer hasn't included it to the design.
(IMHO) the database cycle is more like from business analyst to DBA to development project to performance management and back. None of these functions has ( usually ) enough information to create real business databases. Nothing says that it can't be even one person ( I just have to see first so talented person to believe that it can be done well ).
Otherwise a nice article ( from developers perspective ).
Re:A developer perspective of the world. (Score:2)
The point of object-oriented analysis and design is to define the data model. So yes, any team doing OO work should have a skilled OO A&D person; they are generally called architects.
Beware, though, of self-styled "architects" who are too busy or too important to write code. They are generally useless. Worse, they are good bullshitters, spewing out designs that sound plausible but are hell to implement.
Works for OODB as well. (Score:4, Interesting)
We don't worry about *any* kind of DB administrator. Each developer has their own instance of the database. We don't worry about schema changes that break the database, because we *also* have a way to import/export the database to an XML file. Thus, if the schema radically changes from the deployed version, we just export to XML and re-import, so there is no complex though about 'schema evolution' necessary,
Of course, with everyone having such free ability to make changes that impacts the format of the data store, we need good unit tests to make sure things don't break unintentionally. This is actually one area we need to improve upon. People can make changes that affect the schema easily, and most times, its not an issue... but a lot of times people make changes that would impact the XML format, and they don't always handle it properly.
Unit Tests are Key, but that's nothing new to the concept of refactoring.
Re:Works for OODB as well. (Score:2, Insightful)
I would note that "object oriented databases" tend to resemble the hierarchical or "network" databases of the 1960's. They fell out of favor for various reasons.....until OODBMS started popping up again. The OO croud has not been able to solve many of the same problems that were there in the 60's.
It is true that they tend to be more "organic" than RDB's, but they also lack the rigor of relational algebra that makes RDBMS scalable, manageable, and more queriable. (Although the current crop of RDBMS also fall short of the theory by some accounts.)
Done similar things (Score:3, Interesting)
The Good: Database changes were part of development. When our system worked right (58.3% of the time) All changes would go through QA, a small fix cycle, and we would push code and database changes during the evening (we were running a web site for people who used it during the business day only).
The Bad: People tended to develop with live data. The main problem with this was that if something changed, It might break a unit test.This could be a real problem if Sensitive Data was involved as well.
The Mechanism: We had a script (PERL) that executed a list of sql statements embedded. For a revision, we changed the name of the script (the first script ended in 0001, which for the next revision would be 0002 etc) and checked if that reision had been executed, and if all the previous revisions had been executed; script 0008 could only be run after script 0007 had been run etc. We had a single table that kept track of the current revision.
The nightly backup from live was dumped into the Integration database. The current update script would be run against it, then all functional tests. Every few days we would push code live. Yes, every few days. This was an organisational issue and yes, it casue a lot of headaches.
To improve:
Better QA. We should have had a batch of scripts that could be run agains the DB. Instead, our QA person had to run through them manually.
We should have had test data for the tables that were primarily used for collecting and reporting, instead of runnning against live data. I
Instead of a self executing perl script being the required mechanism, I would have an executable on a machine that tracked the schema name, schema version number
For a give application, we had 3 databases running. One was for collection, one was for reporting, and one was reference data. Only the collection database was backup up, as the other two could be regenerated from source. However, we still needed to revision conrtol the reference databases source, and it was faster to send patches than to update the whole thing each time.
We started moving to an Application Service Provider (ASP) setup where we uysed the same schemas, but with different data in them, for different clients. Since our clients were Local governments (County) we wanted to A) Be able to get a new county up and runniong quickly if we got them as a client, B)Be able to dump whatever existing data they had into that counties db quickly, C) Keep the counties data separate from each other, D) Maintain our versioning. So certain scripts had to be run against County X data of the collection scheme. If I had to do it agains, with what I know now,I'd have had a revision control database that kept track of the other databases (and be self maintaining, why not) With Schema name and data set name, current revision for each.
Perl, not PERL (Score:1, Insightful)
Your credibility goes right out the window if you don't even know that. Reading your post it does seem that you are just throwing out some buzzwords and acronyms, hoping for some cheap karma.
If I wanted to be real nitpicky, there are dozens of other such mistakes - things that are called things they are not and misspelled. Sad, sad.
The key point (Score:5, Insightful)
Databases, however, are a lot harder to iterate - the cost of change is higher than with any other code. Martin Fowler is laying down an approach to manage (not reduce - manage) that cost, and it all comes down to a guess we have to make - do we think the overall cost/benefit tradeoff of an iterative process is better than a Big Design Up Front process ?
On the eXtreme Programming mailing list, there's been a lot of discussion about how to deal with databases - some deny the need for databases altogether, some advocate using Mock Objects for testing and even development etc. It all boils down to the cost of change - it's expensive to change a database design because it is very hard to identify the knock-on effects. Some changes are relatively easy to manage - adding a column is unlikely to actually break anything - but others can wreak havoc with existing applications - changing the type or size of a column for instance.
I'd love to think that the next big improvement in software development tools is not going to be yet another language but a sensible way of tying objects to their persisted data. All the solutions I've seen so far are bolted-on - they either force the database into unnatural positions, or make the objects fit into a model that's not quite what they'd be otherwise.
In the meantime, this article is well worth investigating - the idea of evolving the datamodel in tandem with the migration scripts is very powerful.
Re:The key point (Score:1)
Like I have described elsewhere, relational thinking and object thinking tend to be at odds. They just simply view the world differently. One or the other will eventually have to give up territory IMO. Existing approaches to co-existence tend to neuter one or the other WRT reach and power. This is part of the reason why OO fans and DBA's tend not to get along. (I tend to favor the relational viewpoint. It fits the way I think better.)
XP with Databases (Score:2)
I find that testing "code with side-effects" (ie: database inserts) is the hardest type of code to test, and I haven't yet found a solution that satisfies me.
On the eXtreme Programming mailing list, there's been a lot of discussion about how to deal with databases - some deny the need for databases altogether, some advocate using Mock Objects for testing and even development etc.
As the XP mailing list talks about, you can always create mock objects and test their state, etc, but it's still not quite legitimate. You end up building these massive meta-models that themselves might have issues. Perhaps the best solution I found was to have a "test" instance of your database that would always contain an appropriate seed of test data. If you keep your "side-effect code" and "test the side-effect code" inside of a transaction that you roll-back, you're pretty well off. Unit tests can start taking a long time, though.
Things get worse when you look at Ron Jeffries' Adventures in C# [xprogramming.com] where he starts on the slippery slope of re-implementing the textbox class as a mock object.
Not much of a point here, more of a "me, too" response.
Re:XP with Databases (Score:2)
"scattered willy-nilly" (Score:3, Interesting)
I disagree with this more or less. SQL is often too closely related to the application to put in a separate place. You have to go hunting back and forth to see and manage the relationship. If there is duplication, then I agree that it should be factored to a shared spot. However, beyond cleaning duplication, keep the SQL near where it is used.
If you want to be able to track it, then put some kind of comment marker that a grep-like utility can use to find and gather the SQL if need be. Both approaches are conventions anyhow. My suggestion gives the best of both worlds.
Also, OO proponents tend to use simpler SQL, and thus it might be easier to put wrappers around such trivial SQL. However, a more balanced approach is the use the full power the DBMS rather than re-invent it in your application. Hand-built indexing, joins, filtering, multi-user contention management etc. built into app code is a common sin of OO design IMO, including Fowler's designs, I am sorry to say.
Sometimes I think many OO'ers are motivated by a desire for control, and that is why they would rather reinvent the DB rather than use an existing one. But that is anti-reuse IMO.
Relational thinking and OO tend to be at odds, either way. It is not practical to have both manage the noun models and noun views IMO. Pick one or the other, I say. Thus, either get an OODBMS, or relax the OO design to not duplicate and fight with the RDB and query languages.
Re:"scattered willy-nilly" (Score:3, Interesting)
Re:"scattered willy-nilly" (Score:2)
I agree that OO and relational thinking are at odds, and that mixing them produces poor results. But if you do good OO design, then the kind of database you use shouldn't be a big issue.
The trick is to put all your persistence logic in one tier of the code. I have one app that will happily use several different backing stores (java serialization, XML files, and SQL storage) at the change of a config line. I could easily add code to use an OO database or an LDAP database.
As long as you design that way, there's very little "fighting with the RDB" to do.
Sometimes I think many OO'ers are motivated by a desire for control, and that is why they would rather reinvent the DB rather than use an existing one. But that is anti-reuse IMO.
Just because you have a database that can do something doesn't mean that you should do it there.
Java, for example, has handy built-in stuff for indexing, joins, filtering, and multi-user contention management. I can happily reuse that, remaining database neutral. Better, most outfits have many more programmers than DBAs; if the logic is in the code rather than the database, I have higher development concurrency and a better truck factor.
Reuse is swell, but as a developer my responsibility is to efficiently deliver good software and keep it up over the long term. RDBs are one tool I'll use to achieve this, but only when they deliver the best business value.
Re:"scattered willy-nilly" (Score:2)
Yes, I am. Although I did make use of some SQLisms when optimizing performance.
We agree: If you use all of an RDBMS's power, it probably won't be much of an OO app. If you really do good object work, you probably won't use all of an RDBMS's power.
I'm willing to use whatever tools it takes to get a client's problem solved. I find that for apps over about 10,000 lines of code, I'm fastest with OO tools. That's especially true when the project will go through many iterations; then the flexibility of OO code, especially using modern refactoring tools, is unmatched.
But if I'm doing something quick and dirty or a little something where needs are stable enough that it will never go past v 1.0, I'm glad to write procedural code that's heavily coupled to an RDBMS.
I do not believe Java makes a better DB than RDBMS.
It doesn't need to be. RDBMSes are undoubtedly a better general-purpose solution for storing, fetching, and manipulating arbitrary data. But 98% of the apps I write perform a few operations on very specific data. For a given application, almost all of the theoretically appealing relational possibilities remain unused in practice. Manually writing the few things that a database would do for me often doesn't outweigh the penalties a RDBMS brings with it.
I find that is especially common for server apps with moderate data sets (under, say, 2 GB). In that case, keeping everything in RAM in Java objects is hundreds of times faster than fetching from a database. For that kind of speed improvement, I'm willing to go to the slight extra effort of populating a java.util.HashMap instead of doing "CREATE INDEX".
The same goes for things where the data just doesn't fit well into the relational style. I'm sure somebody has found a way to cram images, drawings, and HTML structures into ANSI SQL, but I sure wouldn't want to look at the code.
But if somebody wants to run arbitrary queries against a large body of data, I'm always glad to dump the stuff out into an SQL database. Crystal Reports and its cousins are swell when people want to go romping through the data.
then the network databases of the 1960's have made their comeback
And if they have, so what?
As professionals, we should be interested in using the right tools for the job, whatever they might be. The technology of paper is thousands of years old, but if a client comes to me with some overblow set of requirements, I think it's my duty to tell them if they'd be better off with a stack of index cards.
Re:"scattered willy-nilly" (Score:2)
Sure.
For indexing, the point of which is to speed finding matches, check out the Java Collections classes. For joins, you just establish the proper relationship between the objects. For multi-user contention management, you wrap things in syncronized blocks.
If you need more than the basic stuff that Java provides, check out the page on the Prevayler site where they list various object indexing and query libraries [prevayler.org].
The problem is how developers design! (Score:3, Interesting)
If you normalize you don't have to worry about null-ability of columns. You don't add and drop columns (usually). Pack those tables behind views and your application doesn't need to change a thing:
CREATE VIEW user_info AS
SELECT *
FROM user u
INNER JOIN user_detail ud ON u.username = ud.username
-- etc
Then your app would simply select from that view.
The problem is that developers don't take the time to properly learn relational theory, instead content to know the basic semantics of the SQL language and call themselves 'fluent in SQL'. They know how to create tables in their GUI of choice (or even code it by hand) and are 'database designers'.
Another benefit is stored procedures. Although abused to include procedural logic in the database, they can help keep database logic out of your application and generally help much along the same lines as views:
CREATE PROCEDURE get_user_info AS
SELECT *
FROM user u
INNER JOIN user_detail ud ON u.username = ud.username
-- etc
In this case the stored procedure would be called in your application.
Re:The problem is how developers design! (Score:1)
While I appreciate the SQL 101 in your post (note sarcasm), I disagree. Languages like PL/SQL are largely designed to hold procedural logic and provide excellent performance.
Re:The problem is how developers design! (Score:1)
If every developer out there had taken 'SQL 101' perhaps we wouldn't be having this conversation in the first place.
Re:The problem is how developers design! (Score:1)
Views are updatable in most DBMS' -- there is a good series of papers here [uk7.net] which discusses the topic more fully.
"Automatically Update all Database Developers" (Score:1)
Also, I should have some input as to what is updated. A DBA shouldn't have complete control over the schema. The DBA and developers need to work together to work out an ideal schema. Ideally, it's msotly worked out *before* any coding is done. DB objects should not be done on the fly in msot cases.
Re:"Automatically Update all Database Developers" (Score:3, Insightful)
That's one way to develop.
Or, instead, you could assume that change is a given and tune your development process for that.
The DBA and developers need to work together to work out an ideal schema. Ideally, it's msotly worked out *before* any coding is done. DB objects should not be done on the fly in msot cases.
Yes, ideally all requirements are identified, and then all the design is done, and then all of the code is written. Ideally. And ideally, you could remove the backspace key from they keyboard.
Alas, most of us don't get to code in an ideal world. The premise behind the various Agile methods (Extreme Programming, Crystal, Scrum, FDD, etc.) is that since the world isn't ideal, we might as well pick development methods that are tuned for the world we live in.
Interesting notion, eh?
Persistent object model design mastery (Score:2)
This gives you the ability to let most of your developers works solely with the object model, and encapsulates things like calculating an age from a birth date and today's date or a full name from first name and last name without polluting the database with miscellanea. Only one or two or a specified small team of people need to be concerned with the design of the database and writing SQL. All of the other developers can just use the Enterprise Objects and not worry about it.
However, this brings me to another point -- all of the successful WebObjects projects that I have seen share one thing in common: a highly skilled architect who has a solid understanding of not only the development process but also how databases work. A good developer teamed with a good DBA is no substitute for one master artiste in this case, as the master can know how to create an object design that meshes precisely with the database to optimize performance and ease of development, as well as making it relatively easy to create the UI.
In fact, a sign of a well-run WebObjects project is that for the first week or two, most of the developers are sitting on their hands or taking long lunches or reading Slashdot. During this time, however, the senior person or two is taking the time to get the underlying data object model right. Only when that is solid does the manager unleash the rest of the troops.
--Paul
Re:Persistent object model design mastery (Score:2)
Or not.
Many people use EOF with Agile methods, which generally treat design as something that should be done on a just-in-time basis, rather than in a big up-front spurt. Indeed, EOF works even better with Agile projects than with traditional once, as it makes continuous schema evolution much easier.
Re:Persistent object model design mastery (Score:2)
Well, I've done it. Repeatedly. So have others, including the author mentioned in the article that this story is about.
You're right that it can't be done just any old way; if you try it with traditional methods, you'll get flattened. It takes special techniques, carefully applied. Plus a team who are all professionals. For more details, see Martin Fowler's book "Refactoring", and Kent Beck's book "Extreme Programming Explained".
Shout as much as you like, but just because you can't do it doesn't mean it's impossible; it just means you don't know how.
Re:The real problem (Score:1)