Database Bigwigs Lead Stealthy Open Source Startup

BobB writes "Michael Stonebraker, who cooked up the Ingres and Postgres database management systems, is back with a stealthy startup called Vertica. And not just him, he has recruited former Oracle bigwigs Ray Lane and Jerry Held to give the company a boost before its software leaves beta testing. The promise — a Linux-based system that handles queries 100 times faster than traditional relational database management systems."

Column oriented databases

[Anonymous Coward]

The article seems to describe the big advantage as being column oriented.

How does this differ than KX System's kdb (www.kx.com) which IIRC is similar in that way; and is alredy in use at many if not most major financial institutions (see their customer list)?

Awesome

[Fyre2012]

This is totally what we need.

With comodity hardware getting faster and cheaper by the minute, having a system that can handle a higher than average load with optimized software is, imho, a winner.

I'm sure everyone here can add some anecdotal evidence to how they had a heavy-hardware, database serving machine die on them because of some software bug.
This is one of the reasons I've been looking forward to ZFS. Hopefully the DB guru's will take the best of what's good about software, drop the legacy crap and really deliver something that's going to handle the kind of load that a good slashdotting delivers with hardware that didn't require a lease to be affordable.

Perfect timing

[defile]

Loading a million random records out of a set of one hundred million records is an enormously difficult task for an RDBMS on commodity hardware (e.g. magnetic rotating disks). This is a more common task than you would think. ORM systems backed by an RDBMS, such as Ruby on Rails, Django, Hibernate, have exactly this requirement and will only demand more as these models become more mainstream. Think about what search engines have to do: find millions among billions, all to show a user a dozen.

These problems are solvable now, but there's a lot of duplication of effort going on that a smart database vendor could solve for us.

Why does a company promising Linux solutions...

[WindBourne]

run Windows for their website? [netcraft.com]

Re:Column oriented?

[stoolpigeon]

info week just ran an article on hp [informationweek.com] getting into data warehousing and bi that had this paragraph pretty early on: Until sitting down with InformationWeek recently, the company has been mum on the initiative--not so much as a peep from its normally talkative marketing team. Indeed, it's an unlikely move into a sector where IBM, Oracle, SAS Institute, and Teradata have years of experience, well regarded products, and loyal customers. Those four vendors--along with Microsoft, which has muscled in on the strength of its SQL Server database--hold about 85% of the $5.2 billion-a-year data warehousing software market, a sector IDC projects will grow 9.5% annually through 2010.
 
so you are right - there's a lot of opportunity there, even for a small player.
 
on a side note, i thought the opening paragraph described the current situation pretty well
  For more than a decade, big companies and sophisticated data aggregators have adopted data warehouses, yet few have mastered them, and many have outright failed in the effort or have been scared off by the complexity. The goal is to give workers access to real-time data across departments and geographic units, but more often than not, data warehouses end up as costly clunkers with outdated, inconsistent, and missing information.
 

Re:Why does a company promising Linux solutions...

[Mad Merlin]

Look again...

$ curl -I www.vertica.com
HTTP/1.1 200 OK
Date: Wed, 14 Feb 2007 23:00:26 GMT
Server: Apache/1.3.33 (Unix)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.4.4
Set-Cookie: PHPSESSID=488de093f5b89a78277a234e1e9886a6; expires=Sat, 10 Mar 2007 02:33:46 GMT; path=/
Last-Modified: Wed, 14 Feb 2007 23:00:26 GMT
Content-Type: text/html; charset=utf-8

One size doesn't fit all

[perfczar]

This is a different kind of issue, really, more like the difference between a CPU and a GPU. At the moment, a good GPU has >100x the performance of a good CPU on a certain class of computations. Column stores will clearly never replace row stores for transaction processing for obvious reasons, but (coupled with a few other architectural decisions) they do exhibit >100x the performance of row stores for the kinds of queries seen in data warehouses.

Also, the two technologies are complementary. The goal is not to replace one thing with another, but to provide more kinds of tools and make them work together. Keep a row store for transaction processing, and feed the data into a column store for analysis in near real-time, much like a video game uses a CPU for AI and a GPU for 3D rendering.

Re:Column oriented?

[Virtual_Raider]

I've worked in DW for a time, and I can tell you that it's not easy to "get it right" because so far it's not something that can be packaged. You can get the data models and fancy machinery, but you will most definitely need an architect to tailor it to the particular organization because all companies work differently on the inside. And that architect will have a dickens of a time understanding how the company works because the bigger they are, the more likely not even their own employees do. As long as there isn't an official structural model imposed on them like it happens with accounting, corporations will grow and be structured however best suits them (or sometimes they just "grow" like wild weeds, unruly and chaotic). And a Data Warehouse is an attempt to code this internal structure and its dealings in a central repository that will serve a number of goals like Business Intelligence, Trend Analysis, etc... So you won't find a product or solution that will fit your company out of the box. It's pretty much like with self-help books. The general idea works in general terms, but you have to adjust it to your own reality and quirks for it to be of any value to you in particular.

Stupid question: Still SQL?

[WoTG]

I've never heard of column based databases prior to this article. Would I be correct in assuming that you still can work with these using regular SQL?