UML, PostgreSQL Get Corporate Support

tcopeland writes "An article on NewsForge highlights some changes in the upcoming PostgreSQL release (v7.5) that are funded by Fujitsu. PostgreSQL core team member Josh Berkus says that "Tablespaces, Nested Transactions, and Java support" are being underwritten by Fujitsu; this has also been mentioned on the postgresql-hackers list. He also says that 7.5 will be "...the most significant new release of the software since version 7.0 almost four years ago". Good times for PostgreSQL users!" And ggoebel writes "Jeff Dike posted a notice to the UML [User-mode Linux] developers mailing list: 'The first bit of news is that as of last Monday, I am working for Intel. They generously offered a full-time position, off-site, with my time mostly spent on UML. This basically means that UML is no longer a part-time, after-hours thing for me, so we should start seeing more work happening on it, especially compared to the last month or two.'"

Good

Good to see PostgreSQL getting some attention !

If only our application here didn't cost millions to develop and wasn't dependant on MS SQL...

GUI Tools

I just started using pgadmin3, and I have a feeling I have a way to go before I get really good at it. I could really use a migration tool since, as with the previous poster, we have a lot of SQL Server based applications I'd like to see moved to something like PostgreSQL.

Frankly, I still like the old TCL based "pgaccess". It was buggy as all get out, and really bogged down on larger databases, but it had some really nice tools such as the visual query designer.

The article mentions a couple of other GUI tools for accessing and maintaining PostgreSQL databases. Has anyone else used these, or are there other tools that people like?

UML is pretty awesome

It's really the future of "shared" webhosting because it balances the power of a full server against the cost of a shared one. Some hosts like JVDS [jvds.com] and RimuHosting [rimuhosting.com] are already doing this and it's great.

Re:UML is pretty awesome

It's really the future of "shared" webhosting because it balances the power of a full server against the cost of a shared one.

I respecfully disagree. While UML gives you excellent isolation, it is an extremely inefficient way to virtualize your server since it does not take advantage (by design) of all the optimizations that UN*X provides. UML is great for kernel developers and applications where isolation is far more important than performance.

In Linux virtual server hosting, the future will be Linux VServer Project [linux-vserver.org]

(ok, I'm somewhat biased, I admit)

That's all fine and dandy, but...

'The first bit of news is that as of last Monday, I am working for Intel. They generously offered a full-time position, off-site, with my time mostly spent on UML. This basically means that UML is no longer a part-time, after-hours thing for me, so we should start seeing more work happening on it, especially compared to the last month or two.'

Will this mean that Intel might have a chance to influence its development? The true benefit of projects such as this is their independence from the big brother corporations who attempt to control the industry/market.

Re:That's all fine and dandy, but...

Well obviously if they are funding development then they will have influence on what gets worked on. What political agenda do you see Intel as likely to have on advancing user mode? It would seem to me that this is fairly typical of Intel software devleopment for the last 15 years -- making sure that there is publically available code highlighting how to do cool things with their CPUs.

Re:That's all fine and dandy, but...

The true benefit of projects such as this is their independence from the big brother corporations

You mean like Sun and HP funding the Apache group [apache.org]?

Or Novell and Ximian underwriting the Mono Project [mono-project.com]?

Or IBM contributing to F/OSS [ibm.com]?

Do you think these and other projects would be where they are today without the backing of serious money/resources?

Re:That's all fine and dandy, but...

That's not the same thing. If the higher ups at Oracle tell the Oracle developers to include a dubios feature that may be bug ridden, the developers DO IT because they were told to.

If IBM tells the apache group to put in dubious and buggy code, the apache group tells them to buzz off.

There is a difference, even if it isn't obvious at first glance.

clarification please...

UML ....

(1) Unified Modeling Language?

or (2) User Mode Linux?

Methinks (2), given that I work alot with (1) and have never heard of Jeff Dike

Re:clarification please...

Sheesh, click on the link! It's user mode linux.

Solid stuff, that PostgreSQL...

...RubyForge [rubyforge.org] has been running on it for almost a year now, no problems.

Only a half million records and only about 75K queries a day, so it's not a huge DB... but it's definitely getting the job done.

UML is no longer a part-time, after-hours thing

This basically means that UML is no longer a part-time, after-hours thing for me

You have my deepest sympathy.

UML

OK, UML is User Mode Linux [sourceforge.net]. Got it. No, no, I'm not confused, I get the coincidence with the other extremely widespread use of the acronym. No prob, Dude.

UML

Who the hell is Jeff Dike and why is he working on the Unified Modeling Language [uml.org]? And why does Intel care about it?

Oh, you meant User-mode Linux? Well, why didn't you say so? Sometimes I think these writeups are intentionally confusing.

Table spaces?

Does this mean that PostGreSQL will actually be able to write *directly* to disk cluster? That would be one serious performance boost! My only request is that they do us all a favor and make sure that we can fragment the tables across spaces. It tends to suck when one table fills an entire drive, and it refuses to use all the space on the other drives.

Re:Table spaces?

"Tablespaces" allow you to put individual tables on different storage devices. Prior to tablespaces, an entire database had to be on one device*.

You are referring to two completely different technologies:

(1) "Writing directly to disk cluster" - By that you seem to mean direct disk access, not through the filesystem. I don't even think this is part of the PostgreSQL TODO, because there is just not a very strong need. Are you experiencing performance problems in this regard?

(2) "fragment tables across spaces" - By that you mean "Table Partitioning". That allows you to break up a single table across multiple storage devices. That would be very valuable technology, but as far as I know, won't make 7.5.

If all these features really work out for 7.5, they should call the release 8.0, and maybe they will.

*: There are some tricks you can use if you need to move a single table to a different device prior to 7.5. I think symlinks work fine, but if it's important, I'd wait for 7.5 or ask on the -general list to make sure it's correct.

Re:Table spaces?

"Tablespaces" allow you to put individual tables on different storage devices. Prior to tablespaces, an entire database had to be on one device*.

Strictly speaking, that's not true. You can move things around manually, and some have done so, but it's not pretty, not easy, and not easy to maintain. Implementation of tablespaces in PostgreSQL simply allows its users to easily do what was previously an arcane-voodoo art. So clearly, it's a big step up. But, you already knew that. ;)

"Writing directly to disk cluster" - By that you seem to mean direct disk access, not through the filesystem. I don't even think this is part of the PostgreSQL TODO, because there is just not a very strong need. Are you experiencing performance problems in this regard?

That's correct. AFAIK, there is no desire to implement raw partition support. The speed difference is minimal and the required code is large. Basically, you wind up writing a FS and associated buffer management into the database. The return generally is not very high. It used to be, many years ago. These days, filesystem technology and implementations are plenty fast. Those that want raw partition access, IMO, are simply living in the past.

If all these features really work out for 7.5, they should call the release 8.0, and maybe they will.

You are correct. Accordingly to the list, the numbering constantly goes back and forth. From what i gather, they are waiting to see what features actually make it in. Depending on the scope of changes, they'll then determine the version number. As a rule of thumb, people are calling it 7.5, simply because nothing else has been blessed.

Please don't think I'm correcting what you've said. You've said nothing that I disagree with. I'm simply adding a followup remark. ;)

Cheers!

Re:Table spaces?

By that you mean "Table Partitioning". That allows you to break up a single table across multiple storage devices.

For the uninitiated and lazy, is there any compelling reason why that's better than putting the database files on a RAID and letting the OS split the table across devices?

Re:Table spaces?

> For the uninitiated and lazy, is there any compelling reason why that's better than putting the
> database files on a RAID and letting the OS split the table across devices?

Sure, you might want to distribute your data across multiple arrays. For example - keep your logs and tempspace on an fast & expensive raid 0+1 array of fast (15k drives). Then put small OLTP stuff on a another raid 0+1 array. Then put your huge graphic images, documents, etc on a much more economical RAID5 array.

I use multiple arrays all the time for performance and economics (in db2 & oracle) - this is cool to see postgres pick itup.

Re:Table spaces?

The answer to this question depends on what your database looks like. For most small, general purpose databases the RAID approach is great. Fast, simple and not much planning required.

However, for larger or more complex systems there are some advantages to splitting tables over multiple disk systems. For example, tables with lots of little niggling disk writes (access tables, change logs, temp tables) can go on a fast (possibly striped) disk system. You don't have to waste high-priced, high performance RAID on archived data (if it crashes, restore from tape), or on large media files etc stored as blobs or clobs.

These are just examples, but on a large server with several different disk sytems available, this technology lets the database designer match storage system performance characteristics much more accurately than a simple raid.

Re:Table spaces?

They are going to reconsider this if someone can write a caching system that can beat the os but so far that hasn't happened.

It's a little more complicated I think. Using the filesystem has other advantages as well:

(1) PostgreSQL can work well with other applications running. Let's say you invent the best caching algorithm possible, then you still have two seperate caches, one for PostgreSQL and one for everything else. That means you have to dedicate the machine to PostgreSQL and have a high PostgreSQL cache (but any other app will suffer), or give postgres a low amount of cache space and it will suffer.

(2) The postgres developers don't want to worry about the bugs involved in making their own filesystem. Also, who's to say they can make a filesystem as fast right off the bat? It might be a huge development effort, with relatively minor benefit for most people.

Re:what's the point?

You've obviously never run a large database before. While a single RAID partition is fine for most uses, when you get into situations where you measure queries by how many are run per second then things really start to hit the fan.

Tablespaces allow you to do things like place a table that is 90 percent read and 10 percent write on one RAID array while taking another table that is maybe 50 percent write and 50 percent read on another table and then taking the Postgres WAL and placing that on a completely different array.

Table usage varies greatly across large databases. Some tables barely get touched, others get written to alot, others get read from alot.

I'm currently running a database where our peak loads are around 35 queries, per second. I've actually symlinked table locations to put my most heavily accessed tables on a seperate RAID array from the rest of my database. This gave me a 3 fold increase in speed. This is really noticed when we do things like VACUUM the db.

I'm a programmer

I suspect a lot of people here are. To me, and probably to most of them, UML is Unified Modelling Language. Hell, do a google search [google.co.uk] for UML and the top hit is to the UML website [uml.org].

I know it's too much to ask OSS projects not to pick confusing acronyms and names, but I'd like to think that story submitters or at least editors could a little clearer.

Good tools out there for PostgreSQL....

...some on PGFoundry [pgfoundry.org], some still on GBorg [postgresql.org].

PLUG: For example, there's this little SQL query analysis [postgresql.org] utility!

All Welcome and expected - expect more..

This is great news, not only for the projects involved, but for FOSS in general.

Also this is consistent with the Open Source Paradigm. Where it is in the interests of companies to improve the software, and the advantages far outweigh the disadvantages of them not being exclusive. It is this philosophy, in my opinion, that will beat proprietary software models such as Microsoft, and it is these companies that are key in stopping those who want to halt the advancments of FOSS using idiotic patents and other invalid IP arguments.

OLAP still missing...

I'm really looking for an OLAP implementation on PostgreSQL... It's getting more an more important but it's still not there!

I made my company switch from SQL Server to PostgreSQL but now I have to export data every day from PostgreSQL to SQL Server just to get my OLAP reports!

As soon as OLAP is there I'll definitely get rid of SQL Server.

Postgres is kicking butt

and taking names. In addition to Fujitsu's additions, they are also doing point-in-time recovery. They have multiple replication solutions. It's an absolutely wonderful database to develop for.

It's got several really cool features, such as the ability to create your own index types, the ability to create your own column types, the ability to create rules for updating views, and a lot of other things that make it an absolute joy to work with.

The only thing I don't like about it is that it needs the ability to read bytea's as if they were BLOBs. Then life would be perfect!

From Fujitsu's pile, tablespaces is the most interesting feature I see - and that's actually pretty cool. That's one of the things that really allows you to realize the logical/physical separation that relational databases promise.

This rules!

I'm loading more than half a million records into a Postgres db on my iBook as I write this, and I gotta say that pgsql is cool as hell. The data type support alone (polygons?!?!) makes it worth the small amount of extra effort it takes to get it up and running.

Postgres flat blows away MySQL in every way I can thnk of except for the fact that one has to "manually" vacuum (cleanup + reindex) the db ... but that's what cron is for. The only things I miss from my MSSQL days is the ability to do on-the-fly data type changes on columns; this is actually a good thing because now I'm not so lazy about designing the db right in the first place. ;-)

If you're out there playing with MySQL or MSSQL, you owe it to yourself to give Postgres a shot.

Re:This rules!

Just having a polygon datatype is kinda cool in itself, but the fact that PostgreSQL really supports using R-tree indexes and thus make efficient geometric queries quickly and easily is really great.

PostgreSQL is probably the most well-polished and useful open source project there is (gcc being the runner up, I skip linux since there really are plenty of decent OSS alternatives to it). Good going PostgreSQL team!

Nested Transactions

Thank god.

Good news!

Certainly good news! :) PostgreSQL is a very robust and complete database, enjoyed by many academic users (mostly because of its excellent implementation of different SQL standards...) It's nice to hear that a company is backing them up now. UML and Intel, really cool, too. It's not as good as Linus/OSDL, but definitelly equivalent to the Linus/Transmeta years. So, in general, is this the road for the free world now? Backed up by powerful companies who also benefit? I certainly hope so.

More servers running PostgreSQL...

...can be found on the Big List Of GForge Sites [gforge.org].

Props to Tim Perdue for picking a solid database on which to build GForge [gforge.org]!

User-Mode Linux Management

...If you want to manage a lot of UML virtual machines, I _highly_ recommend UMLazi [umlazi.org]. It has a very slick configuration file format-- configuration directories instead of a single file, which makes it really easy to manipulate with scripts--, and they've obviously put a lot of thought into security.

I had a few problems getting it started, but the developers were very helpful.

Why corporate self-interest can be good for OSS

There is more than just Fujitsu supporting PostgreSQL and the reasons there is corporate interest is pure unadulterated self-interest of the best kind.

Postgres is getting really close to the functionality and capabilities of the Big Commercial Enterprise DBMS, close enough that anyone can see that bridging that gap is quite doable. Most of the arguable weaknesses in Postgres are in the more esoteric high-end feature space, as it is already strong and quite feature complete for most routine RDBMS work. And the upcoming new version addresses a great many of those weaknesses. As the article said, this is going to be a major release.

The self-interest part is that it is a HELL OF A LOT CHEAPER for a corporation to pay people to add those last few features and bits that they want to Postgres than to pay an unholy amount of money to buy the required Oracle licenses. The Postgres engine is clean and fundamentally pretty good in an engineering sense, and so enterprise feature tweaks are relatively cheap. It is all about dollars and sense at the end of the day. Purchasing Postgres plus feature development is almost always going to be vastly cheaper than buying Oracle. And unlike Oracle, it is pretty much a one-time fixed cost. It is worth repeating that the engineering strength and scalability of the underlying Postgres platform is the primary reason the market is evolving this way. The gap between MySQL and high-end RDBMS is comparatively much too great for a company to fund closing that gap because a lot of additional arguably unrelated work may be required because of the internals. This increases time to delivery of features, increases the cost of adding high-end features, and increases the risk of problems.

If Oracle suddenly dropped its enterprise licensing costs by a couple of order of magnitude, then it would seriously threaten Postgres development. But since that is unlikely to happen, corporate money will continue to flow into making Postgres a formidable Oracle replacement, which it is already well on its way to being.

PostgreSQL Rocks

yes!

i've been a die hard fan of PG since 6.5-ish

windows port

7.5 will contain a native windows port with no external dependencies. You can find the current binary version here. [postgresql.org]

Even though it is currently in beta it works very well. The port is now being downloaded over 2000 times a week and increasing all the time.

Also in PostgreSQL 7.5 - Native Windows Port

Looks like version 7.5 will also include a native Windows port. Prior to this, PostgreSQL on Windows has always required Cygwin (which offers a lot of great stuff in and of itself) to run.

postgre who?

Trying to make postgre's database like oracle's is definetly not the road to follow, more complexity and features (even tablespaces) don't make a better database, try to follow the simple and modular apporach of mysql and stop stuffing postgre unless you want to make it unbearable as Oracle, that on most common installations need a battalion of people to make it work descently.

that's nice, but

... where are distributed transactions? Is that even planned? That's the one feature I would love to see.

UML...

is a terrible acronym, since it already means Unified Modeling Language. Since User-mode is hyphenated, shouldn't it be UL? Oh, wait, that stands for Underwriter's Laboratory. Oh, well, they need to change the name.

acronym collision

I thought the "UML" in the headline meant that Fujitsu funded "Universal Modeling Language" support in PostgreSQL. I figured that RDBMS designers were smart enough to put a rules processing engine into a layer compatible with their SQL processing. Especially now that there's work underway on "compilable UML". Oh well, back to the drawing board.

PostGreSQL needs online backup

Any idea when online database backup will be implemented for PostGreSQL? PostGreSQL is enterprise-level with this sole exception!

I mean, what 24x7 application does not need to backup the database while concurrently running transactions?

UML

Not to be rude, but who chose the name UML for User Mode Linux?

Do they have a diagram or something I can look at? I want to really understand what User Mode Linux "is".

Still trying to figure it out

I have given UML a try a number of times (I need it right now too!) but I've never figured it out. If anyone happens to have a link to a good HOWTO, whitepaper or guide I'd love to have it. I'm sure others would too.

Performance and replication?

I'm currently using Mysql, and I am willing to switch to PGsql with the many more features it offers. But a few things I want to see some people give some comments:

1. Performance. Specifically, how does it compare with Mysql? Some real app comparison would be very nice.
2. Replication. Last time I checked, there is no mature, native, OSS implementation for master-slave type replication for syncing databases between servers.
3. Windows binary. Mysql is painless to install on Windows, does PGsql have this yet? This is for development use. (production database on Windows is looking for trouble, I speak from experience)

Re:Good to Hear...

the primary DB System for so long has been MySQL. PHP coders don't have too much for an alternative

Au contraire, there are PHP interfaces for PostgreSQL, Oracle, Sybase, and MSSQL built right in to the source distribution. I seem to recall that back in the Bad Old Days before Mac OS X, when you had to compile things yourself, building PHP with all the necessary libraries was a huge pain, but now it's a trivial thing. Marc Liyanage maintains a PHP module package [entropy.ch] that snaps right into the built-in Apache web server on your Mac, and it already has most of the necessary bells and whistles [entropy.ch] built in.

Re:Good to Hear...

the primary DB System for so long has been MySQL.

Care to qualify that statement? Ever hear of Oracle? Or DB2 or SQL Server or Sybase or...?

Re:Hehe...

Yeah, this is flamebait. I can't help it. And you 50 Postgres users love it anyway, don't you?

PgSQL is of my choice for a database.

Superior technology is only understandable by superior elite intellect, as you admit.

Yeah, this flamebait is better than yours.