F/OSS Flat-File Database? 702
Leemeng writes "I'm looking for a simple, free, and F/OSS flat-file database program. I'm storing info about Wi-Fi access points that I come across, maybe 8-9 fields per entry. I've outgrown Notepad. This info is for my own reference only; it is not going on a Web server. Googling was unhelpful, with results skewed towards SQL, Access (MS), and Oracle, all of which would be overkill for my purposes. My criteria are: it must be simple, F/OSS, must work in Windows Vista, preferably use a portable format, must not be an online app, and must not require Java. Does such a beast exist?"
Python? (Score:5, Informative)
Can't be Java... well, how about Python?
Here is [ideaspike.com] a completely free (PD, not GPL-style "you're free to do as we tell you") database engine that will do what you have described thus far.
The database engine is about 19k bytes (not a typo), has no dependencies (other than Python itself), supports a useful subset of SQL so you can actually create flexible queries that produce well-sorted results from your database, and it works everywhere Python does, which is to say, it works pretty much everywhere. It's just as happy operating on a command line as it is on a web server. The results (the actual databases) are 100% portable from OS to OS. I use it on various linuxes, OS X, and Windows for tasks very similar to yours.
Comes with tutorial examples, sample databases and extensive docs. In a 13k (not a typo) archive.
Re:Python? (Score:5, Insightful)
Re:Python? (Score:5, Informative)
That's all I have for ya, offhand. ;-)
Re:Python? (Score:5, Informative)
Re:Python? (Score:4, Informative)
Other than that the dbtxt looks very nice indeed.
Re:Python? (Score:4, Insightful)
In what specific ways does the supposed "bloat" of SQLite affect the user, exactly?
What kind of "maintenance" does the user need to do on the SQLite source code? It's maintained well enough for some some pretty big users [sqlite.org]. Who maintains your code and why should I trust you with my data?
How can SQLite be "difficult to use" when it's just standard SQL with a standard Python DBAPI interface?
It's really nice that you wrote a cute little datastore. But there's no need to badmouth what you perceive as the competition.
Re:Python? (Score:5, Insightful)
IOW, SQLite, as "lite" as it is compared to Oracle, MySQL, PostgreSQL, etc. is still overkill.
I'm with the GP -- dbtxt looks really cool. Although, I do gotta say that OpenOffice.org would fit the bill just fine for what his requirements are: Calc can be used as a flat-file database, and I think there is support for simple CSV databases in Base.
Re:Python? (Score:5, Interesting)
Second, I'm not sure that you know any better than I do what the poster really needs. Open office calc might be an option, but certainly not if 800k is too much space; open office is what, like 200 megs? And dbtxt might also be an option, if doing everything in Python is what the person wants; I love Python, but there are plenty of people who don't. I'm sure that all three proposals are equally valid.
Now, to defend SQLite: in most cases, dealing with a relational database is "overkill" not because of SQL per se, but because of the fact that software needs to be installed and configured*, and because complex sets of processes need to be managed and maintained. SQLite is useful, in part, because it provides the functionality of a relational database without the complexity of the engine. Now, if you don't want or need SQL, then you might not consider that particular feature to be of particular benefit. Nonetheless, there are many situations where SQL might not be required, but where it might be useful, where in the past SQL might not have been used, but where now, because of SQLite, it will be used. There are very few cases where real-world information is not most accurately represented as relational data (in comparison to flat data). SQLite allows you to avoid making compromises regarding the way that that data is stored. And SQLite is fast.
*(Although some interpreted languages might require that sqlite's libraries be installed on a system for it to be used by a program, that is not the case for compiled languages like C, C++, Obj-C, etc., where the sqlite libraries can be compiled directly into the code distributed to end users.)
Re:Python? (Score:4, Funny)
Re:Python? (Score:4, Informative)
Re:Python? (Score:4, Informative)
Re:Python? (Score:5, Informative)
Re:Python? (Score:5, Funny)
That'd make the first one a shameful link, yes?
Re:Python? (Score:5, Interesting)
Sure, it can be. For instance, adding 800k of stuff to your application means that you are giving up control. You might have the source (which is going to be somewhat larger in this case)... but do you have the time to dig into it if there's a problem? Does reporting the problem necessarily mean it's even going to get fixed? For instance, I've reported many problems to various authors, some of which got fixed, some didn't. Likewise, adding a feature can be non-trivial with a large (source code) project. Compare that to a very lightweight thing that, let's say, almost does what you want but you just need feature X. Might be a whole lot easier to add that to a 20k "almost-there" set of source than megabytes of source, is that not possible?
Distribution of a class module with a python app is an entirely trivial thing to do. For that matter, you can take the class and put it in the same file with your application if that appeals, giving you the advantages of atomic distribution -- harder for an end-user to separate things and make whole app, as you handed it to them, fail to work.
And I suppose the elephant in the room is that the end user may have no need for the features. I sell a commercial program with features not available in other programs in the same general application space, functionality-wise; that doesn't mean it will sell to anyone on that basis unless they actually need those features, right? In this specific case, based on the article, the needs described seemed to be met by what I'd written, and hence my post.
PostgreSQL whips the living hoo-ha out of SQLite; it is larger, but offers more features (sound familiar?) Yet SQLite has a place, as your advocacy clearly indicates. I would venture that the difference between SQLite and class dbtext is similar, just occupies another place on the size/features scale. In the end, the user, based on their requirements, may go "what a POS" and step up the ladder. From either one.
dbtxt also seems to be very stable. It's never failed for me, nor have I ever received any indication that it wasn't doing what it was supposed to do when used as indicated. Also -- as a user myself -- I'm quite interested in fixing it, should it turn out to have problems.
Also, just as an aside, If someone can explain cross-platform file locking in python to me in terms I can understand, and if python's ability to lock can be made to implement a reasonable form of "wait for lock to release", I'd like to add that as a feature, too. I found python's docs on locking to be more than a little opaque. I'm sure it's just me.
Re: (Score:3, Informative)
It's not just you. Python exposes the locking mechanisms of the underlying platform and doesn't try to abstract any further than that.
If you want simple, portable locking, use this recipe from ASPN:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65203 [activestate.com]
Re: (Score:3, Informative)
Wait a sec... That's not reliability; that's confidence. Reliability is a performance metric; confidence is not. I suspect — looking at some relevant SQLite pages [sqlite.org] — that dbtext might actually be more reliable, because my bug list is at zero and has never moved off zero since release, though it is used thousands
Re:Python comes with SQLite (Score:5, Informative)
import sqlite3
mydb = sqlite3.connect('sample.db')
mydb.execute("create table contacts (fname text, lname text, email text)")
mydb.execute("insert into contacts values('Spooky','Monster','spook@spammity.spam')")
mydb.commit()
mydb.close()
You can then use the free and open SQLite database browser [sourceforge.net] to browse, edit, and print your table.
You may think you're keeping it simple by using a flat file, but you're really not. It may be somewhat easier to manually edit, but it's also easier to screw up, and I've never heard of one with the ability to undo changes.
Re:Python comes with SQLite (Score:5, Interesting)
Class dbtxt includes the ability to undo changes, and features human-readbility for the flat database files. It's trivial and not error-prone at all to mod a database file to delete, or undelete a record, and of course you can do it through the database engine as well.
Re:Python comes with SQLite (Score:4, Interesting)
set first "Spooky"
set last "Monster"
set email "spook@spammity.spam"
# In practice it's really easy to put values into variables
mydb eval {insert into contacts values (:first,:last,:email)}
The advantage? That code is now totally armour-plated against SQL injection attacks as well as being fast. Which is nice, really really nice.
Re: (Score:3, Informative)
I don't know if the python bindings are of that same quality; if they are, they're top notch.
The sqlite3 module as shipped with Leopard says that it supports question mark substitution, so his example would be better written like:
BTW, the same could be written for PostgreSQL like:
Hopefully sqlite3 will get the same standard substitution in the fu
Re: (Score:3, Informative)
Actually, the Python SQLite adapter supports the standard Python DB API (PEP 249 [python.org]), and so you can just follow its recommendations (which includes using placeholders).
Re:Python comes with SQLite (Score:4, Insightful)
2) He's explicitly mentioned several times that this is for his own private non-web use.
3) Your fly is open.
Why are we trying to promote python? (Score:5, Informative)
He doesn't need python. He just needs a database. He can download a precompiled binary for windows [sqlite.org] that allows one to work with the database at the command line [sqlite.org]. Python is not necessary.
And if the command line is too much, as others have noted there is already a convenient firefox extension [mozilla.org] for graphically interacting with a sqlite database.
Re:Python? (Score:5, Interesting)
I think is exactly what you are requesting.
Re:Python? (Score:5, Funny)
Next!
Re:Python? (Score:5, Interesting)
If it were not for the Flat File requirement then a simple Python shelve or Perl Tie would be the most logical solution since they are both part of the standard library so don't require installing libs on random computers you might use or port to.
of those two Perl Tie is probably the most suited because it's backed by a real DB operating off the disk not fully in memory.
But why not do both: use YAML as the DB backing the Perl Tie.
then you get a nice human readable flat file.
Re:Python? (Score:4, Interesting)
Even if you don't end up using it for this particular app, it definitely deserves a look. Although it'll never displace XML, it definitely answers at least some of the questions XML attempts to answer, reduces "tag bloat," and is easy on the eyes.
Spectate Swamp Desktop Search (Score:3, Informative)
Re: (Score:3, Informative)
No, it isn't just that. It is used to show that what you are quoting is verbatim, for instance in cases where an assertion is unique, questionable, or worded unusually. This also works for obvious spelling errors, of course.
Re:Python? (Score:5, Funny)
Re:Python? (Score:5, Funny)
Re:Python? (Score:4, Funny)
Re:Python? (Score:5, Insightful)
I can't legally utilize GPL'd source code within a commercial application without doing some very specific (and not always possible) things that the GPL license instructs me I must do. Any other ways I can help with your basic F/OSS education? I'll be here all day.
Re: (Score:3, Insightful)
Re: (Score:3, Informative)
Yes, that's pretty much what I said, only with more details. What's your point?
Re:Python? (Score:4, Insightful)
Your imagination, not to put too fine a point on it, has failed you.
Yes, in fact, it does.
I have come across a snippet of GPL'd source code. I want to use it in my gigabyte of source code commercial app, which, prior to such inclusion, is legally mine to distribute, which is a use I make of this object, that is, my application. However, as quite common in commercial efforts, I cannot legally distribute the source code of my application. So, one day I use the GPL'd source code by compiling said GPL source into the commercial app. I mod it to better suit my use of it. My app is suddenly subject to a change in usability, imposed by the GPL, such that I no longer have the option of using the same way (which act can also be characterized as distribution) to earn money, and my customers no longer have the option to use my altered application. This viral alteration of the usability of my application is entirely courtesy of the GPL. In order to reverse this alteration and keep the code snippet in use in the application, I must perform actions that I actually do not have the option of choosing due to previously existing obligations (essentially source code distribution.) The end result is that the GPL'd code cannot be used if the commercial application is to remain usable.
To be blunt, the word "use", (as well as all of its natural permutations) which you depend upon so heavily in your flawed analysis, doesn't have the limited meaning you wish it had — nor has it ever, or will it ever, have such a limited meaning. This is why your argument is specious.
Re:Python? (Score:5, Insightful)
Why should the GPL hackers share with you if you aren't willing to share with them?
You don't like their license. That's fine, they don't like your license either.
Re:Python? (Score:5, Interesting)
Re:Python? (Score:4, Insightful)
Yes, you missed the point. You can do it with PD software, specifically with the software in the post that this thread depends from (class dbtxt.) PD source allows you more freedom(s) at the user end. GPL source allows you (a lot) more control at the author end.
Re: (Score:3, Interesting)
There is a philosophical position that I think is important. The term "share and share alike" is something that most kids have learned in childhood for centuries for the greater good of society. The GPL simply puts that in the license.
I am a capitalist, seriously, I honestly think that real people deserve real compensation for their work. Actually, let me rephrase that, people have the righ
Re:Python? (Score:4, Insightful)
It's my code; I chose to make it PD; and I do get something for it, I get to add a shoulder to the wheel, as it were, if anyone will accept my contribution... and I enjoy the idea that the time I spent saves someone else some of theirs, hopefully such that they can create something above and beyond what I already did. Sometimes that's enough. I've gotten lots of help off the net, no strings at all -- so I don't find it at all peculiar to offer something back likewise sans strings.
Some works we contribute; some things we ask a return for. I'm ok with both approaches, really, it is the ones in the middle (supposedly "free", but with substantial strings) that don't attract me either as a user or a contributor. That's just me. What seems to rile people up is if I talk about the reality of those strings. That's when we get these threadfests.
Re:Python? (Score:5, Informative)
Re:Python? (Score:5, Insightful)
IMO, the people who still, after all these years, haven't figured out how to make money with GPL software are the failures, not the GPL.
Its just a software license; it doesn't preclude you from charing money for your work, and it doesn't mandate you make your work avaiable to the general public. I can't fathom why some seemingly intelligent people have such a hard time grasping this concept.
Re: (Score:3, Interesting)
Just because you don't necessarily want/need to make it public doesn't mean you want to distribute it as binary-only or not be able to use other GPL code in the development or expand it in unforeseen ways in the future.
I'm not a programmer, but in one of my past jobs I used to hire developers for projects. One of the caveats I applied to everything we did was that it had to be GPLed, partly for flexibility down the line if we needed to mod
Re: (Score:3, Interesting)
Re:Python? (Score:4, Insightful)
Proprietary licensing is specifically designed to make the licensee dependent on the licenser. If there's bugs in the software then the licensee can't fix it and there is only one place the licensee can go to get the bugs fixed, the licenser, who is free to refuse to fix the bugs or demand exhorbinate fees to do so. Same with new features or support.
If you wouldn't do this to someone directly then giving code to someone else so they can do it is unethical.
Re:Python? (Score:5, Insightful)
To be fair, a proprietary licenser is also free to fix the bugs ASAP and well, provide such fixes for free, also to provide new features for free if they like. I do this all the time with a commercial application. A PD author also retains such freedom. These are not benefits that exclusively arrive via the GPL. And it is well to consider that with a GPL, PD or commercial application, one is free to demand money to fix something, to add features to it, to explain or teach its use, or even to simply use it. Doesn't mean you'll get it, but it doesn't mean you won't, either.
I don't think you've really analyzed this far enough. Giving someone something, and then telling them what they can and cannot do with it, carries ethical problems in the form of imposing your will on someone else (and it also devalues the gift, in my personal opinion.)
"Here's a gift of a book; but you can only read it if you use LED lights." "Here's a gift of a cat; but you can only have it if you'll eat it." "Here's a gift of a some money, but you must spend it upon me."
You see? This is why the term "freedom" has always seemed to go so poorly with the coercive requirements that the GPL applies to the various recipients in the chain of "gifts."
Re:Python? (Score:5, Funny)
Re: (Score:3, Insightful)
You're more than welcome to use with any other software, you just can't distribute.
Re: (Score:3, Informative)
Use it in any project that isn't GPL'd.
Agreed - this is a particular problem when it comes to library code that folks have either intentionally or ignorantly GPL'd. If I'm not modifying the library code, just using it, I don't want to have to jump through hoops on the calling interface. I don't want to have to build as a shared library with a runtime link. It's a pain, and an unnecessary headache. I'm more than happy to help distribute the source I built against, but I don't want it GPLing my app in the process.
So, the freedom I want is the
Re: (Score:3, Interesting)
Did I say it was unreasonable? It's your code, you're free to put any conditions you want on it. I encourage you to do so. PD, GPL, BSD, commercial, etc., that's your call. With these choices come consequences to the end users of the application and/or the source code, though, and there's no squirming out of them.
What I said was that the GPL imposes "you're free to do as we tell you" conditions while holding up PD in contras
No Java? (Score:5, Insightful)
I would have recommended HSQL, but you don't want Java. Frankly, usually, when we're talking databases I won't say "use a spreadsheet", but with 10 fields, you might as well use a spreadsheet. Of course OpenOffice.org Base is out, because it uses HSQL.
Something like CSQL [sourceforge.net] might fit, but I have no experience with it.
Re: (Score:3, Insightful)
Must not be "online" -- does that rule out web apps? Even if they'll run on your local machine?
And why no Java? Can't be for efficiency, otherwise you wouldn't be running Vista. Maybe something's broken with your Java?
Also, what are the required features? You may have outgrown Notepad, but have you outgrown vim, grep, and friends? That's probably why there's not much F/OSS for this kind of thing. Just a guess.
Re:No Java? (Score:5, Insightful)
This guy wants a spreadsheet, he just doesn't know it. Excel--or the free alternative from OpenOffice.org--will do everything he could possibly want, and although it saves as its own infernal file format, it exports competently into a tab or comma-delimited format.
All the extra "requirements" are just pseudo-intellectual mumbo-jumbo that have no bearing on reality.
Err ... (Score:4, Informative)
Otherwise what's wrong with a simple database like MySQL or PostgreSQL on your computer?
Re: (Score:3, Informative)
For example, if you ask it to read and write this:
May 20,5/20,1.000
you get this:
20-May,20-May,1
That might be okay if those were really two dates and a number, but I was never asked if they were. (OOO Calc is just as bad: it asks, and by default modifies the data).
Re:Err ... (Score:4, Informative)
Smith,John,703-555-5555
into
Smith,John,-5407
The latter being 703 minus 555 minus 5555.
when importing a CSV file. No joke. There are ways around it, but the default is pretty braindead.
Re:Err ... (Score:5, Funny)
Re: (Score:3, Insightful)
I work at an investment firm, and we get CSVs from our vendors with SEDOLs in them. Excel will helpfully (and silently) strip the leading zeros on UK securities, for example. Oops.
Re:Err ... (Score:4, Informative)
That's what I do in Excel 2003, anyway, to stop it from stripping the leading zeros in SSNs.
Re:Err ... (Score:5, Insightful)
OOO? (Score:5, Informative)
Re:OOO? (Score:5, Insightful)
Re: (Score:3, Informative)
I was using it about a year ago and I found it buggy and hard to use. I was using it to access
PostgreSQL.
I like the other parts of OpenOffice that I use, Write, Calc, Draw.
Of course, I have not had time to look at it since then.
Re: (Score:3, Informative)
So really it's 3 main problems are being buggy, hard to use, slow* and totally bloated
Actually it's not half as buggy as it was a few releases back - I can actually use it now to fill in a form without it crashing. Just tried opening a windows XP made odb file in Ubuntu (OOo 2.4) and it's slow as hell and bugged to death, so you might add portability to that list of main problems.
[Yes I've filed b
sqlite (Score:5, Informative)
I find it amazing that you didn't come across it in Googling...
Re:sqlite (Score:4, Insightful)
Re:sqlite (Score:5, Informative)
https://addons.mozilla.org/en-US/firefox/addon/5817 [mozilla.org]
Requirements (Score:3, Informative)
SQLite (Score:5, Informative)
There are GUI clients that work fine for this sort of thing, SQL is simple for doing basic things. One file, one database.
Re: (Score:3, Informative)
how about... (Score:4, Insightful)
Umm... just write a few one-liner perl scripts to get info out.
sed, awk, and grep... (Score:4, Informative)
heh (Score:5, Funny)
Perl DBFile (Score:3, Interesting)
Easy, fast, simple.
Flat file is useless (Score:5, Insightful)
Re:Flat file is useless (Score:4, Insightful)
Well... XML is human readable in the same way as a sand is edible.
If the OP wants to use a tool better than notepad for editing, he probably should try Open Office.
Re: (Score:3, Funny)
<ssid>MyNet</ssid>
<encryption>
<type>WEP</type>
<key_size>128</key_size>
<key>0xAAAAAAAAAAAAAAAA</key>
</encryption>
<channel>11</channel>
</entry>
Very human readable.
Re: (Score:3, Funny)
Spreadsheet (Score:4, Informative)
SQLIte or BDB (Score:4, Informative)
BDB is *not* a relational database though, it's just a storage/indexing engine, which is used most notably by MySQL as a backend. SQLite on the other hand is a full file-based RDBMS with a small runtime, so it might be a bit of overkill for you in this particular scenario.
But both of them run on Windows, Linux and the BSDs, so you won't have portability problems. And most languages have bindings for them.
Uhh (Score:3, Interesting)
If you are just doing tabulated data in a piece meal form, Excel with an Access back-end will do the trick, I'd be willing to be you're going to be able to find templates that help you start. You could also use File-maker (which is overkill, but easy to use) and go to the user community for starting templates.
A CSV File? (Score:3, Interesting)
Depends what you want to be able to do with your data? If it's just quick searches to find local wifi points then it seems overkill to use a multi-platform/FOSS database
GDBM (Score:5, Informative)
If that doesn't satisfy your need, take a look at Berkley DB. It offers a more sophisticated interface than DBM.
What about XML? (Score:4, Interesting)
Read the XML records into an STL container for easy access.
How about a spreadsheet? (Score:3, Informative)
I recently moved my collection of serial numbers out of a defunct proprietary program and into a spreadsheet - couldn't be happier.
Try Metakit (Score:4, Informative)
Re: (Score:3, Informative)
For those not familiar with metakit,
it straddles a very interesting (to me at least)
line between "some csv files i threw together" and "full on sql db" in terms of it's storage
and query semantics, while at the same time being
a flat file like sqllite.
And it really deserves to be better known from a
theoretical standpoint, in how it structures the
data... especially some of the more interesting
applications it's been put t
For your reference only (Score:5, Insightful)
So why not simply write out a tab-delimited file? Retrieval is as simple as reading every line, splitting by the delimeter, and regexp'ing whichever field you're examining. 15'000 records takes about two seconds on a modern machine. Of course, for anything more complicated, that same file gets easily imported into your favourite spreadsheet application.
You don't need to worry about locking because you're the only one using it. And otherwise, your application simply locks the file handle, or creates & destroys a traditional lock-file.
It's a thirty-line perl script (of legible perl). You can do it in JScript as a local HTA if you want the benefit of html etc. interfaces.
Consider an xml-based db (Score:3, Informative)
eXist [sourceforge.net] is one alternative; while I haven't personally used it the home page indicates it's a fairly capable project.
Sedna [ispras.ru] also appears to be feature-rich.
There was a similar discussion on Slashdot specifically with reference to XML databases, here [slashdot.org].
Happy hunting -
How about a full Oracle Database 11g Enterprise (Score:4, Funny)
PCFile (Score:3, Informative)
http://www.umich.edu/~archive/msdos/database/pcfile/ [umich.edu]
Mind you, you'll have to toss Jim Button $10 as it's shareware!
Sounds like he wants an APPLICATION not a framewrk (Score:4, Insightful)
It is still a database system but.. (Score:3, Informative)
Otherwise, I agree with most posters who say just use a simple text and/or xml file if the data volume is relatively small. That should be more than enough!
SixD
Kexi (Score:3, Interesting)
http://www.kexi-project.org/ [kexi-project.org]
You really do NOT want a DBMS, use a spread sheet. (Score:5, Interesting)
use a spreadsheet (Score:3, Insightful)
Berkeley? (Score:3, Informative)
Re: (Score:3, Informative)
Re: (Score:3, Interesting)