Data Munging with Perl 66
Data Munging with Perl | |
author | David Cross |
pages | 283 |
publisher | Manning Publications |
rating | 9 |
reviewer | chromatic |
ISBN | 1-930110-00-6 |
summary | Dave explores Perl's unique and compelling abilities tomanage and manipulate data of all types, sizes, and shades. |
The Scoop
Larry Wall, so goes the story, needed to glue together two systems on opposite sides of the country. Calling on the virtues of Laziness (why throw together something for just one job) and Hubris (why not write a new language?), he created Perl. Though it's found new niches in the post-web world, Perl earns its bread and butter munging data.Dave Cross has put together a friendly and handy compendium of techniques, tricks, and best practices. Suitable for raw novices to experienced intermediates, Data Munging with Perl is a gentle but firm romp from flat text, past structured and binary files, to the realm of custom parsers. Clean examples and lots of modules accompany the explanations.
What's to Like?
The book plots a natural course through topics ordered by complexity. It opens with a theoretical overview of data processing. This introduces terminology and outlines the general types of data one might encounger. Additionally, the author writes with the authority of experience when exploring the basic approaches and best practices. While other books aimed at novice users shy away from programs-as-filters and data structures, Cross prefers to instill good habits from the start.Beyond munging data, the book provides a decent introduction to idiomatic and effective Perl programming. While the brief tutorial won't magically produce new JAPHs, the thoughtful and continual devotion to good technique and skill will inspire smarter programmers. More important than knowing many useful tricks is knowing when and how to use a handful of tools -- and where to go for more.
The overall level of quality is excellent. The binary data chapter stands out as the clearest explanation available, and the information on munging dates and times will save readers plenty of grief. Additionally, the entire parsing section introduces a handful of powerful but sorely-underused tools to handle HTML, XML, and even creating custom parsers. Rounding out the curriculum is an appendix that explores the larger modules, mentioned earlier, in more detail (XML::Parser, DBI, Date::Manip).
What's to Consider?
Only two things might turn readers from this book. The first is its deceptive length. While the text is short, the examples are clear and the text packs a lot of wallop in what's there. Careful readers who follow the links to other resources will have little trouble supplementing their education. (On the other hand, another ten pages describing Parse::RecDescent would have been a nice addition. It's hard to fault the author for deferring to the module's voluminous documentation.)Second, longtime Perl programmers may find little new material, particularly if they are familiar with the wealth of modules on the CPAN. The intended audience is clearly new and underexperienced programmers. While there's plenty of good advice presented well, the book falls more toward the tutorial side of the aisle than the reference section. This does not detract from the book, but it does narrow the base of potential readers slightly.
The Summary
Manning Publications continues its fine line of Perl books with the consistent and powerful Data Munging with Perl. Coders looking to transform data somehow and hackers who want to take advantage of Perl's unique features will improve their knowledge and understanding. If you find yourself working with files or records in Perl, this book will save you time and trouble.Table of Contents
- Introduction
- Data, data munging, and Perl
- General practices to use when munging data
- Generally useful Perl idioms
- Pattern matching
- Data Munging
- Unstructured data
- Record-oriented data
- Fixed-width & binary data
- Simple Data Parsing
- More complex data formats
- HTML
- XML
- Building your own parsers
- Conclusion
- Looking back -- and ahead
- Modules reference
- Essential Perl
You can purchase this book at ThinkGeek.
Re:The power of paper? (Score:1)
Why is this important? I've never gotten a good job out of an ad. I only ever have gotten good jobs by friends referring me to other people. For those of you who've been around in the industry for a while, you know this is true...
Re:good to see (Score:1)
Agreed. I love Python, but I still use Perl for data munging. It is without a doubt the best text processing tool available (and that's not exactly a niche market either).
Re:Perl vs. Python (Score:1)
Sheesh.
Re:The power of paper? (Score:1)
Question (Score:1)
Just wondering....
--
Re:The power of paper? (Score:1)
The original contention was that texts online are FREE, so buying a 2nd monitor (vs a book) doesn't support that argument at all.
Re:Question (Score:1)
Perl needs a Parser::Pony::Postings module...
(Hi Dave... :)
Re:Question (Score:1)
Re:The power of paper? (Score:1)
BTW: This two minute posting limit is REALLY annoying!
----
Re:yep, it's a good book but not the CookBook (Score:1)
The Cookbook is in dire need of a new edition. The latest print edition is circa 1998 and there are about 25 pages of errata. My recommendation is for everyone to hold off buying O'Reilly's Perl Cookbook until they release a new edition.
Re:The power of cntrl-F (Score:1)
Re:Question (Score:1)
Re:The power of paper? (Score:1)
Re:yep, it's a good book (Score:1)
--
good to see (Score:1)
--
Re:Boycott This Book!!! (Score:1)
Nice internal conflict you have there...
So who should judge who can get information and who cant?
books are fine... but... (Score:1)
Re:yep, it's a good book (Score:1)
Re:Answer by example (Score:1)
Re:The power of paper? (Score:1)
you go ahead and buy the monitor. I'll buy the book, tell my boss I'm researching my latest project,and you'll find me out in the park by the lake with my laptop while you're stuck in your office...
Oh,and you might want to go research the retention studies that compare how well people remember what's learned on a monitor vs printed material. I'm sure there are some nice references online (I know I've seen them, but I've forgotten where...)
Follow that link (or was it kink...) (Score:1)
(Error 404: File or resource not found)
Re:So let me get this straight... (Score:1)
Re:So let me get this straight... (Score:1)
Re:Answer by example (Score:2)
Or even, *cough*, MS-Word files. A better example of an unstructured file might be a straight ASCII text file (although nitpickers might point-out a straight text file could contain structured elements; i.e., SGML, XML, HTML).
Re:Question (Score:2)
Re:Answer by example (Score:2)
Word processing files are unstructured.
Not if you write them properly, for example, if it's a heading, actually set it to be a heading, don't just make it bold. Good word processors support this, and that's why you can use them to write books and legal documents, which need to be maintained and updated just as programs do. Word even has built-in version control!
Manning should choose a new cover theme (Score:2)
Don't judge a book by it's cover, I know, but I still think they'd sell a lot more of their books (I have two, possibly three), if they'd get some kind of cover scheme which displays anything other than those fruity portraits they're using now.
Cheers,
RIAA and Perl Data Munging (Score:2)
Kevin Fox
--
Re:Answer by example (Score:2)
Re:Boycott This Book!!! (Score:2)
I thought the GPL [gnu.org] implies that you shouldn't discriminate who gets to use your code, and I thought that the Open Source Definition [opensource.org] explicitly says that one "[5] must not discriminate against any person or group of persons [and [6]] must not discriminate against any person or group of persons."
I'm looking & looking, but I just do see anything anywhere about it being okay to only pay attention the parts you find convenient or expedient. Maybe you can point me in the right direction here?
In the meantime, this is for me, a non-spammer, regular working shmoe, a very educational & useful book. I'm not gonna support a boycott of it just because it doesn't jibe with your situational ethics...
Re:Question (Score:2)
Ascii is so structured that decoding it is trivial. Unicode is still structured, but not as trivial. Mp3 might be much more hairy, and then the ones you describe almost sound like "meta-formats", which provide a framework for bundling other formats together -- thus leading to high level structure & low level disorder, or at least complex & hard to decode order.
Re:The power of cntrl-F (Score:2)
So when I'm thumbing through my book, looking for a specific phrase, I have to algorithmically and frantically scan every page in the book. This also introduces the risk that my feeble organic eyes might actually miss the word.
I prefer using my browser's control-F "find" feature, or grep, or what have you to pick out the key word(s) of my current interest.
The magic of a book (Score:2)
Reading a book is something that'll mentally bring you back into your inner classroom, just like smelling a box of crayons will bring you back to your inner child. Make no mistake about it, there's definately magic in dem dar books.
Re:Boycott This Book!!! (Score:2)
Re:The power of paper? (Score:2)
"Death Of Books Imminent: Film At 11"
Personally, I mark up my books all the time. You know, with ink. Like when I made a big red circle & arrow pointing to the part in Unix In A Nutshell that reminds me that for join to work, the input files have to be sorted.
--
Re:Unstructured data (Score:2)
Unfortunately, those are the operations that are in the inner loop of a tokenizer for any formal language (C, HTML, etc.)
A fast built-in that returned the numeric value (as ORD does) of character N of a string would be clunky, but would provide a way to speed things up without going outside Perl. The Perl-ish way for doing such things involves regular expressions, and that's an example of "if the only tool you have is a hammer, everything looks like a nail".
Re:Unstructured data...FOOL (Score:2)
Theoretically, a highly optimizing regular expression compiler that looked at multiple statements containing regular expressions could generate an efficient tokenizer from such Perl code, but that's not what's inside the Perl engine.
Re:The power of paper? (Score:2)
A good computer book provides thorough end to end coverage of a subject (a great one lays it out in a way that is easy to understand and possibly fun).
--
Re:Question (Score:2)
Well, that isn't correct. XML is a markup language (eXtensible Markup Language) which very well can be used to repressent both structured and unstructured data.
It all depends on what query language you use for XML, there are serveral different right now.
For unstructured data (parsing and transforming documents) XSL [w3.org], XQL [w3.org] etc are useful.
For structured data, check out XML-QL [w3.org].
Re:Question (Score:2)
The opposite, unstructured data, is simply when the data is not related to other parts. As someone mentioned, a plain document is a good example of unstructured data.
Re:The power of paper? (Score:2)
These are personal reasons though, I guess different people have different requirements.
Re:Question (Score:2)
I have seen a LOT of wierd data in my time...
Re:The power of paper? (Score:2)
Re:Question (Score:2)
Re:Manning should choose a new cover theme (Score:3)
Although I think "Data Munging with Perl" is probably less usefull to me, and I probably wont get it, the cover is cool.
By your use of the word "fruity" I take it to mean you are afraid of looking queer. Not to worry, I'm sure you already do.
Re:Unstructured data (Score:3)
Effective Perl Programming (Score:3)
ISBN: 0201419750
Publisher: Addison Wesley (1998)
Fun collection of Perl idioms and some good stuff on h2xs.
Re:Boycott This Book!!! (Score:3)
munge
information. 2. A comprehensive rewrite of a routine, data structure or
the whole program. 3. To modify data in some way the speaker doesn't
need to go into right now or cannot describe succinctly (compare
{mumble}). 4. To add {spamblock} to an email address.
In this case Dave means 'doing stuff with data' akin to the Jargon File's third definition of the word.
--
Niklas Nordebo | niklas at nordebo.com
Re:The power of paper? (Score:3)
Books don't require batteries that might run down, or suffer from any of a dozen other complaints against the "true portability" of electronic systems for getting documentation.
If you're happy with phosphors, more power to you, but if I want a reference, I want one that is as portable as I am; one without leashes to the power grid (even if they're only intermittent), and one with some editorial intelligence up front to filter down to the topics I care about, rather than a kitchen soup like the web where I have to sift through 10000 google hits to find the page that really answers my question.
Answer by example (Score:3)
Re:Unstructured data (Score:3)
The people who create and maintain perl are smart enough to realise that no tool is universally useful.
Mixes of C and perl simply require the appropriate compiled
You can precompile a binary
If ease of distribution is paramount, write the parser in C, embed a perl interpreter in it and code the perl portion appropriately.
Re:The power of paper? (Score:3)
I figured the use of "tomb" was intentional. After all, it's where you put dead trees.
Re:Unstructured data (Score:3)
Yes. It's embarassing that Perl needs help from C to ... manipulate strings.
(I don't want to use that because the mix of C and Perl makes portability more difficult. All-Perl code you just put in the right directory and run. Mixes of C and Perl require compilers, package managers, makefiles, and installers. The target is shared-hosting services, where users may not have shell access. It's seriously annoying that Perl does this simple operation so slowly.)
Physical quality of Manning's books? (Score:3)
Re:The power of paper? (Score:4)
The best part about a book -- a well written book, not a "How to Be and Unleashed Dummy in 21 Days" book -- is the time and care put into it by a host of professionals, whereas a Web resource tends to be cobbled together from a community of geniuses and idiots alike.
Look at Slashdot -- some of it is great, some of it would wither a pile of dog poo it's so bad. php.net is similar -- the function reference is good if you're looking for arguments to a rarely used function, but the user-contributed stuff is off-and-on useful.
That's partially why you pay $50 for a good tech book -- the team of people needed to put together a *good* book is quite expensive. You need a knowledgeable author, a clued-in editor, a savvy fact-checker... all these people cost money.
"Beware by whom you are called sane."
Re:Question (Score:4)
yep, it's a good book (Score:4)
Like Chromatic, though, I really wished that the section on Parse::RecDescent had been longer...
Unstructured data (Score:4)
We have an engine which processes such data, but it's slow, because it's in Perl. Most of the time goes into modules recommended in this book, like HTML::Parser. The big problem is that simple tokenizing, like extracting HTML tags, is incredibly slow in Perl. The classic "get next character, get character class for character, switch on character class" operation is something Perl does very badly.
Yes, you can write low-level C functions and call them from Perl to deal with such problems, but that kills portability.
Re:The power of paper? (Score:4)
One thing that I haven't seen mentioned yet is that books are easier to read than monitors. Monitors just can't match a book's DPI, and the higher resolution of the printed page can actually improve reading speed and retention and reduce eye strain. That may or may not be a big issue for you, but it can be a big deal and a reasonable justification for the extra expense. Another advantage of a printed book is that the author has already gone to the trouble of cobbling together the data for you so that you don't have to spend your time scrounging the web for it; if you're a consultant getting paid $100 per hour it doesn't take much time scouring the web for information to add up to more than the cost of the book.
OT Note: the correct term is tome (from the Greek word meaning to cut, and the same root as in medical procedures ending in -otomy, as tomes were originally produced by cutting a long scroll into smaller sections) not tomb (which is where somebody is buried).
Boycott This Book!!! (Score:4)
Books like this give corporations the tools they need to destroy our privacy and strip us of our rights. How do you think Double Click puts the information about you it sells into useable form? With techniques it learns from this type of book. Same goes for the corporate websites you visit, your supermarket, etc.
Information wants to be free, but not the information in this book. Data mining and Data munging techniques should never have left the hallowed halls of academe. Once they enter the public domain, they are immediately exploited by greedy corporations. The author should have thought about that before writing a book like this.
If you buy or support books like this, you have lost any right to complain about your privacy being violated. If you are serious about privacy, boycott this book!
Re:The power of paper? (Score:5)
--
Re:Question (Score:5)
Log files are generally fairly structured data.
CSV files are structured data.
Free flowing ASCII text is unstructured data. :).
Shakespeare's sonnets, however well formed, are unstructured data
(unless you can come up with a parser that recognizes iambic pentameter...
Falling somewhere in the middle is binary data. It has a structured format but freeform contents. Consider the various sound, image, and video formats. Maybe Shakespeare's sonnets could fall into this category too... :)
There are situations where you could want to analyze each form. Parsing Apache log files is a slightly different task than analysing formal XML documents or sloppy HTML pages or messy ASCII email. This book helps give you a feel for which situation you may be dealing with, and thus what tools & techniques might be useful for that situation.
Though some will tell you otherwise, this book has nothing to do with "Buffy the Vampire Slayer." Sorry, grep.
Re:yep, it's a good book (Score:5)
Next year I'll be writing a book about Parse::RecDescent (or its successor Parse::FastDescent) and grammatical parsing techniques.
Damian
The power of paper? (Score:5)
Perhaps I'm still stuck in the paper age (somewhere between bronze & silicon), but I find myself spending $50 a pop for progamming books I only skim through. If I need reference material, I hit PHP.net (for my PHP projects [ridiculopathy.com]).
Am I missing something?
Re:The power of paper? (Score:5)
Art At Home [artathome.org]
Re:Boycott This Book!!! (Score:5)