How To Build a TimesMachine (nytimes.com) 41
necro81 writes: The NY Times has an archive, the TimesMachine, that allows users to find any article from any issue from 1851 to the present day. Most of it is shown in the original typeset context of where an article appeared on a given page — like sifting through a microfiche archive. But when original newspaper scans are 100-MB TIFF files, how can this information be conveyed in an efficient manner to the end user? These are other computational challenges are described in this blog post on how the TimesMachine was realized.
OCR? (Score:3)
Seems like this is the obvious choice.
Maybe just the headlines or the first paragraph and then link to a compressed version of the image file or PDF (not the TIFF itself for Jebussake).
Re: (Score:2)
Sounds like a good enough concept to make 2 movies...
Re: (Score:2)
Re: (Score:1)
Also interesting are the ads.
Will they be censoring old writings that might, umm, trigger the modern sensibility?
Re: (Score:2)
Something like this has already been done in New Zealand. I don't see what the big problem is. As previously suggested, just have a low-resolution scan of the original text displayed next to a plain-text version.
Here's an example of the original text when I searched for a street address near to me:
http://paperspast.natlib.govt.... [natlib.govt.nz]
And here's the plain-text version, complete with OCR errors:
http://paperspast.natlib.govt.... [natlib.govt.nz]
As a bonus, they both have the searched-for text highlighted as well.
Re: (Score:2)
It's obviously more than just OCR, because when you mouse over the page it will hightlight each article you're looking at, however oddly that article was fit into the page (including highlighting associated pictures, etc). That's actually a pretty cool site, in fact that archive might even be a better reason to get a subscription than the actual current edition.
The free preview doesn't give you much, but it does show the issue from February 1, 1979. In the top right is a story about Ayatollah Khomeini arr
Re: (Score:2)
It's obviously more than just OCR, because when you mouse over the page it will hightlight each article you're looking at, however oddly that article was fit into the page (including highlighting associated pictures, etc). That's actually a pretty cool site, in fact that archive might even be a better reason to get a subscription than the actual current edition.
The free preview doesn't give you much, but it does show the issue from February 1, 1979. In the top right is a story about Ayatollah Khomeini arriving back in Iran from exile, which is interesting to see the contemporary story. On the bottom-left of the front page is a smaller article, without pictures, that continues on page 21. Here's the headline:
Security Agency Holds A Quiet, Crucial Power Over Communications
The article begins "For the last quarter century, one of the Government's most secret agencies has played an important, largely undisclosed role in shaping the nation's privately owned ..." (that's your free preview). If I zoom in on the paper I think I can make out the rest of that sentence as "... communications network of [broadcast towers?], underground cables, satellites, and computers." The list of subjects on that article lists astronautics, communications, internal security, internal communications, National Security Agency, telephones, and United States. 37 years ago today the NY Times was reporting on the NSA holding power over the communications infrastructure of the US.
Page 4 has an ad for the latest TI calculators, available at Bloomingdale's.
This story is on page 7:
SOLAR ENERGY HELD STILL DECADES AWAY Panel Does Not Expect Major Shift Until Technology Is Ready for Conversion in Electricity A panel of leading specialists, convened a year ago at the request of the White House to assess prospects for generating electric power from sunlight, has concluded that the ultimate prospects are "bright" but that for at least a decade the technology will not be sufficiently advanced to initiate a major conversion effort.
Old newspapers are very interesting. I thought I knew all about World War II. Then I started reading the newspaper, starting in October 1938. I am reading the Canberra Times, since it is freely available on the internet. And Australia has such a great website [nla.gov.au]to read it on! It's a bit sad that the Library of Congress doesn't seem to have this kind of system for American newspapers. I guess that's what living in a society that believes in perpetual copyright gets you.
TIFF? (Score:2)
Isn't there another format that would be better than TIFF? How about TGA? :p
Re: (Score:2, Informative)
Isn't there another format that would be better than TIFF? How about TGA? :p
TIFF can use Group IV compression on monochrome images, whereas TGA is limited to RLE.
Re: (Score:2)
TIFF is ideal for black and white scans. There are multiple compression options and it contains DPI/dimension information.
What's your argument for TGA?
Re: (Score:2)
Easier-to-decode, fixed-length header? :p
Re: (Score:2)
There are a lot more considerations. For one, you can start with an existing library for TIFF. Overall file size is smaller as you can use JPEG compression for photos and LZW, RLE or CCITT for B&W text (not that they had mixed compression - nobody seems to use that).
Also, TIFF is already the standard storage format for document imaging. This is likely the format their archives were already in.
Re: (Score:2)
The one thing we miss for the web is a sort of JPEG-compressed image data but with a PNG-lossless alpha channel.
Re: (Score:2)
Apparently, you can use a bit of SVG magic to use a PNG for an alpha mask on a JPEG: http://peterhrynkow.com/how-to... [peterhrynkow.com]
Re: (Score:2)
4-bit PNG would be a better choice for black and white pages as it will retain detail on text and can handle the greyscale photos okay.
But it would need to be something else for spreads with color.
nobody cares, google did it first (Score:5, Funny)
Re: (Score:2)
Yeah, no shit ... by browser just blocks NYT entirely ... no, I'm not turning on cookies and javascript so I can see your paywall.
I've just cut out the crap and blocked it entirely. NYT has ceased to exist as far as I'm concerned.
Colossus must be fed (Score:2)
Although, sadly, they apparently never came to an agreement with The New Zork Times. [archive.org]
Goolge news archive (Score:1)
I browse the Google newpaper articles a lot in Google.
It changes the perspective on why we are the way we are now and how we have changed.
Newspaper used to put all admisions and discharges from the local hospital and hotels/motels in the area. John Reese from Clevelend checked into the corner motel, he is in town discussing a business deal with Jim Smith.
Job postings from the 40-late 50's that included a female and a male section.
And the biggest one was the false advertising and ads made to look like artic
Unlike those pesky old Microfiche and Paper copies (Score:2)
these newer electronic versions of the archives are much simpler for the Ministry of Truth to manage. A minitrue employe, Mr. Winston Smith has stated that though his job maintaining the integrity of the archives has changed little it was much more satisfying to be able to verify his corrections are reaching the masses quicker and more thoroughly than in the past when it could take months for corrections to propagate, and there were less lose ends when no one was really sure if a copy went down a memory ho
ridiculous scan density (Score:2)
who needs an archival scan of NEWSPRINT TEXT at 600+DPI? Most paper photographs (remember those?) only have about 300DPI worth of information on them *if you are lucky*, you'd be wasting your time scanning any higher for those. Making legal digital duplicates of typeset documents only requires 150DPI (which is the same as standard Fax which also happens to be a legal service method).
(source: several years experience dealing with document archiving and photo digitising on a commercial as well as a personal b
Re: (Score:2)
who needs an archival scan of NEWSPRINT TEXT at 600+DPI? Most paper photographs (remember those?) only have about 300DPI worth of information on them *if you are lucky*, you'd be wasting your time scanning any higher for those. Making legal digital duplicates of typeset documents only requires 150DPI (which is the same as standard Fax which also happens to be a legal service method).
(source: several years experience dealing with document archiving and photo digitising on a commercial as well as a personal basis)
Legal requirements on DPI probably are intended to make something legible. Making something pleasantly readable is a higher bar. Newspaper printing is was an analog process, especially for old newspapers. 300DPI is readable and probably fine, for today. 20 years from now it is going to seem like be the equivalent of a wax cylinder recording.
PNG? (Score:2)
Re: (Score:2)
Meh. Still want a Phillips Coffee Machine. (Score:2)
Coffee? Coffee. Coffee! COFFEE!!