Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
Check out the new SourceForge HTML5 internet speed test! No Flash necessary and runs on all devices. ×
The Media

How To Build a TimesMachine (nytimes.com) 41

necro81 writes: The NY Times has an archive, the TimesMachine, that allows users to find any article from any issue from 1851 to the present day. Most of it is shown in the original typeset context of where an article appeared on a given page — like sifting through a microfiche archive. But when original newspaper scans are 100-MB TIFF files, how can this information be conveyed in an efficient manner to the end user? These are other computational challenges are described in this blog post on how the TimesMachine was realized.
This discussion has been archived. No new comments can be posted.

How To Build a TimesMachine

Comments Filter:
  • by The-Ixian ( 168184 ) on Monday February 01, 2016 @02:32PM (#51416195)

    Seems like this is the obvious choice.

    Maybe just the headlines or the first paragraph and then link to a compressed version of the image file or PDF (not the TIFF itself for Jebussake).

    • It's obviously more than just OCR, because when you mouse over the page it will hightlight each article you're looking at, however oddly that article was fit into the page (including highlighting associated pictures, etc). That's actually a pretty cool site, in fact that archive might even be a better reason to get a subscription than the actual current edition.

      The free preview doesn't give you much, but it does show the issue from February 1, 1979. In the top right is a story about Ayatollah Khomeini arr

      • by dj245 ( 732906 )

        It's obviously more than just OCR, because when you mouse over the page it will hightlight each article you're looking at, however oddly that article was fit into the page (including highlighting associated pictures, etc). That's actually a pretty cool site, in fact that archive might even be a better reason to get a subscription than the actual current edition.

        The free preview doesn't give you much, but it does show the issue from February 1, 1979. In the top right is a story about Ayatollah Khomeini arriving back in Iran from exile, which is interesting to see the contemporary story. On the bottom-left of the front page is a smaller article, without pictures, that continues on page 21. Here's the headline:

        Security Agency Holds A Quiet, Crucial Power Over Communications

        The article begins "For the last quarter century, one of the Government's most secret agencies has played an important, largely undisclosed role in shaping the nation's privately owned ..." (that's your free preview). If I zoom in on the paper I think I can make out the rest of that sentence as "... communications network of [broadcast towers?], underground cables, satellites, and computers." The list of subjects on that article lists astronautics, communications, internal security, internal communications, National Security Agency, telephones, and United States. 37 years ago today the NY Times was reporting on the NSA holding power over the communications infrastructure of the US.

        Page 4 has an ad for the latest TI calculators, available at Bloomingdale's.

        This story is on page 7:

        SOLAR ENERGY HELD STILL DECADES AWAY Panel Does Not Expect Major Shift Until Technology Is Ready for Conversion in Electricity A panel of leading specialists, convened a year ago at the request of the White House to assess prospects for generating electric power from sunlight, has concluded that the ultimate prospects are "bright" but that for at least a decade the technology will not be sufficiently advanced to initiate a major conversion effort.

        Old newspapers are very interesting. I thought I knew all about World War II. Then I started reading the newspaper, starting in October 1938. I am reading the Canberra Times, since it is freely available on the internet. And Australia has such a great website [nla.gov.au]to read it on! It's a bit sad that the Library of Congress doesn't seem to have this kind of system for American newspapers. I guess that's what living in a society that believes in perpetual copyright gets you.

  • Isn't there another format that would be better than TIFF? How about TGA? :p

    • Re: (Score:2, Informative)

      by Anonymous Coward

      Isn't there another format that would be better than TIFF? How about TGA? :p

      TIFF can use Group IV compression on monochrome images, whereas TGA is limited to RLE.

    • TIFF is ideal for black and white scans. There are multiple compression options and it contains DPI/dimension information.

      What's your argument for TGA?

      • Easier-to-decode, fixed-length header? :p

        • There are a lot more considerations. For one, you can start with an existing library for TIFF. Overall file size is smaller as you can use JPEG compression for photos and LZW, RLE or CCITT for B&W text (not that they had mixed compression - nobody seems to use that).

          Also, TIFF is already the standard storage format for document imaging. This is likely the format their archives were already in.

    • by SeaFox ( 739806 )

      4-bit PNG would be a better choice for black and white pages as it will retain detail on text and can handle the greyscale photos okay.
      But it would need to be something else for spreads with color.

  • by Thud457 ( 234763 ) on Monday February 01, 2016 @02:50PM (#51416309) Homepage Journal
    sticking it behind a paywall will cut down on bandwidth usage dramatically.
  • by Anonymous Coward

    I browse the Google newpaper articles a lot in Google.
    It changes the perspective on why we are the way we are now and how we have changed.

    Newspaper used to put all admisions and discharges from the local hospital and hotels/motels in the area. John Reese from Clevelend checked into the corner motel, he is in town discussing a business deal with Jim Smith.

    Job postings from the 40-late 50's that included a female and a male section.

    And the biggest one was the false advertising and ads made to look like artic

  • these newer electronic versions of the archives are much simpler for the Ministry of Truth to manage. A minitrue employe, Mr. Winston Smith has stated that though his job maintaining the integrity of the archives has changed little it was much more satisfying to be able to verify his corrections are reaching the masses quicker and more thoroughly than in the past when it could take months for corrections to propagate, and there were less lose ends when no one was really sure if a copy went down a memory ho

  • who needs an archival scan of NEWSPRINT TEXT at 600+DPI? Most paper photographs (remember those?) only have about 300DPI worth of information on them *if you are lucky*, you'd be wasting your time scanning any higher for those. Making legal digital duplicates of typeset documents only requires 150DPI (which is the same as standard Fax which also happens to be a legal service method).

    (source: several years experience dealing with document archiving and photo digitising on a commercial as well as a personal b

    • by dj245 ( 732906 )

      who needs an archival scan of NEWSPRINT TEXT at 600+DPI? Most paper photographs (remember those?) only have about 300DPI worth of information on them *if you are lucky*, you'd be wasting your time scanning any higher for those. Making legal digital duplicates of typeset documents only requires 150DPI (which is the same as standard Fax which also happens to be a legal service method).

      (source: several years experience dealing with document archiving and photo digitising on a commercial as well as a personal basis)

      Legal requirements on DPI probably are intended to make something legible. Making something pleasantly readable is a higher bar. Newspaper printing is was an analog process, especially for old newspapers. 300DPI is readable and probably fine, for today. 20 years from now it is going to seem like be the equivalent of a wax cylinder recording.

  • Really, this isn't that complicated a problem. It's a newspaper, run the things through a compession algorithm such as png, drop the colorspace down to something reasonable for a newspaper. Hell, black and white is actually just fine but you can keep 256 colors if it makes you feel warm and fuzzy and still drop that 100MB to a 10-100k tops. Then OCR it all for search. Tada.
  • Coffee? Coffee. Coffee! COFFEE!!

Advertising is the rattling of a stick inside a swill bucket. -- George Orwell

Working...