Forgot your password?
typodupeerror
Programming

Clojure and Heroku Predict Flight Delays 109

Posted by kdawson
from the fully-buzzword-compliant dept.
murphee writes "Flight delayed again? Should have asked FlightCaster, a new site using statistical analysis to predict the delay of your flight in real-time. What's even better, the services is fully buzzword compliant: it's built with Clojure, distributed with Hadoop, served with Rails, and hosted on Heroku. This interview with one of the FlightCaster developers gives the gory details on architecture, Clojure tips, and your boss a reason to let you have all the multimethods and macros you can eat. Seems like now that O'Reilly's publishing a LISP book, the Age of Parenthesus has come..."
This discussion has been archived. No new comments can be posted.

Clojure and Heroku Predict Flight Delays

Comments Filter:
  • cool story (Score:1, Funny)

    by el_tedward (1612093)
    cool story, bro
  • Unnecessary (Score:3, Insightful)

    by JohnPetrucci00 (1621221) on Friday August 21, 2009 @01:13PM (#29148227)
    This is unnecessary because they tell you at the airport if your flight is delayed.
    • Or on that new-fangled interweb.
    • So what? The tech is still interesting.

    • Re:Unnecessary (Score:5, Insightful)

      by Wuhao (471511) on Friday August 21, 2009 @01:32PM (#29148479)

      They tell you at the airport if your flight is delayed, when the airline actually posts that your flight is delayed. For various reasons, this does not necessarily happen promptly. For example, airlines may hold off announcing a delay until very close to departure, because they haven't ruled out using a different plane than what was scheduled, or because they think it'll be a close enough delay that it's worth keeping everyone at the gate ready to board, or because the information just didn't get posted. Anyone who's ever waited at the gate 10 minutes past departure next to a sign that says "ON TIME" with no plane in sight knows what I'm talking about here.

      Supposedly, this software tries to analyze airport traffic across airlines to try and figure out which flights are going to get delayed by ATC. So, its aim is to predict certain delays before they happen, much less before they make it onto the airport departure and arrivals screens, or the airline websites.

      How well it works, I couldn't tell you.

      • Re:Unnecessary (Score:4, Informative)

        by Fnkmaster (89084) * on Friday August 21, 2009 @01:49PM (#29148687)

        Last 2 times I have flown afternoon flights on Continental from Newark to Charleston, SC I have experienced delays on both legs of the trip there and back. The flights back from Charleston to Newark have been delayed by at least 2 hours both times. I fly this route regularly and it's very frequently delayed (>6 of the 12 times I've flown it in the last year). In fact, last time (2 weeks ago), my 6pm flight took off at 8pm, and the 4pm flight still hadn't taken off yet.

        I checked this route in their system and it says it is 85% on-time, 11% delayed less than an hour, and 4% delayed an hour or more.

        I am highly skeptical of this service, based on that initial result. I'll try it out the next few times I fly though.

        • Last 2 times I have flown afternoon flights on Continental from Newark to Charleston, SC...

          Now there's your problem. As for it being "on-time", does it mean on-time arrival? Philly/New York flight times are padded by 60-90 minutes. So if you plane takes off less than an hour late, it will probably land "on-time."

        • by PylonHead (61401)

          The service predicts whether a specific flight that takes off in the next six hours will be on time.

          So the specific flight you checked had an 85% of being on time. It wasn't telling you that the route was on time 85% of the time.

          It uses real time data about the incoming aircraft for the route, the general airport situation, and regional weather to make its prediction.

          In order to do a real test you'll have to check the system on days you're actually flying.

          • by Fnkmaster (89084) *

            I actually checked tomorrow's flight to see what they thought of a general flight on this route, without benefit of today's real-time data. That's what led me to conclude something is off.

            • by PylonHead (61401)

              They may still have weather data, and potentially seasonal data, or day of the week information.

      • Re: (Score:2, Informative)

        by Anonymous Coward

        Most delays are NOT ATC based delays. The airlines simply tell you they are so you dont complain as much.

        Most delayrs are the airlines fault or weather reasons. For example, ATC has no controll over a crew not prepping a plane in time for takeoff (maybe due to mechanical issues etc). ATC deals fairly with all planes in the system.

        Your delayed plane probably lost its takeoff slot and had to wait to be reinserted into the system at the next available time. NOT ATC's fault at all your plane couldnt take

        • by SL Baur (19540)

          Most delays are NOT ATC based delays. The airlines simply tell you they are so you dont complain as much.

          I'm not so sure about that. The only time I was told a plane was being held as an ATC delay, we were sitting on the tarmac with everything sealed. We *were* flying into a major tropical storm in the US southeast.

          There have been two occasions I've been scheduled on canceled flights. The first time I wasn't informed until I reached the check in area (SFO). The second time I wasn't informed until about the time of scheduled departure. Both of those times were due to the plane failing maintenance checks (f

      • by umghhh (965931)
        assume it works for us i.e. customers then soon airlines will have a look at it too to predict and prevent? What happens when they try to alter the future that has already been predicted - crash of worlds or something. Fascinating anyway. Especially reading TFA is like learning a new foreign language (except LISP that is)
      • by pjt33 (739471)

        For various reasons, this does not necessarily happen promptly.

        Quite right. I once went to the airport for a flight that was still listed as "On time" even though the airport that the plane was coming from had been closed by snow and the plane was stuck there. Of course, I only found that part out later.

    • by Em Emalb (452530) <ememalb@NospaM.gmail.com> on Friday August 21, 2009 @01:32PM (#29148493) Homepage Journal

      This is unnecessary because they tell you at the airport if your flight is delayed.

      Back in my day you had to WALK over to a big sign and try to track down your flight information, or, even worse, you had to talk to the person standing behind the counter. And we all know that that ain't diet Dr Pepper they're drinking in that cup on their counter, no sir, that's 100% pureed baby souls. One time, I was taking a flight to Santa Barbara and I couldn't remember the airport code...well, that's not true. I could remember it, but I'd forgotten my tri-focals, so I couldn't read the blasted sign, and I had to go to that counter to find out about my flight. Well, after standing in line for close to 3 months, I finally get to the counter and the "lady" behind the counter shoots a 4 foot flame from her anus that burned my ticket to little cinders.

      Where was I? Oh yeah, I'm stoked for something like this. It beats havin' ta walk, sonny-Jim.

      • the "lady" behind the counter shoots a 4 foot flame from her anus that burned my ticket to little cinders.

        Wait, she only assaulted you but left your luggage intact? You must have platinum elite status, you lucky dog.

      • by SL Baur (19540)

        Back in my day you had to WALK over to a big sign and try to track down your flight information, or, even worse, you had to talk to the person standing behind the counter. And we all know that that ain't diet Dr Pepper they're drinking in that cup on their counter, no sir, that's 100% pureed baby souls. One time, I was taking a flight to Santa Barbara and I couldn't remember the airport code...well, that's not true. I could remember it, but I'd forgotten my tri-focals, so I couldn't read the blasted sign, and I had to go to that counter to find out about my flight. Well, after standing in line for close to 3 months, I finally get to the counter and the "lady" behind the counter shoots a 4 foot flame from her anus that burned my ticket to little cinders.

        That sounds like a *great* idea for a game. Add in airport security with automatic weapons and service personnel with concealed weapons and I think you have yourself a winner. If you make it alive past the smiling boarding stewardess (who has concealed knives or something like that) you win!

    • Re:Unnecessary (Score:5, Interesting)

      by ajs (35943) <ajs@ a j s . c om> on Friday August 21, 2009 @01:48PM (#29148677) Homepage Journal

      This is unnecessary because they tell you at the airport if your flight is delayed.

      Oh dear my, no.

      The airlines actually make it a strict policy to lie about delays. They don't release that information until many minutes and often hours until after they know about it. I used to work in the air traffic industry, and the data that I had access to at the time would show me delays that were scheduled by the FAA up to a couple of days in advance, but the airlines kept strict control over that information because leaking it would mean that competitors could offer to pick up passengers from delayed flights.

  • by loteck (533317) on Friday August 21, 2009 @01:16PM (#29148275) Homepage
    Everyone is busy Googling 75% of the terms used in the summary trying to figure out what it even says. Someone in the know care to interpret?
    • Re: (Score:3, Interesting)

      Exactly. This project is simply a case study on how to build a web application on top of the most obscure infrastructure imaginable. That must be why there aren't many posts yet. All the bleeding-edge early-adopter \. flame-stokers were recruited for FlightCaster and are under NDAs.
      • by Firehed (942385)

        You say that in jest, but I know the flightcaster team and it's four times bigger than it should be for a company as young as they are. I wouldn't be surprised if they picked up a couple of slashdotters in the past few days.

      • by tcopeland (32225)

        > on top of the most obscure infrastructure imaginable.

        I don't know... look at the Aloha on Rails [alohaonrails.com] session summaries; Blake Mizerany says:

        Heroku has over 32,000 apps running on it currently, some serving hundreds of requests per second.

        Not bad.

    • In the summary? You must be new here... (Did you buy your ID?)
      Nobody reads the summary anymore. ^^

  • Works like a charm (Score:3, Informative)

    by Fahrvergnuugen (700293) on Friday August 21, 2009 @01:18PM (#29148299) Homepage
    Southwest's site says my flight is delayed 45 minutes. Flightcaster says its right on time! (for those interested, its southwest flight 2978)
  • The names (Score:1, Flamebait)

    by oldhack (1037484)

    Programming, I've been doing this for living for two decades, but I have no clue what those things are. Well, except for Rails I did read a bit about.

    Probably all for the best, I'd guess.

    • <quote><p>
      Programming, I've been doing this for living for two decades, but I have no clue what those things are. Well, except for Rails I did read a bit about.
      </p><p>
      Probably all for the best, I'd guess.</p></quote>

      Seriously?!?! Two decades and you've never looked at LISP?!?! I've only got a mere 1.2 decades and I've been in the mode of learning like
      a son-of-a-bitch on all this stuff after slacking for most of the easy-peasy dot-com boom. Showing up and only learning
      • by oldhack (1037484)
        *P Yeah, thanks for the career tip. Kind move out of this green patch.
      • by Abcd1234 (188840)

        Showing up and only learning what you need
        for your job is not going to help you see a third decade. But hey, your career man...

        Forget your career. Learning new, unique programmings languages is just fun and interesting, IMHO, and keeps me interested in the process.

        That said, I can't stand Lisp. Honestly, if Perl can be written off as a "write-only" language, and can't see why Lisp doesn't go the same way. Now Haskell, on the other hand...

        • by bADlOGIN (133391)
          Haskell is on on my list of things to check out. There's a nifty "easy to swallow intro" at: http://learnyouahaskell.com
          and I'm hoping to get over a previous slightly abusive exposure I got to it in college. Being young and foolish,
          I didn't like the instructor and transfered my dislike onto the language:)
          • by j1m+5n0w (749199)
            Learn you a haskell [learnyouahaskell.com] is certainly a good place to start. Real world haskell [realworldhaskell.org] is also available on the web, if you want more detail than LYAH provides, particularly as it pertains to building applications that deal with IO, networking, interaction, etc...
  • Heroku link broken (Score:5, Informative)

    by Mr. Sketch (111112) <mister.sketchNO@SPAMgmail.com> on Friday August 21, 2009 @01:25PM (#29148397)

    The link for Heroku is broken, it points to http://slashdot.org/heroku.com [slashdot.org].
    Here is a working link:
    http://heroku.com/ [heroku.com]

  • by Anonymous Coward

    Yet another website that only works for people in its own country, yet doesn't bother mentioning what country that might be. Sadly, the rest of the world knows this arrogance to be an obvious American trait.

  • by GameMaster (148118) on Friday August 21, 2009 @01:26PM (#29148411)

    Can someone fill me in on what this would be good for? Since it's statistics based and not, officially, affiliated with the airlines, you can't really expect to use it to arrive at the airport later than officially scheduled (I suppose you could, but there's a good chance that your, particular, flight will buck the statistical averages and will take off sooner than predicted without you). If they worked, directly, with the airlines and got the airlines to guarantee that they wont take off earlier than the statistical model predicts then I could see it being useful but that's never gonna happen because they only care about finishing as many flights per hour as possible.

    • I was just hoping somebody could fill me in on what the summary actually means. there was so much jargon in there I couldn't decipher any of it.

      Summary fail.

      • Re: (Score:2, Informative)

        by FlyingBishop (1293238)

        What exactly do you need clarified? The first sentence says exactly what it does, it predicts flight delay times. The rest talks about the tools used to do it. And though the tools are perhaps overhyped, referring to the tools isn't really a buzzword, just a statement of fact. Now, if the summary had said something along the lines of

        "Flight delayed again? Shoulda asked FlightCaster, a new site leveraging the power of the cloud to provide real-time statistical analysis of flight delays. It uses a combination

    • I wouldn't use it for deciding when to go to the airport, but as for deciding how much time you have in the airport, that is valuable.

      Peace of mind - I'd rather know that I'll probably be alright despite the extra-long security line.

      I'd like to know - hey, the plane's not here, and it looks like I'll have plenty of time to eat.

      This is, could be quite convenient - especially if coupled with a notification on flight arrival or boarding.

    • by mcmonkey (96054) on Friday August 21, 2009 @02:01PM (#29148835) Homepage

      Absolutely nothing, huh! Say it again...

      Well, no. You would not use this to decide when to go to the airport for your flight. In that case, you're not interested in likelihoods, you're interested in the specific condition of your flight.

      When you would use this is in the flight booking stage. For example, your itinerary involves connecting flights. You don't to be stuck with hours to kill in the airport between flights, but you also don't want to miss the connection.

      If the schedule says I'll have 30 minutes between flights, what are the odds I miss my connection?

      • When you would use this is in the flight booking stage. For example, your itinerary involves connecting flights. You don't to be stuck with hours to kill in the airport between flights, but you also don't want to miss the connection.

        If the schedule says I'll have 30 minutes between flights, what are the odds I miss my connection?

        So this is basically made for contestants on The Amazing Race.

    • I have two flights with the same travel time and cost to pick from, and I'll pick the one least likely to be delayed.
  • ... start naming my software with really cool names, then get somebody to intereview me, then I too could have my very own /. story!
  • So is the Summary!

    GG K-dawg!

  • Hasn't FlightStats.com been doing something similar for years, just without the trendy technologies?

  • Clearly this is an application built and designed by and for the terrorists.

    Let the fear mongering begin!

  • by reginaldo (1412879) on Friday August 21, 2009 @01:59PM (#29148803)
    From the article: 'There are only two of us that have been working on the research side of things...'

    So there are 2 guys that built this machine learning process, distributed using cascade and hadoop, and they built and distributed an app to show the results using rails and heroku?

    These guys probably eat my code as a breakfast snack. Seriously, how do I become that badass?
    • Re: (Score:1, Funny)

      by Anonymous Coward

      We badass scientific and architectural types think in abstractions (that's pretty Lispy), kernels (that's numerical), and codes (that's crazy numerical). We don't write programs, we design languages within which the desired program is a trivial expression, or at least beta-reduces to a trivial expression.

      Then you need an army of mortals to figure out how to package it in a way that it can be used by other mortals, but that's not really our problem, is it?

    • by Bluesman (104513)

      Learn Lisp. You could write all that with a few macros and have time left over before lunch to read all of Paul Graham's essays and think about how awesome you are.

    • Do you have pointy hair by any chance? Regression analysis may sound super cool, but it is nothing new, and it is often very poorly implemented. So before you say those guys are badasses, which they may be (that I don't know), at least make sure that their application works as intended. For all you know, it may not even work properly.

      Farecast.com [farecast.com] for example is one such super cool showcase concept that simply doesn't work (and that probably never will -- except may be to advertise its coolness factor). Far

  • I know this might sound less cool and trendy, but just ask either the agent working the flight or one of your fellow passengers if they know the chances/likelihood of the flight getting delayed. I have worked part-time at the Atlanta airport for 4 years now, and I can easily tell you which flights at which time are more than likely to get delayed, on what days of the week, with certain weather patterns. Along with this, I also know the patterns of our particular tower, what they are likely to do for any g
    • Re: (Score:1, Funny)

      by Anonymous Coward

      Most of us who have attempted to enjoy any kind of social interaction with anyone condemned to work in an airport have not relished in the experience.

  • your flight will be delayed. expect it. seasoned travellers know delays are common and should be expected. expecting to get there at the scheduled time is unrealistic. always leave large buffers. always expect delays and be pleasantly surprised if there are none.

  • In their faq, they mention that the don't try to predict the departure delay, they focus on the arrival delay - specifically to avoid misunderstandings that might cause missed flights.

    One thing this might tell you is how long you'll be on the tarmac until you actually leave - giving you a better idea of when you'll arrive much sooner.

    While that doesn't directly affect the passenger, it can be extremely useful for whoever is picking you up.

  • by Chris Mattern (191822) on Friday August 21, 2009 @02:30PM (#29149243)

    I caught their act in Vegas! They were absolutely amazing!

  • It does not really help you. Because when it says it will probably be delayed, then what does this change for you? You still have to sit there and wait. Because when it's not delayed, and you're not there, then what?

    • by Wuhao (471511)

      I could see this as being nice once I get to the airport. I try to time my arrival close to boarding, and once I'm there, the gate area is usually pretty full. I can go down a few gates and make myself comfortable, maybe even get a drink, but if my flight really does start boarding in 5-10 minutes, why bother?

      If I knew that there was an 80% chance that I'd be waiting there for an extra half hour or so, I'd be more inclined to go grab a bite to eat, or at least find a power plug and do some e-mail. There's a

  • by todd10k (889348)
    that article make absolutly no sense to anyone else?
  • O'Reilly might not want to bother...

    Check it out: http://gigamonkeys.com/book/
    • by certron (57841)

      I second this recommendation of Practical Common Lisp. I even bought it in hardback! The table of contents for the O'Reilly book seems a little... lacking. Effective functional programming like this really requires a different mindset from , and the table of contents makes me start to wonder if there will be enough background provided or if it will just go into libraries and frameworks as fast as possible.

      Don't forget that after you are done with Practical Common Lisp, you can move on to ANSI Common Lisp by

      • by bADlOGIN (133391)
        I loved the "holy crap" moment I had when reading the section in PCL about "before" "after" and "around" which I've
        used a career Java developer without seeing so much as a hit of credit from the Aspect Oriented community. Perhaps
        the props are out there, but I felt some more overt credit was due for the fact the Lisp has had this for some time:)
        • by certron (57841)

          (Not that anyone is going to read this, so late after the article posting, on a deep comment thread, no less, but...)

          The great thing is that because Lisp is so regular in its syntax and has macros available, it can be extended to include an object system, aspect oriented programming, and even language-support for the Actor model like in Erlang (for example, Termite).

          I wonder if anyone has called Lisp "the oldest language of the future" yet... Then again, smarter minds than mine have already expressed their

    • [WIth Practical Common Lisp free & from Apress] O'Reilly might not want to bother...

      Right, because if there is one book available free that addresses a language, there's no reason to bother having any more books covering that language.

    • Except that Clojure is NOT a dialect of Common Lisp. I wouldn't give someone an 'intro to Java' book if they wanted to learn C#.

      Why not read a book specifically tailored to Clojure? [pragprog.com]
  • I want it to predict whether the doctor will see the patient on time, early, or 10/20/30/40+ minutes late.

  • I like to use FlightAware [flightaware.com] for realtime/historical tracking maps/stats of flights. I wish there were a way to mash up these delay data services with FlightAware.

    If a Google Maps layer stream were as popular an API as RSS has become, we might see all kinds of data sources integrated into really helpful visualizations.

  • How are they going to keep this cluster of servers running (they use EC2, so they don't have THAT much overhead), crunching all that data? I don't see why anyone would pay anything for this, after all its just an educated guess...

  • by Animats (122034) on Friday August 21, 2009 @04:16PM (#29150395) Homepage

    They're scraping free data from the FAA web site [faa.gov] and FlightStats [flightstats.com], then pumping it out into an iPhone app feed.

    But they're not using a really good data source. The high quality system is PASSUR RightETA [passur.com]. This system uses hundreds of radar receivers near airports to pick up the transponder signals from aircraft. It doesn't transmit. Any radar in the area that triggers an aircraft transponder causes the transponder to emit, and the PASSUR receivers pick that up. Using multiple receivers and time of flight calculations, the aircraft can be located very precisely. In fact, this is more accurate than single-point radar. You can buy a feed of this data, but it's not free.

    • by Pulzar (81031)

      If you read some of their FAQ, you'd see that they are not scraping the data from the FAA web site, but are instead modeling the air traffic control algorithms for allocating plane take-off and landing slots at the airports. They claim that the vast majority of the delays can be predicted that way (whether that's right or not is another questions).

  • Blatant self-promotion:

    I produced a commercial screencast with Phil Hagelberg, edited by Clojure creator Rich Hickey. Many people have found it useful for learning Clojure, or even just learning about what it is and what it can do:

    http://peepcode.com/products/functional-programming-with-clojure [peepcode.com]

  • Clojure, Rails, Hadoop, jeez - what's next, white belts and fixed-gear bicycles? This is probably what these guys look like. [latfh.com]
  • What, these guys are giving customers useful data, without the airlines' consent? Take them down! It's obviously a copyright violation [slashdot.org] or something, because it's not making the airlines more money.
  • by bytesex (112972)

    I've been out of the buzzword loop for a while, but these 'multimethods' - isn't that anything and everything that programmers should avoid like the plague ? I mean, checking the type of some vage supertype inside a function body - even perl warns against that !

  • Seems like now that O'Reilly's publishing a LISP book, the Age of Parenthesus has come

    I, for one, welcome our new parenthetical overlords.

If bankers can count, how come they have eight windows and only four tellers?

Working...