Clojure and Heroku Predict Flight Delays 109
murphee writes "Flight delayed again? Should have asked FlightCaster, a new site using statistical analysis to predict the delay of your flight in real-time. What's even better, the services is fully buzzword compliant: it's built with Clojure, distributed with Hadoop, served with Rails, and hosted on Heroku. This interview with one of the FlightCaster developers gives the gory details on architecture, Clojure tips, and your boss a reason to let you have all the multimethods and macros you can eat. Seems like now that O'Reilly's publishing a LISP book, the Age of Parenthesus has come..."
cool story (Score:1, Funny)
Unnecessary (Score:3, Insightful)
Re: (Score:2)
Re: (Score:2)
So what? The tech is still interesting.
Whoosh! (Score:1)
Re: (Score:2)
If it's whooshed me, I'm not alone - currently modded +3 insightful.
Re:Unnecessary (Score:5, Insightful)
They tell you at the airport if your flight is delayed, when the airline actually posts that your flight is delayed. For various reasons, this does not necessarily happen promptly. For example, airlines may hold off announcing a delay until very close to departure, because they haven't ruled out using a different plane than what was scheduled, or because they think it'll be a close enough delay that it's worth keeping everyone at the gate ready to board, or because the information just didn't get posted. Anyone who's ever waited at the gate 10 minutes past departure next to a sign that says "ON TIME" with no plane in sight knows what I'm talking about here.
Supposedly, this software tries to analyze airport traffic across airlines to try and figure out which flights are going to get delayed by ATC. So, its aim is to predict certain delays before they happen, much less before they make it onto the airport departure and arrivals screens, or the airline websites.
How well it works, I couldn't tell you.
Re:Unnecessary (Score:4, Informative)
Last 2 times I have flown afternoon flights on Continental from Newark to Charleston, SC I have experienced delays on both legs of the trip there and back. The flights back from Charleston to Newark have been delayed by at least 2 hours both times. I fly this route regularly and it's very frequently delayed (>6 of the 12 times I've flown it in the last year). In fact, last time (2 weeks ago), my 6pm flight took off at 8pm, and the 4pm flight still hadn't taken off yet.
I checked this route in their system and it says it is 85% on-time, 11% delayed less than an hour, and 4% delayed an hour or more.
I am highly skeptical of this service, based on that initial result. I'll try it out the next few times I fly though.
Re: (Score:2)
Now there's your problem. As for it being "on-time", does it mean on-time arrival? Philly/New York flight times are padded by 60-90 minutes. So if you plane takes off less than an hour late, it will probably land "on-time."
Re: (Score:2)
The service predicts whether a specific flight that takes off in the next six hours will be on time.
So the specific flight you checked had an 85% of being on time. It wasn't telling you that the route was on time 85% of the time.
It uses real time data about the incoming aircraft for the route, the general airport situation, and regional weather to make its prediction.
In order to do a real test you'll have to check the system on days you're actually flying.
Re: (Score:2)
I actually checked tomorrow's flight to see what they thought of a general flight on this route, without benefit of today's real-time data. That's what led me to conclude something is off.
Re: (Score:2)
They may still have weather data, and potentially seasonal data, or day of the week information.
Re: (Score:2, Informative)
Most delays are NOT ATC based delays. The airlines simply tell you they are so you dont complain as much.
Most delayrs are the airlines fault or weather reasons. For example, ATC has no controll over a crew not prepping a plane in time for takeoff (maybe due to mechanical issues etc). ATC deals fairly with all planes in the system.
Your delayed plane probably lost its takeoff slot and had to wait to be reinserted into the system at the next available time. NOT ATC's fault at all your plane couldnt take
Re: (Score:2)
Most delays are NOT ATC based delays. The airlines simply tell you they are so you dont complain as much.
I'm not so sure about that. The only time I was told a plane was being held as an ATC delay, we were sitting on the tarmac with everything sealed. We *were* flying into a major tropical storm in the US southeast.
There have been two occasions I've been scheduled on canceled flights. The first time I wasn't informed until I reached the check in area (SFO). The second time I wasn't informed until about the time of scheduled departure. Both of those times were due to the plane failing maintenance checks (f
Re: (Score:2)
Re: (Score:2)
For various reasons, this does not necessarily happen promptly.
Quite right. I once went to the airport for a flight that was still listed as "On time" even though the airport that the plane was coming from had been closed by snow and the plane was stuck there. Of course, I only found that part out later.
Re:Unnecessary (Score:5, Funny)
This is unnecessary because they tell you at the airport if your flight is delayed.
Back in my day you had to WALK over to a big sign and try to track down your flight information, or, even worse, you had to talk to the person standing behind the counter. And we all know that that ain't diet Dr Pepper they're drinking in that cup on their counter, no sir, that's 100% pureed baby souls. One time, I was taking a flight to Santa Barbara and I couldn't remember the airport code...well, that's not true. I could remember it, but I'd forgotten my tri-focals, so I couldn't read the blasted sign, and I had to go to that counter to find out about my flight. Well, after standing in line for close to 3 months, I finally get to the counter and the "lady" behind the counter shoots a 4 foot flame from her anus that burned my ticket to little cinders.
Where was I? Oh yeah, I'm stoked for something like this. It beats havin' ta walk, sonny-Jim.
Re: (Score:2)
the "lady" behind the counter shoots a 4 foot flame from her anus that burned my ticket to little cinders.
Wait, she only assaulted you but left your luggage intact? You must have platinum elite status, you lucky dog.
Re: (Score:2)
Back in my day you had to WALK over to a big sign and try to track down your flight information, or, even worse, you had to talk to the person standing behind the counter. And we all know that that ain't diet Dr Pepper they're drinking in that cup on their counter, no sir, that's 100% pureed baby souls. One time, I was taking a flight to Santa Barbara and I couldn't remember the airport code...well, that's not true. I could remember it, but I'd forgotten my tri-focals, so I couldn't read the blasted sign, and I had to go to that counter to find out about my flight. Well, after standing in line for close to 3 months, I finally get to the counter and the "lady" behind the counter shoots a 4 foot flame from her anus that burned my ticket to little cinders.
That sounds like a *great* idea for a game. Add in airport security with automatic weapons and service personnel with concealed weapons and I think you have yourself a winner. If you make it alive past the smiling boarding stewardess (who has concealed knives or something like that) you win!
Re:Unnecessary (Score:5, Interesting)
This is unnecessary because they tell you at the airport if your flight is delayed.
Oh dear my, no.
The airlines actually make it a strict policy to lie about delays. They don't release that information until many minutes and often hours until after they know about it. I used to work in the air traffic industry, and the data that I had access to at the time would show me delays that were scheduled by the FAA up to a couple of days in advance, but the airlines kept strict control over that information because leaking it would mean that competitors could offer to pick up passengers from delayed flights.
Re: (Score:2)
What kind of delays would be known days in advance?
Most of them.
Delays are typically based on traffic patterns, which are established as flight plans are filed weeks in advance. As that information takes shape, the FAA restricts departure times to control the airspace pro-actively.
Then, as weather predictions become available and reasonably accurate (starting 48 hours or so before takeoff), that information is factored in, and further action is taken.
If you find your flight delayed, chances are it is pretty much exactly on-time from the FAA's perspective. T
Lack of story comments so far (Score:5, Insightful)
Re: (Score:3, Interesting)
Re: (Score:2)
You say that in jest, but I know the flightcaster team and it's four times bigger than it should be for a company as young as they are. I wouldn't be surprised if they picked up a couple of slashdotters in the past few days.
Re: (Score:2)
> on top of the most obscure infrastructure imaginable.
I don't know... look at the Aloha on Rails [alohaonrails.com] session summaries; Blake Mizerany says:
Not bad.
Re: (Score:1)
In the summary? You must be new here... (Did you buy your ID?)
Nobody reads the summary anymore. ^^
Works like a charm (Score:3, Informative)
Re: (Score:3, Informative)
The names (Score:1, Flamebait)
Programming, I've been doing this for living for two decades, but I have no clue what those things are. Well, except for Rails I did read a bit about.
Probably all for the best, I'd guess.
Migt want to learn before your job goes to India.. (Score:2)
Programming, I've been doing this for living for two decades, but I have no clue what those things are. Well, except for Rails I did read a bit about.
</p><p>
Probably all for the best, I'd guess.</p></quote>
Seriously?!?! Two decades and you've never looked at LISP?!?! I've only got a mere 1.2 decades and I've been in the mode of learning like
a son-of-a-bitch on all this stuff after slacking for most of the easy-peasy dot-com boom. Showing up and only learning
Re: (Score:2)
Re: (Score:2)
Showing up and only learning what you need
for your job is not going to help you see a third decade. But hey, your career man...
Forget your career. Learning new, unique programmings languages is just fun and interesting, IMHO, and keeps me interested in the process.
That said, I can't stand Lisp. Honestly, if Perl can be written off as a "write-only" language, and can't see why Lisp doesn't go the same way. Now Haskell, on the other hand...
Re: (Score:2)
and I'm hoping to get over a previous slightly abusive exposure I got to it in college. Being young and foolish,
I didn't like the instructor and transfered my dislike onto the language:)
Re: (Score:2)
Heroku link broken (Score:5, Informative)
The link for Heroku is broken, it points to http://slashdot.org/heroku.com [slashdot.org].
Here is a working link:
http://heroku.com/ [heroku.com]
Re:Heroku link broken (Score:5, Funny)
in seventeen syllables
using Heroku?
Re: (Score:1)
Re: (Score:2)
Heroku haiku
Easy for all beginners
No syntax errors
Define "International" flights not supported (Score:1, Insightful)
Yet another website that only works for people in its own country, yet doesn't bother mentioning what country that might be. Sadly, the rest of the world knows this arrogance to be an obvious American trait.
Re:Define "International" flights not supported (Score:4, Insightful)
Oh, come on. I'm american, and I agree with GP. If it doesn't support "International," it should at least define the perspective of what defines "International."
Re: (Score:2)
If you're German, and you've heard of this site called "FlightCaster" and try and look up domestic German flights there. If you get a message that says that they don't support "Interneational" flights, you're going to say "huh? I queried a DOMESTIC flight!"
Sure, it's at FlightCaster.com and not FlightCaster.de - but it's not at FlightCaster.us either.
Re: (Score:2)
Although it could be real soon, because they did register the domain flightcaster.us
Re: (Score:2)
And yes - he meant international meaning involving any other country but the US, not between multiple countries.
Re: (Score:2)
Re: (Score:2)
Depends; Vancouver, Washington, USA or Vancouver, British Columbia, Canada?
Re: (Score:2)
Re: (Score:2)
Italy to South Africa would be an International flight and doesn't even touch the US.
What's it good for? (Score:5, Insightful)
Can someone fill me in on what this would be good for? Since it's statistics based and not, officially, affiliated with the airlines, you can't really expect to use it to arrive at the airport later than officially scheduled (I suppose you could, but there's a good chance that your, particular, flight will buck the statistical averages and will take off sooner than predicted without you). If they worked, directly, with the airlines and got the airlines to guarantee that they wont take off earlier than the statistical model predicts then I could see it being useful but that's never gonna happen because they only care about finishing as many flights per hour as possible.
speaking of buzzwords... (Score:1)
Summary fail.
Re: (Score:2, Informative)
What exactly do you need clarified? The first sentence says exactly what it does, it predicts flight delay times. The rest talks about the tools used to do it. And though the tools are perhaps overhyped, referring to the tools isn't really a buzzword, just a statement of fact. Now, if the summary had said something along the lines of
Re: (Score:2)
I wouldn't use it for deciding when to go to the airport, but as for deciding how much time you have in the airport, that is valuable.
Peace of mind - I'd rather know that I'll probably be alright despite the extra-long security line.
I'd like to know - hey, the plane's not here, and it looks like I'll have plenty of time to eat.
This is, could be quite convenient - especially if coupled with a notification on flight arrival or boarding.
Re:What's it good for? (Score:5, Insightful)
Absolutely nothing, huh! Say it again...
Well, no. You would not use this to decide when to go to the airport for your flight. In that case, you're not interested in likelihoods, you're interested in the specific condition of your flight.
When you would use this is in the flight booking stage. For example, your itinerary involves connecting flights. You don't to be stuck with hours to kill in the airport between flights, but you also don't want to miss the connection.
If the schedule says I'll have 30 minutes between flights, what are the odds I miss my connection?
Re: (Score:1)
When you would use this is in the flight booking stage. For example, your itinerary involves connecting flights. You don't to be stuck with hours to kill in the airport between flights, but you also don't want to miss the connection.
If the schedule says I'll have 30 minutes between flights, what are the odds I miss my connection?
So this is basically made for contestants on The Amazing Race.
Re: (Score:2)
Mabye if I ... (Score:2)
Not only is the Service Buzzword compliant (Score:2, Insightful)
So is the Summary!
GG K-dawg!
FlightStats? (Score:1)
Hasn't FlightStats.com been doing something similar for years, just without the trendy technologies?
Re: (Score:2)
you mean one of the primary data sources for flightcaster?? yes
What is the "Age of Parenthesus"? (Score:1)
Re:What is the "Age of Parenthesus"? (Score:4, Funny)
Re: (Score:2)
Terrorism (Score:1)
Clearly this is an application built and designed by and for the terrorists.
Let the fear mongering begin!
Holy schnikes (Score:4, Funny)
So there are 2 guys that built this machine learning process, distributed using cascade and hadoop, and they built and distributed an app to show the results using rails and heroku?
These guys probably eat my code as a breakfast snack. Seriously, how do I become that badass?
Re: (Score:1, Funny)
We badass scientific and architectural types think in abstractions (that's pretty Lispy), kernels (that's numerical), and codes (that's crazy numerical). We don't write programs, we design languages within which the desired program is a trivial expression, or at least beta-reduces to a trivial expression.
Then you need an army of mortals to figure out how to package it in a way that it can be used by other mortals, but that's not really our problem, is it?
Re: (Score:2)
Learn Lisp. You could write all that with a few macros and have time left over before lunch to read all of Paul Graham's essays and think about how awesome you are.
Re: (Score:2)
Do you have pointy hair by any chance? Regression analysis may sound super cool, but it is nothing new, and it is often very poorly implemented. So before you say those guys are badasses, which they may be (that I don't know), at least make sure that their application works as intended. For all you know, it may not even work properly.
Farecast.com [farecast.com] for example is one such super cool showcase concept that simply doesn't work (and that probably never will -- except may be to advertise its coolness factor). Far
ask someone! (Score:1)
Re: (Score:1, Funny)
Most of us who have attempted to enjoy any kind of social interaction with anyone condemned to work in an airport have not relished in the experience.
i can beat sw (Score:1)
your flight will be delayed. expect it. seasoned travellers know delays are common and should be expected. expecting to get there at the scheduled time is unrealistic. always leave large buffers. always expect delays and be pleasantly surprised if there are none.
Arrival Delay (Score:2)
In their faq, they mention that the don't try to predict the departure delay, they focus on the arrival delay - specifically to avoid misunderstandings that might cause missed flights.
One thing this might tell you is how long you'll be on the tarmac until you actually leave - giving you a better idea of when you'll arrive much sooner.
While that doesn't directly affect the passenger, it can be extremely useful for whoever is picking you up.
Clojure and Heroku? (Score:4, Funny)
I caught their act in Vegas! They were absolutely amazing!
There's a big problem with this: (Score:2)
It does not really help you. Because when it says it will probably be delayed, then what does this change for you? You still have to sit there and wait. Because when it's not delayed, and you're not there, then what?
Re: (Score:2)
I could see this as being nice once I get to the airport. I try to time my arrival close to boarding, and once I'm there, the gate area is usually pretty full. I can go down a few gates and make myself comfortable, maybe even get a drink, but if my flight really does start boarding in 5-10 minutes, why bother?
If I knew that there was an 80% chance that I'd be waiting there for an extra half hour or so, I'd be more inclined to go grab a bite to eat, or at least find a power plug and do some e-mail. There's a
umm (Score:1)
WIth Practical Common Lisp free & from Apress. (Score:2)
Check it out: http://gigamonkeys.com/book/
Re: (Score:2)
I second this recommendation of Practical Common Lisp. I even bought it in hardback! The table of contents for the O'Reilly book seems a little... lacking. Effective functional programming like this really requires a different mindset from , and the table of contents makes me start to wonder if there will be enough background provided or if it will just go into libraries and frameworks as fast as possible.
Don't forget that after you are done with Practical Common Lisp, you can move on to ANSI Common Lisp by
Re: (Score:2)
used a career Java developer without seeing so much as a hit of credit from the Aspect Oriented community. Perhaps
the props are out there, but I felt some more overt credit was due for the fact the Lisp has had this for some time:)
Re: (Score:2)
(Not that anyone is going to read this, so late after the article posting, on a deep comment thread, no less, but...)
The great thing is that because Lisp is so regular in its syntax and has macros available, it can be extended to include an object system, aspect oriented programming, and even language-support for the Actor model like in Erlang (for example, Termite).
I wonder if anyone has called Lisp "the oldest language of the future" yet... Then again, smarter minds than mine have already expressed their
One Book Per Language? (Score:3, Insightful)
Right, because if there is one book available free that addresses a language, there's no reason to bother having any more books covering that language.
Re: (Score:1)
With my draft ANSI K&R in hand, I don't need any other books, period.
Re: (Score:2)
Why not read a book specifically tailored to Clojure? [pragprog.com]
We need other predictors too (Score:2, Insightful)
I want it to predict whether the doctor will see the patient on time, early, or 10/20/30/40+ minutes late.
Re: (Score:2)
That's easy. Is your doctor in the USA? 40+ minutes is a safe bet IME.
No fancy statics needed!
FlightAware (Score:2)
I like to use FlightAware [flightaware.com] for realtime/historical tracking maps/stats of flights. I wish there were a way to mash up these delay data services with FlightAware.
If a Google Maps layer stream were as popular an API as RSS has become, we might see all kinds of data sources integrated into really helpful visualizations.
Business Model? (Score:2)
How are they going to keep this cluster of servers running (they use EC2, so they don't have THAT much overhead), crunching all that data? I don't see why anyone would pay anything for this, after all its just an educated guess...
Really good ETA info is available, for a price. (Score:4, Interesting)
They're scraping free data from the FAA web site [faa.gov] and FlightStats [flightstats.com], then pumping it out into an iPhone app feed.
But they're not using a really good data source. The high quality system is PASSUR RightETA [passur.com]. This system uses hundreds of radar receivers near airports to pick up the transponder signals from aircraft. It doesn't transmit. Any radar in the area that triggers an aircraft transponder causes the transponder to emit, and the PASSUR receivers pick that up. Using multiple receivers and time of flight calculations, the aircraft can be located very precisely. In fact, this is more accurate than single-point radar. You can buy a feed of this data, but it's not free.
Re: (Score:2)
If you read some of their FAQ, you'd see that they are not scraping the data from the FAA web site, but are instead modeling the air traffic control algorithms for allocating plane take-off and landing slots at the airports. They claim that the vast majority of the delays can be predicted that way (whether that's right or not is another questions).
Clojure Screencast (Score:1)
Blatant self-promotion:
I produced a commercial screencast with Phil Hagelberg, edited by Clojure creator Rich Hickey. Many people have found it useful for learning Clojure, or even just learning about what it is and what it can do:
http://peepcode.com/products/functional-programming-with-clojure [peepcode.com]
Those developers are cooler than you. (Score:2)
Re: (Score:1)
Thank you sir or madam. Your link is priceless.
Useful data, and from a third-party? (Score:2)
Oo (Score:2)
I've been out of the buzzword loop for a while, but these 'multimethods' - isn't that anything and everything that programmers should avoid like the plague ? I mean, checking the type of some vage supertype inside a function body - even perl warns against that !
Someone had to say it... (Score:2)
Seems like now that O'Reilly's publishing a LISP book, the Age of Parenthesus has come
I, for one, welcome our new parenthetical overlords.