Taco Bell Programming 394
theodp writes "Think outside the box? Nah, think outside the bun. Ted Dziuba argues there's a programming lesson to be learned from observing how Taco Bell manages to pull down $1.9 billion by mixing-and-matching roughly eight ingredients: 'The more I write code and design systems, the more I understand that many times, you can achieve the desired functionality simply with clever reconfigurations of the basic Unix tool set. After all, functionality is an asset, but code is a liability. This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them — Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don't overthink things, and arrive at simple, scalable solutions.'"
which language is best? (Score:5, Insightful)
I wasn't entirely convinced, but he did have the resume. Seems Mr Dziuba is from the same school of thought. I read the full introduction to the DevOps page and I'm still not entirely sure what it's about. We should work together and deliver on time, or something like that.
Re:8 keywords? (Score:5, Insightful)
exactly.
Those 8 keywords are + - > [ ] . ,
More crap from Ted Dziuba. (Score:3, Insightful)
Good grief, I think this is yet another useless article from the Ted Dziuba/Jeff Atwood/Joel Spolsky crowd. They spew out article after article after article with, in my opinion, bullshit "insights" that don't hold any water in the real world. Yet they've developed such a large online following, mainly of "web designers", "JavaScript programmers" and "NoSQL DBAs", that it tricks a lot of people in the industry into thinking what they say actually has some usefulness, when it usually doesn't.
Yeah, it's great when we can write a few shell or Perl scripts to perform simple tasks, but sometimes that's just not sufficient. Sometimes we do have to write our own code. While UNIX offers a very practical and powerful environment, we shouldn't waste our time trying to convolute its utilities to all sorts of problems, especially when it'll be quicker, easier and significantly more maintainable to roll some tools by hand.
Re:which language is best? (Score:5, Insightful)
The DevOps thing is yet another crock of shit on par with 'managing programmers is like herding cats' and web2.0
Re:8 keywords? (Score:5, Insightful)
Ook! Ook?
Simplicity (Score:5, Insightful)
The complexity people seem to delight in putting into things always amazes me. I was recently working at a major bank (they didn't like me eventually as I'm bad at authority structures). Anyway the area I was working on involved opening bank accounts from the web site. Complicated, right? The new account holder has to choose the type of account they want (of about 7), enter their details (name, address, etc), and press go. Data gets passed about, the mainframe makes the account, and we return the new account number.
Gosh.
So why, oh tell me why, did they use the following list of technologies (usually all on the same jsp page) [I may have missed some]
HTML
CSS
JSP (with pure java on the page)
Javascript (modifying the page)
JQuery
XML
XSLT
JDBC with Hibernate
JDBC without Hibernate
Custom Tag library
Spring (including AOP)
J2EE EJBs
JMS
Awesome. All this on each of the countless pages, each custom designed and built. Staggering. In fact, the site needed about 30 pages, many of them minor variations of each other. The whole thing could have been built using simple metadtata. It would have run faster, been easier to debug and test (the existing system was a nightmare), and easily changeable to suit the new business requirements that poured in.
So instead of using one efficient, smart programmer for a while, then limited support after that, they had a team of (cheap) very nervous programmers, furiously coding away, terrified lest they break something. And yes, there were layers and layers of software, each overriding the other as the new programmer didn't understand the original system, so added their own. Palimpsest, anyone?
And yet, despite my offers to rebuild the whole thing this way (including demos), management loved it. Staggering.
But I still like to keep things simple. And yes, my name is Simon. And yes, I do want a new job.
Re:Software is not food (Score:5, Insightful)
You can easily have a little more or less salt, sugar or flour in your food. However, software is not so forgiving. Change one character and you screw up badly..
Just try substituting a tsp with a tbsp of salt in your favorite recipe and then tell me food it forgiving.
Re:Simplicity (Score:5, Insightful)
Complexity creates bugs
Bugs create employment
Re:Code reuse, junk food example? (Score:5, Insightful)
I've found the best reuse comes from simple modules, not from complex ones that try to do everything. The one that tries to do everything will still be missing the one feature you need. It's easier to add the features you need to the simple one because it's, well, simpler. With the fancier one you have to work around all the features you don't need to add those that you do need, creating more reading time and more mistakes.
Agreed. With most complex frameworks there is also the additional overhead of having to do things in a particular way. If you try to do it differently or need to add a feature that wasn't designed for in the original framework, you often find yourself fighting it rather than working with it. At that point you should ditch the framework, but often it's not your decision to make, and then cost of redoing things once the framework is removed makes it impractical.
It's easy to overthink even in the simplest cases (Score:3, Insightful)
I once had a pair of command line tools that both printed lists of words (usernames, actually, one per row), and I wanted to find out how many unique ones there were. Obviously, the right hand side part of the pipeline was going to be something along the lines of " | sort -u | wc -l", but then I got utterly stuck by the left hand side. How can I combine the STDOUTs of two processes? Do I really need to resort to using temporary files? Is there really no tool to do the logical opposite of the "tee" command?
You are probably thinking: "Oh, you silly person, that's so trivial, you must be very incompetent", but in case you aren't, you might want to spend a minute trying to figure it out before reading on. I even asked a colleague for help before realizing that the reason I could not find a tool for the task was quite an obvious one: such a tool does not exist. Or actually it kinda does, but only in an implied sense: what I was hoping to achieve could be done by the humble semicolon and a pair of parens. I only had to put the two commands in parens to run them in a subshell, put a semicolon in between, so one will run after the other is finished, and I was done. I guess it was just that the logical leap from "This task is so simple, there must be a tool for this" to "just run the commands one after another" was too big for my feeble mind to accomplish.
So I guess the moral of the story is, even if you want to use just one simple tool, you may be overthinking it :-)
What is Devops? (Score:4, Insightful)
I read the linked Devops article and know even less about than before I read the article. It's full of management buzzwords and I'm sure a CIO would love it, but what does it mean?
How does Devops help?
The Devops movement is built around a group of people who believe that the application of a combination of appropriate technology and attitude can revolutionize the world of software development and delivery.
Beyond this multi-disciplinary approach, the Devops movement is attempting to encourage the development of communication skills, understanding of the domain in which the software is being written, and, crucially, a sensitivity and passion for the underlying business, and for ensuring it succeeds.
oh yeah, that clears it up. All it takes is a passion for the underlying business and it's sure to succeed!
Re:Reference ? (Score:3, Insightful)
Re:Simplicity (Score:4, Insightful)
Re:which language is best? (Score:5, Insightful)
Wget for crawling tens of millions of web pages using a 10 line script? He doesn't understand crawling at scale.
There's a lot more to it than just following links. For example, lots of servers will block you if you start ripping them in full, so you need to have a system in place to crawl sites over many days/weeks a few pages at a time. You also want to distribute the load over several IP addresses, and you need logic to handle things like auto generated/tar pits/temporarily down sites, etc. And of course you want to coordinate all that while simultaneously extracting the list of URLs that you'll hand over to the crawlers next.
His other example is also bullshit. Tens of millions of webpages are not that much for a single PC, it hardly justifies using MapReduce, especially if you're only going to process pages independently with zero communication between processes.
MapReduce is all about cutting the dataset into chunks, then alternating between 1) an (independent) processing phase on each chunk, and 2) a communication phase where the partial results are combined. And where this really pays off is when you have so much data that you need a distributed filesystem.
Re:which language is best? (Score:5, Insightful)
>>I asked him which programming language was his favorite, expecting it to be something like Lisp or Forth, but he said, "shell script."
Shell script is awesome for a large number of tasks. It can't do everything (otherwise we'd just teach scripting and be done with a CS degree in a quarter), but there's a lot of times when someone thinks they're going to have to write this long program involving a lot of text parsing and you just go, "Well, just Cut out everything except the field you want, pipe it through sort|uniq, and then run an xargs on the result." You get done in an hour (including writing, args checking, and debugging) what another person might spend a week doing in C (which is spectacularly unsuited for such tasks anyway).
Re:Code reuse, junk food example? (Score:3, Insightful)
Re:8? I thought it was 3 ... (Score:3, Insightful)
I thought there were 6 - up, down, strange, charm, top and bottom?
Re:8 keywords? (Score:3, Insightful)
I saw this same behavior in Internet explorer a few days ago. Someone complained that "Firefox isn't working", because an ASP page had a malformed link in it. IE was "smart" enough to unmangle it and display it. Firefox chose not to try to outthink the programmer and reinterpret the mess that had been passed to it. The users assumption was that Firefox was broken. I would argue the opposite.
Re:Let me tell you a story (Score:3, Insightful)
As a side note, I have a hard time with the concept that it took the VMS guys 30000 lines of code to do what could be done with a handful of regular expressions. They were either really bad at it, or it had grown for years and nobody had the guts to purge the dead code.
Re:Let me tell you a story (Score:1, Insightful)
Yeah I've heard it before too. The trick is that he's either trolling or, more likely, coming from the Windows world.
The perception is that point-and-click administration in Windows is more "professional" and less buggy than "lowly" scripts. Of course anyone who is a true admin knows that's the polar opposite of the the truth. And why one well-paid Unix admin can outperform 10 MCSEs struggling to keep their servers afloat. Microsoft has gotten better lately wrt tools, but overall Windows servers still are light years behind a *nix server with solid hardware. That's the root of the issue imho.
Re:which language is best? (Score:3, Insightful)
(otherwise we'd just teach scripting and be done with a CS degree in a quarter)
because the programming language has so much to do with CS?
Re:Let me tell you a story (Score:3, Insightful)
This kind of story makes me laugh when I see/hear anecdotes that have management talking about metrics like LoC.
Re:Code reuse, junk food example? (Score:2, Insightful)
Re:Let me tell you a story (Score:4, Insightful)
Re:Let me tell you a story (Score:5, Insightful)
This seems to be an odd criticism.
It's like calling perl/python/C subroutines "brittle" because if you change the arguments or return values of any of them, all hell can break loose.
'Brittle' to me means that ridiculous assumptions don't hold true *often*, leading to breakage, like this gem I found the other day in a developer's installation script:
envfile = `ls -1tr /tmp/*.tar | tail -1` /apps/prod /apps/prod/${envfile}
cp ${envfile}
tar -xvf
In other words - the envfile is "the single most recent tar file in the /tmp directory," with no checks or verification to ensure that it was the right one before you blasted it on into production.
That's what I'd call 'brittle' programming, anyway - likely to break, and break spectacularly because you haven't thought through your requirements clearly, or bothered to verify that inputs are reasonable and sane.
Re:which language is best? (Score:5, Insightful)
Re:Let me tell you a story (Score:3, Insightful)
Why would you assume he's talking about pipe I/O. If you're talking portability and dependencies, then yes, I'd say something that uses a bunch of smaller tools might have more brittleness than something that is entirely contained in code controlled by the maintainer. Its really not that far a stretch to say that you upgrade a machine and a newer version of some utility changes some output your script depends on, and boom, your process now comes to a halt. That's what brittle means in this situation.
Re:8 keywords? (Score:3, Insightful)
"When you work in a monkeyhouse you're more used to having shit thrown at you".
Re:Let me tell you a story (Score:5, Insightful)
and how is that any different from a python library being updated and changing your program. Completely pointless argument.
Re:Let me tell you a story (Score:3, Insightful)
Fixed that for you. In other words your argument can be applied to any programming anywhere.
Re:Let me tell you a story (Score:1, Insightful)
Yeah, Unix is great but I think pipes and awk are a bit brittle for production.
One reason I would tend to stay away from shell scripts is that they're not brittle enough. I do scientific programming and I would rather have a program crash hard and obvious than have it silently "fix" an error and give the wrong result.
If I use a language like Java then I can write a program where unless everything is exactly as I expect then there's a full crash (array out of bounds, etc.). But that seems to be harder with shell scripts.
Re:Let me tell you a story (Score:4, Insightful)
LOCs is roughly as meaningless as valuing a document by its word count. You could spend tons on research on something summed up in a few pages, or get an endless word diarrhea of mindless babble spewed out at 300 WPM. But people need to measure progress. Yes, I've seen how it gets when nobody measures progress and everyone pretends the last 10% of code will suddenly turn a turd into a gem, if so expect the people with survival skills to disappear some 80% into the project. Another disastrous variation is to leave it entirely up to the subjective opinion of the manager, which in any sizable company means your career depends on your favor with the PHB and his lying skills compared to the other PHBs.
Saying it's bad is like shooting fish in a barrel. Coming up with a good system of objectively measuring code design and quality that works in a large organization is ridiculously hard. Particularly since everybody tries to wiggle out of the definitions and go with what you measure, if you made avoiding LoC a metric then the lines would compacted to the point of obfuscation with hideous cross calling to save lines. You want people to hit a sane level of structuring and code reuse, neither LoC bloat nor 4k compos.
Re:which language is best? (Score:3, Insightful)
IIRC, uniq only collapses adjacent lines that are identical. So hence sort|uniq.
Maybe there's a flag or something that will change that behavior? It'd probably have to do something on the order of sort anyway, though.
Re:which language is best? (Score:5, Insightful)
>>A skilled C programmer also needs less than 1 hour for something like that.
Hmm, well if you want to time yourself, here's a common enough task that I automate with shell scripts. I just timed myself. Including logging in, doing a detour into man and a 'locate access_log' to find the file, it took a bit less than 4 minutes.
tail -n 100 /var/log/apache2/access_log | cut -f1 -d" " | sort | uniq
Grabs the end of the access_log and shows you the last few ip addresses that have connected to your site. I do something like this occasionally. Optionally pipe it into xargs host to do DNS lookups on them, if that's how you prefer to roll.
I'm honestly curious how long it will take you to do it in C, with/without the DNS lookup. Post source if you don't mind.
Re:Weak error handling (Score:5, Insightful)
It is interesting that wget does not handle errors other than ignoring them and trying to continue. The original poster's first and second point are not addressed. Does that mean the operator has to manually monitor the crons and restart the ones that failed?
The site is really, really slow. Some sites will take half an hour to feed out a page.
And you still haven't looked at the wget(1) man page, or you'd know about the --read-timeout parameter.
Maybe they're overloaded. Maybe their denial of service detection software has tripped and is metering out bytes very slowly in defense. You don't want this to hold up the entire operation. Last week, for some reason, "orbitz.com" did that.
Not holding up your operation is why you use multiple tools that can run concurrently. A wget of orbitz.com taking forever won't prevent the wget of soggy.com that you scheduled for half an hour later, and neither will stop the parser.
Of course, if you design an all-eggs-in-one-basket solution that depends on sequential operations, you deserve what you get.
How do you schedule orbitz.com to go off and then soggy.com to go off later? What of you are handling hundreds of different web sites? Hundreds of crons? How do you retry later on sites that are very slow at the moment? How would you know that wget timed out due to slow download?
The site doesn't return data at all. Some British university sites have a network implementation which, if asked for a HTTPS connection, does part of the SSL connection handshake and then just stops, leaving the TCP connection open but sending nothing.
This requires a special timeout.
Yes, the --connect-timeout.
The connection has been made so it is not --connect-timeout it is --read-timeout. That is the problem, there is no different timeout when you are slowly getting data vs getting no data.
The site doesn't like too many simultaneous connections from the same IP address. We limit our system to three simultaneous connections to a given site, so as not to overload it.
wget limits to a single connection with keep-alive per instance. (If you want more, spawn more wget -nc commands)
You missed the point; it is not more connections it is limiting connections. Say I am crawling five different sites using host spanning and they all link to the same site. Since there is no coordination between the wgets it is possible for all of the to connect to the same site at the same time. What if I have 100 crawlers at the same time?
The original poster is right; using wget ignores errors (timesout) and does not report them so there is no way of programaticly figuring out what went wrong and react to it.
Things wget does not do: avoid known non responsive pages, requeue requests that have timed out or log them so that are not tried again, coordinate multiple crawls so they do not hit the same server simultaneously, handle errors itself. There are probably more.
This is a perfect example of the 80/20 rule. The "solution" may cover 80% of the problem but that final 20% will require so much babysitting as to make it unusable. Wget is not an enterprise level web crawler.
Re:Let me tell you a story (Score:3, Insightful)
One, that very rarely happens with the common unix utilities.
Two, ever heard of change management, testing and QA? If you're the kind of idiot who flings patches and updates willy-nilly onto a production box then sorry, but you deserve to have the sky fall on your head.
Re:which language is best? (Score:3, Insightful)
Actually, I have a masters in CS. I was trying to make a lexical pun of sorts saying it's not about "programming languages" but about "programming", which, in my mind, is more about the problem solving and design than the actual implementation of a particular program. Once you learn how to program - how to solve problems and design a solution - implementing it in any particular language is just a matter of getting the syntax right.
The programming language itself is a tool. Any particular problem can have solutions in any number of languages, and certain languages are more suited than others for particular problem classes.