Slashdot is powered by your submissions, so send in your scoop


Forgot your password?
Businesses Programming Unix

Taco Bell Programming 394

theodp writes "Think outside the box? Nah, think outside the bun. Ted Dziuba argues there's a programming lesson to be learned from observing how Taco Bell manages to pull down $1.9 billion by mixing-and-matching roughly eight ingredients: 'The more I write code and design systems, the more I understand that many times, you can achieve the desired functionality simply with clever reconfigurations of the basic Unix tool set. After all, functionality is an asset, but code is a liability. This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them — Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don't overthink things, and arrive at simple, scalable solutions.'"
This discussion has been archived. No new comments can be posted.

Taco Bell Programming

Comments Filter:
  • by phantomfive ( 622387 ) on Sunday October 24, 2010 @05:49PM (#34006690) Journal
    Reminds me of a job interview I did once with an old guy, he had around 30 different programming languages on his resume. I asked him which programming language was his favorite, expecting it to be something like Lisp or Forth, but he said, "shell script." I was a bit surprised, but he said, "it lets me tie pieces in from everywhere and do it all with the least amount of code."

    I wasn't entirely convinced, but he did have the resume. Seems Mr Dziuba is from the same school of thought. I read the full introduction to the DevOps page and I'm still not entirely sure what it's about. We should work together and deliver on time, or something like that.
  • Re:8 keywords? (Score:5, Insightful)

    by Anonymous Coward on Sunday October 24, 2010 @05:52PM (#34006696)


    Those 8 keywords are + - > [ ] . ,

  • by Anonymous Coward on Sunday October 24, 2010 @05:52PM (#34006700)

    Good grief, I think this is yet another useless article from the Ted Dziuba/Jeff Atwood/Joel Spolsky crowd. They spew out article after article after article with, in my opinion, bullshit "insights" that don't hold any water in the real world. Yet they've developed such a large online following, mainly of "web designers", "JavaScript programmers" and "NoSQL DBAs", that it tricks a lot of people in the industry into thinking what they say actually has some usefulness, when it usually doesn't.

    Yeah, it's great when we can write a few shell or Perl scripts to perform simple tasks, but sometimes that's just not sufficient. Sometimes we do have to write our own code. While UNIX offers a very practical and powerful environment, we shouldn't waste our time trying to convolute its utilities to all sorts of problems, especially when it'll be quicker, easier and significantly more maintainable to roll some tools by hand.

  • by visualight ( 468005 ) on Sunday October 24, 2010 @06:02PM (#34006750) Homepage

    The DevOps thing is yet another crock of shit on par with 'managing programmers is like herding cats' and web2.0

  • Re:8 keywords? (Score:5, Insightful)

    by hardburn ( 141468 ) <[hardburn] [at] []> on Sunday October 24, 2010 @06:07PM (#34006788)

    Ook! Ook?

  • Simplicity (Score:5, Insightful)

    by SimonInOz ( 579741 ) on Sunday October 24, 2010 @06:11PM (#34006808)

    The complexity people seem to delight in putting into things always amazes me. I was recently working at a major bank (they didn't like me eventually as I'm bad at authority structures). Anyway the area I was working on involved opening bank accounts from the web site. Complicated, right? The new account holder has to choose the type of account they want (of about 7), enter their details (name, address, etc), and press go. Data gets passed about, the mainframe makes the account, and we return the new account number.


    So why, oh tell me why, did they use the following list of technologies (usually all on the same jsp page) [I may have missed some]
    JSP (with pure java on the page)
    Javascript (modifying the page)
    JDBC with Hibernate
    JDBC without Hibernate
    Custom Tag library
    Spring (including AOP)
    J2EE EJBs

    Awesome. All this on each of the countless pages, each custom designed and built. Staggering. In fact, the site needed about 30 pages, many of them minor variations of each other. The whole thing could have been built using simple metadtata. It would have run faster, been easier to debug and test (the existing system was a nightmare), and easily changeable to suit the new business requirements that poured in.

    So instead of using one efficient, smart programmer for a while, then limited support after that, they had a team of (cheap) very nervous programmers, furiously coding away, terrified lest they break something. And yes, there were layers and layers of software, each overriding the other as the new programmer didn't understand the original system, so added their own. Palimpsest, anyone?

    And yet, despite my offers to rebuild the whole thing this way (including demos), management loved it. Staggering.

    But I still like to keep things simple. And yes, my name is Simon. And yes, I do want a new job.

  • by GiveBenADollar ( 1722738 ) on Sunday October 24, 2010 @06:13PM (#34006822)

    You can easily have a little more or less salt, sugar or flour in your food. However, software is not so forgiving. Change one character and you screw up badly..

    Just try substituting a tsp with a tbsp of salt in your favorite recipe and then tell me food it forgiving.

  • Re:Simplicity (Score:5, Insightful)

    by MichaelSmith ( 789609 ) on Sunday October 24, 2010 @06:14PM (#34006830) Homepage Journal

    Complexity creates bugs

    Bugs create employment

  • by syousef ( 465911 ) on Sunday October 24, 2010 @06:33PM (#34006934) Journal

    I've found the best reuse comes from simple modules, not from complex ones that try to do everything. The one that tries to do everything will still be missing the one feature you need. It's easier to add the features you need to the simple one because it's, well, simpler. With the fancier one you have to work around all the features you don't need to add those that you do need, creating more reading time and more mistakes.

    Agreed. With most complex frameworks there is also the additional overhead of having to do things in a particular way. If you try to do it differently or need to add a feature that wasn't designed for in the original framework, you often find yourself fighting it rather than working with it. At that point you should ditch the framework, but often it's not your decision to make, and then cost of redoing things once the framework is removed makes it impractical.

  • by Meriahven ( 1154311 ) on Sunday October 24, 2010 @06:39PM (#34006966)

    I once had a pair of command line tools that both printed lists of words (usernames, actually, one per row), and I wanted to find out how many unique ones there were. Obviously, the right hand side part of the pipeline was going to be something along the lines of " | sort -u | wc -l", but then I got utterly stuck by the left hand side. How can I combine the STDOUTs of two processes? Do I really need to resort to using temporary files? Is there really no tool to do the logical opposite of the "tee" command?

    You are probably thinking: "Oh, you silly person, that's so trivial, you must be very incompetent", but in case you aren't, you might want to spend a minute trying to figure it out before reading on. I even asked a colleague for help before realizing that the reason I could not find a tool for the task was quite an obvious one: such a tool does not exist. Or actually it kinda does, but only in an implied sense: what I was hoping to achieve could be done by the humble semicolon and a pair of parens. I only had to put the two commands in parens to run them in a subshell, put a semicolon in between, so one will run after the other is finished, and I was done. I guess it was just that the logical leap from "This task is so simple, there must be a tool for this" to "just run the commands one after another" was too big for my feeble mind to accomplish.

    So I guess the moral of the story is, even if you want to use just one simple tool, you may be overthinking it :-)

  • What is Devops? (Score:4, Insightful)

    by hawguy ( 1600213 ) on Sunday October 24, 2010 @06:41PM (#34006968)

    I read the linked Devops article and know even less about than before I read the article. It's full of management buzzwords and I'm sure a CIO would love it, but what does it mean?

    How does Devops help?

    The Devops movement is built around a group of people who believe that the application of a combination of appropriate technology and attitude can revolutionize the world of software development and delivery.


    Beyond this multi-disciplinary approach, the Devops movement is attempting to encourage the development of communication skills, understanding of the domain in which the software is being written, and, crucially, a sensitivity and passion for the underlying business, and for ensuring it succeeds.

    oh yeah, that clears it up. All it takes is a passion for the underlying business and it's sure to succeed!

  • Re:Reference ? (Score:3, Insightful)

    by lostmongoose ( 1094523 ) on Sunday October 24, 2010 @06:49PM (#34007016)
    I'm more disturbed by the fact that you know what dead donkey ass tastes like...
  • Re:Simplicity (Score:4, Insightful)

    by NorbrookC ( 674063 ) on Sunday October 24, 2010 @06:53PM (#34007032) Journal
    It seems to me that the point is that programmers have a variant of "if all you have is a hammer, every problem is a nail" saying. In this case, they have a huge toolbox, so every time they need to drive a nail, it means that they must design and use a methodology that will, eventually, cause the nail to be pushed into place, instead of just reaching for the hammer and getting the job done.
  • by martin-boundary ( 547041 ) on Sunday October 24, 2010 @06:59PM (#34007068)
    Sadly, Mr Dziuba has the right idea but uses terrible examples in his blogpost.

    Wget for crawling tens of millions of web pages using a 10 line script? He doesn't understand crawling at scale.

    There's a lot more to it than just following links. For example, lots of servers will block you if you start ripping them in full, so you need to have a system in place to crawl sites over many days/weeks a few pages at a time. You also want to distribute the load over several IP addresses, and you need logic to handle things like auto generated/tar pits/temporarily down sites, etc. And of course you want to coordinate all that while simultaneously extracting the list of URLs that you'll hand over to the crawlers next.

    His other example is also bullshit. Tens of millions of webpages are not that much for a single PC, it hardly justifies using MapReduce, especially if you're only going to process pages independently with zero communication between processes.

    MapReduce is all about cutting the dataset into chunks, then alternating between 1) an (independent) processing phase on each chunk, and 2) a communication phase where the partial results are combined. And where this really pays off is when you have so much data that you need a distributed filesystem.

  • by ShakaUVM ( 157947 ) on Sunday October 24, 2010 @07:02PM (#34007088) Homepage Journal

    >>I asked him which programming language was his favorite, expecting it to be something like Lisp or Forth, but he said, "shell script."

    Shell script is awesome for a large number of tasks. It can't do everything (otherwise we'd just teach scripting and be done with a CS degree in a quarter), but there's a lot of times when someone thinks they're going to have to write this long program involving a lot of text parsing and you just go, "Well, just Cut out everything except the field you want, pipe it through sort|uniq, and then run an xargs on the result." You get done in an hour (including writing, args checking, and debugging) what another person might spend a week doing in C (which is spectacularly unsuited for such tasks anyway).

  • by aztracker1 ( 702135 ) on Sunday October 24, 2010 @07:25PM (#34007220) Homepage
    I think this is one of the reasons why jQuery has become so popular... it does "just enough" in a consistent (relatively) way, with a decent plug-in model... so stringing things together works pretty well, and there is usually a plugin for just about anything you'd want/need. Though it's maybe a bit heavier than hand crafted code, stringing jQuery and plugins is less debt, with more reuse. I do have a few things in my current js toolbox... namely some JS extensions, json2 (modified), date.js, jquery, jquery templates and jquery validation... from there I tweak the ui with PIE and wire up the validators... jquery is about 70k (min uncompressed), with 60k for my js/browser extensions, and 60k for jquery extensions... another 20K in site/app scripts... usually dead cache load is around 400k on a page (around 220k with gzip) and in about 12-16 http requests... using deferred script calls, under a second for initial load, and under another 1.4 seconds for js & binding... A bit thick, but works very well.
  • by dakameleon ( 1126377 ) on Sunday October 24, 2010 @07:44PM (#34007332)

    I thought there were 6 - up, down, strange, charm, top and bottom?

  • Re:8 keywords? (Score:3, Insightful)

    by tompaulco ( 629533 ) on Sunday October 24, 2010 @07:49PM (#34007368) Homepage Journal
    I find that most defects in the English that is produced is due to the use of words that are not in the vocabulary. A sufficiently intelligent compiler (listener) is able to successfully compile the code even though the programmer(speaker) did not write it correctly, which unfortunately only reinforces the bad habit of the programmer.
    I saw this same behavior in Internet explorer a few days ago. Someone complained that "Firefox isn't working", because an ASP page had a malformed link in it. IE was "smart" enough to unmangle it and display it. Firefox chose not to try to outthink the programmer and reinterpret the mess that had been passed to it. The users assumption was that Firefox was broken. I would argue the opposite.
  • by Jaime2 ( 824950 ) on Sunday October 24, 2010 @07:59PM (#34007422)
    BTW, awk is a programming language. Really, all you did was to write their process in a different language, not convert it from a custom program to some built in tools.

    As a side note, I have a hard time with the concept that it took the VMS guys 30000 lines of code to do what could be done with a handful of regular expressions. They were either really bad at it, or it had grown for years and nobody had the guts to purge the dead code.
  • by Anonymous Coward on Sunday October 24, 2010 @08:19PM (#34007542)

    Yeah I've heard it before too. The trick is that he's either trolling or, more likely, coming from the Windows world.

    The perception is that point-and-click administration in Windows is more "professional" and less buggy than "lowly" scripts. Of course anyone who is a true admin knows that's the polar opposite of the the truth. And why one well-paid Unix admin can outperform 10 MCSEs struggling to keep their servers afloat. Microsoft has gotten better lately wrt tools, but overall Windows servers still are light years behind a *nix server with solid hardware. That's the root of the issue imho.

  • by gangien ( 151940 ) on Sunday October 24, 2010 @08:37PM (#34007674) Homepage

    (otherwise we'd just teach scripting and be done with a CS degree in a quarter)

    because the programming language has so much to do with CS?

  • by symbolic ( 11752 ) on Sunday October 24, 2010 @09:08PM (#34007890)

    This kind of story makes me laugh when I see/hear anecdotes that have management talking about metrics like LoC.

  • by scourfish ( 573542 ) <scourfish@y a h o o . c om> on Sunday October 24, 2010 @09:27PM (#34007994)
    Your argument is terribly flawed; Taco Bell doesn't serve French fries.
  • by Securityemo ( 1407943 ) on Sunday October 24, 2010 @09:32PM (#34008016) Journal
    The likelihood of pipe I/O changing in the basic UNIX toolkit is near zero. Or is it just that you (and/or managers and other "people-person" types) need someone to sign the dotted line for you to feel certain that things are as they should?
  • by Americano ( 920576 ) on Sunday October 24, 2010 @09:36PM (#34008034)

    This seems to be an odd criticism.

    It's like calling perl/python/C subroutines "brittle" because if you change the arguments or return values of any of them, all hell can break loose.

    'Brittle' to me means that ridiculous assumptions don't hold true *often*, leading to breakage, like this gem I found the other day in a developer's installation script:

              envfile = `ls -1tr /tmp/*.tar | tail -1`
              cp ${envfile} /apps/prod
              tar -xvf /apps/prod/${envfile}

    In other words - the envfile is "the single most recent tar file in the /tmp directory," with no checks or verification to ensure that it was the right one before you blasted it on into production.

    That's what I'd call 'brittle' programming, anyway - likely to break, and break spectacularly because you haven't thought through your requirements clearly, or bothered to verify that inputs are reasonable and sane.

  • by itlurksbeneath ( 952654 ) on Sunday October 24, 2010 @10:13PM (#34008186) Journal
    Bingo. CS has nothing to do with programming languages. It's about PROGRAMMING. Lots of CS grads still don't get this. They are typically the mediocre programmers that move on to project management (or something else that doesn't involve programming) fairly quickly. Or they end up doing horrible ASP web apps and Microsoft Access front ends.
  • by badboy_tw2002 ( 524611 ) on Sunday October 24, 2010 @10:28PM (#34008270)

    Why would you assume he's talking about pipe I/O. If you're talking portability and dependencies, then yes, I'd say something that uses a bunch of smaller tools might have more brittleness than something that is entirely contained in code controlled by the maintainer. Its really not that far a stretch to say that you upgrade a machine and a newer version of some utility changes some output your script depends on, and boom, your process now comes to a halt. That's what brittle means in this situation.

  • Re:8 keywords? (Score:3, Insightful)

    by Shadow of Eternity ( 795165 ) on Monday October 25, 2010 @12:10AM (#34008710)

    "When you work in a monkeyhouse you're more used to having shit thrown at you".

  • by LingNoi ( 1066278 ) on Monday October 25, 2010 @12:15AM (#34008734)

    and how is that any different from a python library being updated and changing your program. Completely pointless argument.

  • by LingNoi ( 1066278 ) on Monday October 25, 2010 @12:20AM (#34008762)

    C++ libraries are brittle because as a whole, and as far as their individual parts go, they assume a certain input set and generate a certain output set. If these assumptions turn out to be incorrect they will fail, sometimes spectacularly, and often it will take a serious amount of time and effort to determine exactly where the problem is.

    Fixed that for you. In other words your argument can be applied to any programming anywhere.

  • by Anonymous Coward on Monday October 25, 2010 @01:08AM (#34008936)

    Yeah, Unix is great but I think pipes and awk are a bit brittle for production.

    One reason I would tend to stay away from shell scripts is that they're not brittle enough. I do scientific programming and I would rather have a program crash hard and obvious than have it silently "fix" an error and give the wrong result.

    If I use a language like Java then I can write a program where unless everything is exactly as I expect then there's a full crash (array out of bounds, etc.). But that seems to be harder with shell scripts.

  • by Kjella ( 173770 ) on Monday October 25, 2010 @01:19AM (#34008980) Homepage

    LOCs is roughly as meaningless as valuing a document by its word count. You could spend tons on research on something summed up in a few pages, or get an endless word diarrhea of mindless babble spewed out at 300 WPM. But people need to measure progress. Yes, I've seen how it gets when nobody measures progress and everyone pretends the last 10% of code will suddenly turn a turd into a gem, if so expect the people with survival skills to disappear some 80% into the project. Another disastrous variation is to leave it entirely up to the subjective opinion of the manager, which in any sizable company means your career depends on your favor with the PHB and his lying skills compared to the other PHBs.

    Saying it's bad is like shooting fish in a barrel. Coming up with a good system of objectively measuring code design and quality that works in a large organization is ridiculously hard. Particularly since everybody tries to wiggle out of the definitions and go with what you measure, if you made avoiding LoC a metric then the lines would compacted to the point of obfuscation with hideous cross calling to save lines. You want people to hit a sane level of structuring and code reuse, neither LoC bloat nor 4k compos.

  • by ShakaUVM ( 157947 ) on Monday October 25, 2010 @02:02AM (#34009128) Homepage Journal

    You prolly should pipe through uniq before sort. You'll get the same results, but sort will be passed less ata leading to faster execution and smaller memory footprint.

    IIRC, uniq only collapses adjacent lines that are identical. So hence sort|uniq.

    Maybe there's a flag or something that will change that behavior? It'd probably have to do something on the order of sort anyway, though.

  • by ShakaUVM ( 157947 ) on Monday October 25, 2010 @03:40AM (#34009516) Homepage Journal

    >>A skilled C programmer also needs less than 1 hour for something like that.

    Hmm, well if you want to time yourself, here's a common enough task that I automate with shell scripts. I just timed myself. Including logging in, doing a detour into man and a 'locate access_log' to find the file, it took a bit less than 4 minutes.

    tail -n 100 /var/log/apache2/access_log | cut -f1 -d" " | sort | uniq

    Grabs the end of the access_log and shows you the last few ip addresses that have connected to your site. I do something like this occasionally. Optionally pipe it into xargs host to do DNS lookups on them, if that's how you prefer to roll.

    I'm honestly curious how long it will take you to do it in C, with/without the DNS lookup. Post source if you don't mind.

  • by jklovanc ( 1603149 ) on Monday October 25, 2010 @04:11AM (#34009628)

    It is interesting that wget does not handle errors other than ignoring them and trying to continue. The original poster's first and second point are not addressed. Does that mean the operator has to manually monitor the crons and restart the ones that failed?

    The site is really, really slow. Some sites will take half an hour to feed out a page.

    And you still haven't looked at the wget(1) man page, or you'd know about the --read-timeout parameter.

    Maybe they're overloaded. Maybe their denial of service detection software has tripped and is metering out bytes very slowly in defense. You don't want this to hold up the entire operation. Last week, for some reason, "" did that.

    Not holding up your operation is why you use multiple tools that can run concurrently. A wget of taking forever won't prevent the wget of that you scheduled for half an hour later, and neither will stop the parser.
    Of course, if you design an all-eggs-in-one-basket solution that depends on sequential operations, you deserve what you get.

    How do you schedule to go off and then to go off later? What of you are handling hundreds of different web sites? Hundreds of crons? How do you retry later on sites that are very slow at the moment? How would you know that wget timed out due to slow download?

    The site doesn't return data at all. Some British university sites have a network implementation which, if asked for a HTTPS connection, does part of the SSL connection handshake and then just stops, leaving the TCP connection open but sending nothing.
    This requires a special timeout.

    Yes, the --connect-timeout.

    The connection has been made so it is not --connect-timeout it is --read-timeout. That is the problem, there is no different timeout when you are slowly getting data vs getting no data.

    The site doesn't like too many simultaneous connections from the same IP address. We limit our system to three simultaneous connections to a given site, so as not to overload it.

    wget limits to a single connection with keep-alive per instance. (If you want more, spawn more wget -nc commands)

    You missed the point; it is not more connections it is limiting connections. Say I am crawling five different sites using host spanning and they all link to the same site. Since there is no coordination between the wgets it is possible for all of the to connect to the same site at the same time. What if I have 100 crawlers at the same time?

    The original poster is right; using wget ignores errors (timesout) and does not report them so there is no way of programaticly figuring out what went wrong and react to it.
    Things wget does not do: avoid known non responsive pages, requeue requests that have timed out or log them so that are not tried again, coordinate multiple crawls so they do not hit the same server simultaneously, handle errors itself. There are probably more.

    This is a perfect example of the 80/20 rule. The "solution" may cover 80% of the problem but that final 20% will require so much babysitting as to make it unusable. Wget is not an enterprise level web crawler.

  • by Hognoxious ( 631665 ) on Monday October 25, 2010 @06:03AM (#34010032) Homepage Journal

    If any single one of the tools you're running on changes its input or output even slightly the whole thing can fall apart in a rather spectacular manner.

    One, that very rarely happens with the common unix utilities.

    Two, ever heard of change management, testing and QA? If you're the kind of idiot who flings patches and updates willy-nilly onto a production box then sorry, but you deserve to have the sky fall on your head.

  • by itlurksbeneath ( 952654 ) on Monday October 25, 2010 @08:48AM (#34010788) Journal

    Actually, I have a masters in CS. I was trying to make a lexical pun of sorts saying it's not about "programming languages" but about "programming", which, in my mind, is more about the problem solving and design than the actual implementation of a particular program. Once you learn how to program - how to solve problems and design a solution - implementing it in any particular language is just a matter of getting the syntax right.

    The programming language itself is a tool. Any particular problem can have solutions in any number of languages, and certain languages are more suited than others for particular problem classes.

Someday your prints will come. -- Kodak