No More QA: Yahoo's Tech Leaders Say Engineers Are Better Off Coding With No Net (ieee.org) 216
Tekla Perry writes: A year ago Yahoo eliminated its test and quality assurance team, as part of project Warp Drive, its move to continuous delivery of code. The shift wasn't easy, Yahoo tech execs say, and required some "tough parenting." But the result has been fewer errors because "when you have humans everywhere, checking this, checking that, they add so much human error into the chain that, when you take them out, even if you fail sometimes, overall you are doing better." And the pain wasn't as great as expected. Yahoo's chief architect and SVP of science and technology discuss the transition.
Sounds like an MBA plan! (Score:5, Insightful)
We were tired of being constantly bogged down by all these mistakes and bugs, so we got rid of the people who kept telling us about all the mistakes and bugs. Now our code is 100% mistake and bug free! Next step, get rid of our expensive experienced coders and replace them with cheaper outsourced coders with "equivalent" experience. We'll save so much money what could possibly go wrong? And the third and final phase of our plan is that in order to motivate our coders we will be paying a bonus that scales with the amount of code written. The more code you write, the better the bonus!
Don't you just love management?
Re: Sounds like an MBA plan! (Score:3, Interesting)
Did you get through TFS? The claim is that overall errors have been reduced but not eliminated.
He could be lying, but if that's the counterclaim, then show some evidence for it.
Re: (Score:2)
Re: Sounds like an MBA plan! (Score:5, Interesting)
While 100% was hyperbole, I think the point stands: if you remove QA then the number of *known* issues is going to go down. Whether your number of *actual* issues go down it's another matter.
I view this as the nightmare scenario, management hears words like automated testing as part of CI and then assumes automation has taken care of everything and they don't need the QA people. When the QA people's input into the issue tracker goes away, the numbers of issues in the graphs go down. Management gives themselves a pat on the back as a problematic metric has improved, without too much thought as to why.
Re: (Score:2, Redundant)
Now, at the time I was working with programmers who were much more skilled than I was, and I learned from them. Yahoo is a much larger company, with a programmers of a more varied skill-set. It is not rationa
Re: Sounds like an MBA plan! (Score:4, Insightful)
The problem with testing your own code is that you're likely to miss entire classes of bugs. You can be very effective at catching the kinds of bugs you can think of, but those are always the easiest bugs to catch in the first place. The tests you write for yourself will never do a good job of catching errors in your assumptions about what the code should do, what kinds of inputs it needs to handle, etc. Catching those kinds of conceptual bugs really requires adversarial testing from somebody who isn't starting from the same set of assumptions.
Re: Sounds like an MBA plan! (Score:5, Insightful)
You can get the same experience (but with less harshness) by giving yourself the goal to never let a bug get past you to QA. Then QA will be a kinder tutor that helps you learn.
Re: Sounds like an MBA plan! (Score:5, Insightful)
Do you think that users' data security is the proper object of harsh lessons for coders?
If you are depending on QA to make sure user data is secure, you're going to fail. That's not what QA does, and hacking is rarely in their skillset.
When companies start caring about security, they usually start a separate team "Red Team" or something whose entire job is to try to hack and find security holes in the project. (I would argue this is not sufficient, because companies who have done this still fail. If you want secure software, it has to start from the beginning. You need to teach your programmers how to write good code, and security has to be a priority on every line of code that is written. Security can't be tacked on as an afterthought).
Re: (Score:3)
SDETs can do unit and white box testing, security testing, performance testing, etc. but they cost as much as a developer, since that's what they really are.
An SDET who is good at hacking will cost more than a developer.
Re: Sounds like an MBA plan! (Score:4, Insightful)
I think it's always necessary to test your own code via unit tests and whatnot prior to letting someone else have at it, but I think it's also important to have that second set of eyes on it from a behavioral standpoint. Often we're too close to the code to really beat it up properly and subject it to really weird edge cases, and good QA people are quite adept at doing that. What I've found works pretty well is having the code go through QA, but still being held to some kind of personal accountability for it as well. Even if it's nothing more than your co-workers laughing at your QA bug reports and saying your code sucks, it's some more motivation to get it right the first time, and if it passes QA you have at least some degree of confidence that it works properly.
Another thing I've learned is that the stereotypical animosity between dev and QA is counterproductive. It's much better to think of them as two parts of a team working toward the same goal, and it's beneficial to foster an environment where the devs actively encourage QA to break stuff instead of trying to blow them off all the time.
Re: Sounds like an MBA plan! (Score:5, Insightful)
Another thing I've learned is that the stereotypical animosity between dev and QA is counterproductive.
Yes. QA is absolutely on your side!! Every time they find a bug for me, I thank them profusely, because better them than a customer.
Re: customer-reported problems (Score:2)
Management takes customer-reported problems very seriously.
And how seriously do they take ex-customers not reporting problems because they gave up in disgust instead?
Re: Sounds like an MBA plan! (Score:2, Insightful)
Call volume is unusually high. You are number [37] in line. Please be patient and listen to the pleasant music for approximately [109] minutes. [buzz buzz brrrrff buzz ggggzhxx buzz buzz ...]
Re: (Score:2, Interesting)
He could be lying, but if that's the counterclaim, then show some evidence for it.
I hold in my hand a bag containing a small, living, pink unicorn.
What do you say? I'm lying? Well, if that's the counterclaim, then show some evidence for it.
What do you say? You can't look in the bag? Well, my point exactly.
Re: (Score:2)
I can't believe it! You found my missing unicorn!
By the way, his name is Schrödinger.
Re: Sounds like an MBA plan! (Score:4, Interesting)
The claim is that overall errors have been reduced but not eliminated.
But with zero testing, how would they know?
Re: (Score:2)
> But with zero testing, how would they know?
Don't even think about believing it. If removing test was a good idea, it would have been done years ago across the industry. You seriously think an entire profession is entirely needless? Like, even for a second?
This is just some dumb scam to reduce corporate overhead to pump dem stox. You'd have more success firing the managers (another bad idea that has been tried).
Re: (Score:3)
"Number of errors" is a meaningless metric. You need to take severity, cost of fixing, cost of finding, and damage done into account. But MBA bean-counters cannot do that, as they do not understand anything they are "managing".
Re: (Score:2)
They understand if you remove people from the payroll and claim it helps, that it makes the stock go up. Say "my production code doesn't need to be tested, I found all the bugs" in a mirror. What does that person look like saying that?
Re: (Score:3)
He could be lying, but if that's the counterclaim, then show some evidence for it.
You're missing the OP's original point. It's difficult to prove anything if you've stopped measuring things.
May be it's just me because I'm not a great developer, but I work that much harder to find bugs within my programs because I know that QA will always try harder than me to find any bugs that I let through.
Furthermore, even if the number of bugs stays constant even without QA (which I very much doubt), without QA those bugs are going to be found many days if not weeks later than usual. And as a develop
Re: (Score:2)
His argument is sound though. Testers don't always follow proper testing procedure, so you'll still get plenty of bugs passing through it, and it takes way too much time for this to feed back into the develop-test cycle. Better to spend all that QA time writing more automated tests of various types, which the devs can run themselves during their much s
Re: (Score:3)
Re: (Score:3)
You forget the team lead: an older woman who doesn't understand the difference between click and double-click and has to do all her testing with one of those laptops that has a virtual scroll bar on the trackpad.
Re:Sounds like an MBA plan! (Score:5, Insightful)
The argument is actually pretty bad. By all means add more automated testing and hopefully make the QA process smoother, but skipping it entirely (they fired the QA team) is a recipe for disaster, and the only thing they have to measure the effectiveness of this is the number of issues in their issue tracker, which was *fed* by the QA team. It's like a company firing all their warranty people and bragging about how their warranty claims have gone to zero. The filed issues may be lower, but that does not speak to the final product.
Testers don't always follow proper testing procedure
This is a double edged sword in my experience. The testers are using the software much like a human would, and they do unexpected human things like the end user would be doing. This particularly helps identify usability issues, which isn't really an automation sort of thing. It's also a lot of fuzzy stuff that is usually not specifically laid out in the spec.
robustness towards matching the spec
Sometimes single minded march toward matching the spec is a bad idea, because the spec is bad and needs to be revisited. Sometimes this plays out in development, and sometimes this plays out in QA when humans clearly have a harder time with the interfaces than expected. Also generally speaking for me a spec is pretty high level, open to interpretation, and refined throughout the process as the details of the implementation get fleshed out.
Re: (Score:3)
Which can be replaced by an easy issue submission process embedded in the program itself. Combined with a program trace of the user's actions, which Yahoo already uses as part of its distributed architecture,
Re: (Score:2)
I wasn't so much about unexpected data input, but a user not knowing what the hell to do to get what they want. Functionality they don't figure out.
Which can be replaced by an easy issue submission process embedded in the program itself.
Yes, just make your users your testers, great idea. Maybe for a community free effort, but for a commercial deliverable, that's *not* where you want things to be.
humans having a hard time using an interface, this is generally obvious to the developers too
Developers are frequently not very much in tune with their target end user perspective. Hence why I see my fellow developers get frustrated with testers and even clients, wondering why the users can'
Re: (Score:3)
User acceptance testing is the only meaningful testing of this sort. There's no way around it really. Black box testers don't have the domain knowledge that end users have, nearly any usability feedback is practically meaningless. How could a blackbox tester possibly really know whether a particular screen layout, or information organization, would make sense to an end user?
Re: (Score:2)
Re: (Score:2)
Unit tests are not functional tests. You NEED functional tests, you SHOULD HAVE unit tests.
Also, why do you think TEST can't follow test procedures, but somehow SOFTWARE will?
This is just a stock pump and dump thing.
Re: (Score:3)
The summary sucks, read the article:
They got rid of the QA team and made the programmers create more unit-tests (this is key).
Because the computers made less mistakes than the QA team.
Re: (Score:2)
Re: (Score:3)
Of course that's not really true, either...
From the QA team: "Some started working on automation [for testing], and they thought that was great—that they didn't have to do the same thing over and over."
So really all they did was move their good QA folks to different departments, where they are now writing QA tests, but just don't get called QA engineers anymore, so Yahoo can get PR with nonsense stories like thi
Re: (Score:2)
They got rid of the QA team and made the programmers create more unit-tests (this is key).
The programmers should have been writing unit tests all along.
.
How is that a justification to eliminate QA?
Re: (Score:2, Informative)
The poster was close, when referring to the actual word "Mistakes" as opposed to its meaning, then the correct word was indeed "is".
So to make it perfectly clear, quotes should have been used as such:
"Mistakes" is a countable noun not a mass noun.
---
Not APK
Re: (Score:2)
Hmm... I think the AC is *mostly* correct but should have used quotes. "Mistakes" is a...
I see the less and fewer thing so many times that I'm damned near certain that few people actually know the difference. My grammar isn't really good enough to go throwing stones, so I just skip it.
Re: (Score:2)
That's a good one. Trolling is a art, after all.
Re: (Score:2)
Re: (Score:2)
Well, this is a special case where the plan is reasonable.
I mean, what good is QA when you don't have any end-users?
Re: (Score:2)
You can't make this stuff up (Score:2, Insightful)
A year ago Yahoo eliminated its test and quality assurance team
The perfect behavior for a company that is worth less than zero. (Subtract their shares of Alibaba from their market cap and you get a negative number).
Re: (Score:2)
This is no longer true because of the tax thing
Re:You can't make this stuff up (Score:4, Funny)
A year ago Yahoo eliminated its test and quality assurance team
Several years ago Yahoo eliminated it's customer base. That had a larger impact on their net worth than QA.
Do they know what QA is? (Score:5, Insightful)
Re: (Score:3)
I couldn't agree more. It sounds like they were more interested in the QA process than actually having people as a part of the overall software development team that try to break code and find flaws.
There are multiple ways to accomplish the goal of finding bugs, where the worst place to have them reported is by customers. Is that what Yahoo is hoping for now?
Re: (Score:2)
.
This.
Oh, they re-invented Test Driven Development. (Score:4, Insightful)
>It has also, he said, forced engineers to develop tools to automate the kinds of checks previously handled by teams of humans. An engineer might go through an arduous process of checking code once—but then would start building tools to automate that process.
I suspect they're too dumb to realize this and just tell everyone, "We're saving money and delivering better code using TDD"
Re: Oh, they re-invented Test Driven Development. (Score:5, Interesting)
Where are my mod points? TDD is not the elimination of test and quality assurance, it's the procedualization / automation of it - along with driving it further forward in the product development cycle.
To do TDD, you need less (or no) QA people at the end, and more QA work in early development. If you choose to do this by firing the QA department, you probably are getting your product to market slower. If, instead, you transition the frustrated programmers who have been stuck in QA since graduating into design, implementation, and maintenance of automated tests for the project, you're probably winning big.
Re: Oh, they re-invented Test Driven Development. (Score:5, Interesting)
I worked quality assurance at a very-much test level without really delving into code. Programmers can be effin' stupid at times. I was testing daemons for communications protocols. I took four approaches. Verify that it does what it's supposed to do per RFC. Investigate the behavior when confronted with stale RFCs. Investigate the behavior when it's confronted with non-RFC input that is commonly available in end-user applications (ie, non-malicious incorrect use), and investigate the behavior when malicious intent is used. I can tell you that programmers working on these protocols wrote them to current-RFC only and that they were quite upset when testing obsolete commands from previous versions of RFC broke things, or when the third-party applications sent stuff to the daemons and broke them. I would have been understanding if the software had failed gracefully, with a warning or a 500 code or the like, but when sending obsolete POP3 commands to a POP3 daemon causes it to crash hard, or when encoding an attachment with Mac BinHex instead of MIME makes it crash, both of which were user-selectable options in then-popular Eudora (the client that the company itself used) then there's a big problem. I didn't even have to get to the level of malicious testing things were so bad.
The point of a good quality assurance tester is to concoct out-of-the-box but plausible tests to try to break it. It has a lot in common with Security; it requires a degree of white-hat maliciousness and a need to use other kinds of experiences from other disciplines to devise the scenarios to use. It also requires the tester to have good communications skills as they may have to work to convince the development team that what they've found actually is a problem and being too antisocial or too introverted might not get problems corrected even if they are important and are discovered.
Yahoo is doing a hell of a job in convincing me that they're really not worth paying attention to anymore. They are taking a position that is risky and is arguably excessively risky relative to the size and position of the company.
Re: (Score:2)
This is important. I generally take the attitude that the QA people aren't idiots, and I assume from the start that whatever they bring me is in fact a problem until I discover otherwise. If there's a disagreement o
Re: (Score:2)
you transition the frustrated programmers who have been stuck in QA
Most traditional QA people I deal with are not programmers, they represent roughly the level of skill of what real end users would be.
you need less (or no) QA people at the end
If you have *no* QA people at the end, you are putting way too much faith in your automated testing. It's true that in practice the 'end' QA goes smoother when you have automated testing in CI, but the tests are not perfect and the real world will involve real humans and you need real human perspective that isn't up to their eyeballs in the implementation. When you build t
Re: (Score:3)
To do TDD, you need less (or no) QA people at the end, and more QA work in early development.
Wrong. You need less QA people at the end and more *BA* work in early development. Those are two totally different things.
Money Quotes (Score:5, Funny)
"We forced excellence into the process"
"caused a paradigm shift in how engineers thought about problems"
"even if you fail sometimes, overall you are doing better"
Re: (Score:2)
A Bullshit Bingo gold mine.
Re: (Score:2)
Hilariously full of idiot speak:
"We forced excellence into the process"
"caused a paradigm shift in how engineers thought about problems"
"even if you fail sometimes, overall you are doing better"
Lol, no shit...I'm going to print this out and frame it. And if anyone in my office ever utters any of those phrases, I'm going to hit them over the head with the framed print.
Awesome Idea! (Score:2)
Can't wait until Boeing and Airbus do this!
Re: (Score:2)
ISO 9000 has been driving "quality" into the design phase of the project for 20 years. If you've stopped "testing quality into the product at the end of development" then, you've adopted QA tech that was mandated in many industries in the mid 1990s.
Re:Awesome Idea! (Score:4, Insightful)
To pretend that there's no end-stage testing is simply ridiculous.
Re: (Score:2)
Put another way, Boeing and Airbus have been doing this for a very long time - when is the last time a passenger jet design was constructed and flown by a "test pilot" to shake out the bugs and check for nasty handling behavior?
Re: (Score:2)
With the fly-by-wire systems, all the testing can be done with mock-up flight decks and flat-screen displays. The funny thing is that glass cockpit displays only were introduced because it was cheaper to simulate expensive instruments using computer graphics, but the pilots preferred the flat screens to the actual instrumentation.
Re: (Score:2)
As an ex-Boeing guy, one of the driving forces behind the whole glass cockpit thing was the versatility of the system- the fact that a given area on the dashboard (where space is always at a premium) could be used to display anything, ie. whatever was most needed or most useful to the pilot at that moment. Infinite flexibility coupled with easy customization was a huge step forward.
Don't need the engine temp readouts at the moment? Use the same space to display the oil-pressure readings or the intake temps
Re: (Score:2)
when is the last time a passenger jet design was constructed and flown by a "test pilot" to shake out the bugs and check for nasty handling behavior?
You are aware that there still are test pilots who fly prototype commercial jet prototypes and often even push the limits [wired.com] of what those vehicles can do. They test sub-systems [youtube.com] for potential failure modes and do other things to "shake out the bugs and check for nasty handling behavior" under extreme conditions that likely would never actually happen on a routine flight.
Admittedly the engineers of these vehicles have a long history to draw from and they are doing mostly minor incremental changes with each new
Re: (Score:2)
Every aircraft made by both companies is first flown by a production test pilot before delivery.
Re: (Score:2)
Boeing takes every new airplane up on a 'shakedown flight'. I know. live right under their flight path (and I used to work there). I have an ADS-B receiver and I can see when they take off and turn right around to land again. Every once in a while, one flies over my house low and slow with the gear stuck down.
Boeing's problem is that they don't do as much 'unit testing' as they used to. Slap it together, shove the pilot into the cockpit and go. If it comes back, deliver it. They are pretty good at making s
How???? (Score:2)
How is checking for bugs supposed to add bugs? If you are not modifying the source code, it should be impossible to add bugs to it. Are they implying that these QA people were mistakenly listing features as bugs, and then the programmers were going in an removing features and replacing them with bugs?
Re: (Score:2)
I expected this reaction. You just don't get it. Testing hasn't been eliminated, simply the silo. I've worked on a few tester-free teams and it's been an unbeatable experience.
Re: (Score:2)
That would make sense.
Re: (Score:2)
By finding a bug that happens once in a blue moon when the stars are aligned, getting development to remove that bug and introduce two new ones that happen more often but can't be fixed now anymore 'cause, you know, release date.
Re: How???? (Score:2)
If software engineers know they have a safety net to pick up their mistakes, they can get complacent and become less careful about making errors, so as to get their work 'finished' quicker. The best way to avoid bugs is not to write them in the first place, but learning the degree of care and atention required is hard, and people will often (maybe instinctively) try to avoid this difficulty if they can. The root is psychology and complacency/laziness.
Re: (Score:2)
Sometimes it's not laziness on the coder's part, but rather pressure from above. Even when using the most efficient development methodologies, care and attention to detail still take time. Some coders just don't have the cojones needed to t
Re: (Score:2)
It doesn't. Rather, it's the improper QA that either introduces bugs or causes them to remain undetected.
Proper QA prevent stories like those found on The Daily WTF.
I have a better idea... (Score:2)
.
I certainly haven't seen an increase in the quality of Yahoo recently, indeed, the mail service's quality has taken a nose dive.
The supposed "success" of this experiment is probably due more to the Yahoo tech execs wanting themselves and Yahoo to look good for the sale of the company than anything else.
Re: (Score:2)
Do you use Warp Drive? Do you even know what Warp Drive is? I don't.
I call Bullshit (Score:2)
"And the pain wasn't as great as expected" (Score:2)
Pain for Yahoo? Or pain for their customers?
I don't care about the total number of errors. What counts, is the severity of those errors. I deal with a misspelled message. I can't deal with a system crash.
I believe that it is very important to have a non-coder to test software. As a friend put it:
If I print out a message saying, "Press the space bar to continue", the user will press every key except the space bar, in combination with ctrl and alt."
A coder won't do that. Coders are simply to smart to test.
Re: (Score:2)
Pain for Yahoo? Or pain for their customers?
Yahoo has the luxury of not being critical software. If there's a bug, people either work around it or use something else; when your product is given away for free, expectations are low.
I wouldn't expect the same plan to work for things like banking software, or medical imaging software, where the consequences of a bug are much more serious.
That explains a lot (Score:2)
Perfect Sense (Score:2)
My doctor is always nagging me about heart care -- less carbs, less sugar, less fatty food, exercise more... nag, nag, nag.
A few months ago I stopped visiting my doctor. I'm much more relaxed now. So far, so good.
Real engineers MUST do Q&A (Score:2)
The problem is a lot of QA isn't very good (Score:2)
Could somebody mod this guy up? (Score:2)
Just to be clear on this (Score:2)
Re: (Score:2)
My opinion on QA is that if you weren't capable enough to be a software developer then you weren't capable of being in QA either.
That applies only to white box testers who need to know how the code works to write code-specific tests. Black box testers don't to need to know how the code works to test the entire application as a user would. The best black box testers are non-programmers, as they tend not to be linear thinkers and can crash the application in odd ways.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
A Yahoo recipe for disaster... (Score:2)
Re: (Score:2)
I'm surprised that's all that happened. I would have thought there would have been some legal consequences.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
that developer should have been fired and prosecuted for vandalism and theft
It would seem that they made a strategic decision that getting him to deliver his code to keep the project close to schedule was more important.
Why I don't code anymore.. (Score:2)
Yeah, f--k that noise.
I work with money now and code for fun. Suggest those with options consider similar alternatives. Life is a lot better.
That Works To A Point (Score:2)
I do a lot of maintenance programming and come in years after the original code was written -- my last project had C f
Well, makes sense if you're shutting down (Score:2)
All the places I've worked, when the end was nigh, QA was the first to go because they were going to start laying people off soon.
Writing Code Writing Novels (Score:2)
I can see their point. The most important bug finder in software development, the one that cannot ever be properly replaced, is the original programmer. No one else can understand the underlying code well enough to replace them. And they can get lazy, with a big enough safety net they can get complacent and start to think that finding problems with what they wrote is someone else's job. But to write really good code, you also pretty much need an outside editor as well. There are many things that you will si
We did this years ago! (Score:3)
Back in the 80s, we wrote code and threw it over the fence to the users. If nobody complained, we assumed it was working fine! Ah, the good old days. What's old is new again!
Re: (Score:2)
To me, software development is more plumbing than anything else. Take data from one end of a pipe, do something with it, store it somewhere, send it off somewhere else. That's where the engineering bit comes in. Problem is, every engineer has their own standard of building stuff.
Re: (Score:2)
Software isn't real engineering, it's just patterns on a screen. How much quality assurance do you need to make a pretty picture appear on a screen?
An obvious troll here, but I'll bite.
There is a real discipline of software engineering. I'll also note that engineers from other engineering disciplines can't just pick up a programming textbook and learn on their own the skills which are needed for proper software engineering either, but it takes a whole bunch of skills and discipline that often isn't even learned in a typical Computer Science curriculum.
Besides, if you think all software development is about making pretty pictures, you are also clueless
Re: (Score:2)
Obvious troll is obvious.
Re: (Score:2)
Re: (Score:2)
My main experience with Yahoo is Yahoo Groups. Worst web app I've used on a regular basis. Pretty much nothing works as you might expect, and it certainly doesn't do it every time.
I second this. And it has been getting worse. Major function breakdowns in 2009, a new site design in 2010 that worse than the old one, then the curse of the "neo" roll-out in 2012, worse still than the one before. It is said to be the largest collection of discussion groups on the Web, but it is almost unusable. There were decent designs for discussion groups for decades to borrow from, this is not rocket science.
At this point, going to to Yahoo Groups has been so painful for so long, that I am reluctant t