At EA, It Can Take a Whole Day To Change 3 Lines of Code (neowin.net) 145
New submitter segaboy81 writes: In 2001 the Manifesto for Agile Software Development was born, and it took the software engineering world by storm. Linux, Windows, Facebook, AAA games, and just about everything else, adheres to this manifesto in some form or another. It is a paradigm that allows teams to work collaboratively on projects in the most effective and streamlined way possible. However, EA may not have gotten the memo. According to a blogpost by former EA developer Adam Berg, different teams take very different approaches to development with one team in particular being especially slow to progress. Adam recounts his experience on the FIFA team where he worked on the Wii, PS Vita, and Nintendo 3DS ports of the game: "I often worked in the realm of competition logic. Testing changes here could mean progressing through several seasons of career mode in order to test out a change. No joke, it would take an entire day to change 3 lines of code and know that it actually worked correctly."
Which three lines of code? (Score:5, Insightful)
Sure its easy to say that it's just "three lines of code", but is the code in a critical performance path in the software? One that where a minor change can alter the performance drastically?
Not all three lines of code are the same.
Re: (Score:2)
Re: (Score:2)
It is because few here have software, much less a honest days work.
Translation?
Er moint gwaaa.
Re:Which three lines of code? (Score:5, Informative)
The headiline should have said "At EA it can take a day to perform full regression testing on 3 lines of changed code". But that's not news...
Re: (Score:2)
Re: (Score:2)
It can take a week to get approval after testing says it's good.
Re: (Score:2)
I was on a project for autonomous driving and driver assistance systems.
Testing involves (amoung automatic tests) driving the car on the road, and making sure changed lines are actually executed during test drives.
Could take days or weeks until the conditions were met that the lines actually were executed.
Re:Which three lines of code? (Score:5, Funny)
The headiline should have said "At EA it can take a day to perform full regression testing on 3 lines of changed code". But that's not news...
EA doing testing? Are you kidding, that would be the biggest news of the year.
Re: (Score:2)
3 lines of code that fast seems a bit rushed. Seriously. Unless you're doing web apps with continuous customer-tested delivery, you need to verify the software before anyone else sees it.
Where's QA? (Score:3, Insightful)
This is a major game publisher, where's the QA team? Their literal purpose is so much higher paid programmers don't have to spend an entire day testing a small change.
Re: (Score:2)
You seem to be under the mistaken impression that EA has proper QA. (As in properly trained and paid)...
Re: (Score:2)
Fair point. I actually worked in a major game publishers QA department for a few years in the late 90's and early 2000's and it was great work for a kid just getting out into the world but I hear nowadays it's just shit working in any capacity for the major publishers.
Re:Where's QA? (Score:4, Informative)
It's the programmer's job to write correct code. You're not supposed to submit untested code and wait for QA to send it back to you.
That's how I've always worked, anyway. I say it takes half a day to change a line of code, including time to test it, commit it, update the bug report, etc.
Re: (Score:3)
Often it makes sense for each developer to have a personal git repo, then test their changes and if all works well, send a pull request upstream.
It might make sense instead to have a personal branch from the dev branch for that purpose.
Reading further in TFA, some silos don't have any sort of test framework. If you change 3 lines affecting what happens if you frotz the troll you must then compile the whole game and play it from start until you get to the troll on level 12 in order to test.
It didn't directly
Re: (Score:2)
It's the programmer's job to write correct code. You're not supposed to submit untested code and wait for QA to send it back to you.
Sure, though the "Testing changes here could mean progressing through several seasons of career mode in order to test out a change." from the summary sounds more like QA, of course that depends a lot on the specific change.
That's how I've always worked, anyway. I say it takes half a day to change a line of code, including time to test it, commit it, update the bug report, etc.
Seems about right. And given that changing a line of code takes half a day it's worth spending the time to make sure someone won't ne
Re: (Score:2)
In my world (and job), I'm responsible my code works correctly within the entire product.
QA steps in to test on different combinations of hardware/OS, and to bombard my code to trigger "once in a blue moon" bugs.
Re: (Score:2)
Probably works in self contained products. I have worked on a project where a central business system is connected to 25 different systems. Any change would go through 8 levels of approvals. Any code change even a single line will probably billed a minimum of 40 hours.
Re: (Score:2)
The "no silver bullet" adage comes to mind...
Re: (Score:2)
Yeah, no, that's not how it works in the games industry for a few reasons.
First of all, we are BADLY understaffed when it comes to QA. The salaries are dismally low, and as important as QA are, most companies are running with 40-60% of what I would consider adequate.
Second, as a programmer, you should be testing your code. It's insane to check it in without a good sense of it's working or not. Particularly AI code like is mentioned here. QA will probably still find bugs in it, but if you didn't at least get
Re: Where's QA? (Score:2)
Re: (Score:2)
If the AI is supposed to be better or more aggressive or something, a unit test won't tell you that. Indeed, new AI code is put in specifically to change the outcome of a particular situation; a unit test is the exact opposite of the thing you want.
That is not what a unit test is supposed to do anyway,
The word "unit" means traditionally "compilation unit". In other words a "unit test", is supposed to test the functions/methods in a unit, e.g. a single class, or a single C file. Run it once, to get the idea
Re: (Score:2)
Only if you don't care about safety and security. Most formal processes don't let coders approve their own code; they specifically require independent tests (yes even unit tests) to protect against even unintended author bias. And yes that does take extra time and effort.
Of course, I'm always battling with the tension between Agile and CI/CD and formal engineering processes for safety-critical systems...
Re: (Score:2)
Of course, I'm always battling with the tension between Agile and CI/CD and formal engineering processes for safety-critical systems...
I'd rather formal engineering process not be agile for safety-critical systems.
That means I also don't have to be as agile because I don't have to run away from safety-critical systems that have gone critical.
Re: (Score:2)
Of course, I'm always battling with the tension between Agile and CI/CD and formal engineering processes for safety-critical systems...
CI/CD is a good idea. You should do it.
But if you don't have manual testing, you're going to miss things. You need multi-layer defense against safety problems.
Re: (Score:2)
I'm responsible for integration testing with the components adjacent to mine, as well as my own unit tests, and for the smoke tests, and the CI/CD framework.
I don't write code, pass a unit test, then throw it over to the wall for QA. That'd just waste everyone's time.
Re: (Score:2)
You wouldn't normally make QA the gate keeper for a 3 line change. You put the change up for review and in parallel you'd run automated builds and some light-weight tests. (CI/CD) before integrating it. Later when you get closer to release, the software as a whole would go through a QA process. You might have code freeze where changes require a more formal process to introduce. And changes would generally be focusing on addressing bug reports from QA rather than new development.
A 3 line bug fix at my compan
Re: (Score:2)
You wouldn't normally make QA the gate keeper for a 3 line change. You put the change up for review and in parallel you'd run automated builds and some light-weight tests. (CI/CD) before integrating it
Sure but what the parent article is describing is well beyond what I would call "light-weight" testing and if it necessitates playing through in career a couple times as stated in the article then the game should be playable enough for a QA person.
A good bit ago now I spent 3 years working QA for a major game publisher and the article's description of what was happening very much made it sound like they had programmers doing QA work to me.
Re: (Score:2)
We're the QA team. :(
Re: (Score:2)
QA can be just as highly paid as devs, or more maybe depending on their seniority. But at every stage of development, from requirements to specs to design and all the way down to testing, the sooner you catch a bug the less it costs a company to fix it. So if a developer finds a bug before QA does, it's a big win. Developers should never throw something over the wall to testers to are the first team to touch it.
Kind of annoys be by working code bases where this doesn't happen. Ie, ran across an issue la
Re: (Score:2)
Absolutely. In my company, the programmers write the code and then they push it to the compilation team to compile. It used to be a bit time consuming until we wrote the automated script that creates a Jira ticket for each compilation error. Everybody's happy. Or at least I think they are: I haven't spoken to the programmers in a while... or even seen them about, come to think of it. Anyway, we know it's all working because we used to get thousands of tickets every day and it's now dropped to zero.
Seriously
You say it like it's something bad (Score:5, Insightful)
Re: (Score:2)
It means that they have poorly defined interfaces between the sections of their code.
Re: (Score:2)
Re:You say it like it's something bad (Score:4, Informative)
In the example given they wanted to change the way the soccer teams iterated their performance over several seasons. Which means that you would have to run through those seasons to make sure it worked correctly. It's not a question of if it would compile or if their were glaring bugs. Now, you can argue that there should have been a separate dev mode which would automate the career mode inputs so as not to need input from the programmer, but even then to run that long of a sim it would still take a bit of time.
Re: (Score:2)
it's a videogame. Effectively they're one-off products that once released, short of a few patches, is never touched again. The code is written to be disposed of because it often is - they release FIFA 2022? Ok, start on FIFA 2023, and you likely will start with new tools, a newer engine, etc, so you discard all the previous code and start from scratch.
Because of this game code is written more for "just make it work" than "m
Re: (Score:2)
Some video games are like that, some are not.
Re: (Score:2)
Re: (Score:3, Insightful)
What crack are you smoking? If you're lucky, that ideal will hold up for v0.1. After that - all bets are off as modifications are needed.
Also, what does "Modern" have to do with anything? I can write 'well-designed' FORTRAN, COBOL, PostScript, R, JavaScript, etc. 'Well-designed' is also very subjective. Was it designed to be fast? In many instances you're usually throwing away all the good design intentions to eek performance out of it. A nice well structured CASE - gone! - Put in a multi-level neste
Re: (Score:2)
Yea, 3 line change. changed "int" to "float" for three functions. That'd be .. fun. Good luck sorting that out in a day unless your 3 line change also included some beefier unit tests.
Re: (Score:3)
Every little change in code can have various hard to predict side effects
Indeed. I'm reminded of a collision detection change on bees in Skyrim. Turning bees into a craft item which can be picked up had the unintended consequence of yeeting the cart into deep space during the intro scene. You know when an unstoppable force hits an immovable object and something has to give.
Re: (Score:2)
No, this can be a fact of life especially when dealing with aging code bases or highly optimized code. In both cases, what appears to be a small change can have unforeseen consequences farther up the call stack.
Re: (Score:2)
While a small change "can" have unforeseen consequences you also "can" be hit by a meteor in the next hour. Bad code leads to more unforeseen than code that is well designed, if you have a code base that 3 lines of code needs 1 day to build and test then you need to fix your aging code base, or your highly optimized code. You are just digging yourself into more technical debt. Also you don't need to fully test the system for each change you make, which deals with the case of unforeseen consequences, that is
Re: (Score:2)
Every code base has some level of technical debt, and that is a fact of life because unless you don't have a release schedule & unlimited budget, debit builds over time. Code reviews are nice and all, but they have a very limited effectiveness especially as the project gets bigger and bigger or if the code base is highly specialized. If the 20+ years doing software development has taught me one thing, it is that you should never underestimate just how big of a error you can make assuming you know just h
Re: (Score:2)
Let's add some context:
Every little change in code
There is very likely going to be critical code that can have far reaching effects, but this should be the exception even in a poorly designed system. Things would need to get pretty bad to make the bulk of your code that fragile.
If any random change has the potential to cause unforeseen problems, the project has serious structural issues that should have been addressed much earlier. If every project is like that, changing jobs is a sensible decision. What a nightmare!
especially when dealing with aging code bases or highly optimized code.
I was hired as
so fast (Score:2)
the only question is, How are they doing it so fast?
a day? you haven't even written the story yet
News? (Score:5, Insightful)
However, this is also why startups have an advantage. Screw ups don't have near the consequences at a startup as at a large established company.
Re: (Score:2)
This actually might be the most bizarre and clueless 'story' I've ever seen on /.
Re: (Score:2)
This actually might be the most bizarre and clueless 'story' I've ever seen on /.
Really? Since your UID is roughly what mine is, you've been registered here for a while. Perhaps your memory's recency effect half-life needs to be bumped up a bit.
Re: (Score:2)
Re: (Score:2)
I choose recurring disappointment over the alternative of becoming jaded
Hey! That's my approach to political elections, too.
Re: (Score:2)
Whipper-snappers
Re: (Score:3)
Here, have some hot grits
Re:News? (Score:5, Insightful)
It turns out to have been a single misplaced comma in a completely different section of code, out of thousands of lines of code. The butterfly effect is alive and well in programming. Three lines in a day can be generous.
I am reminded of Gimli's quote from Lord of the Rings: "With cautious skill, tap by tap - a small chip of rock and no more, perhaps, in a whole anxious day - so we could work”
Re: (Score:3)
Re: (Score:2)
Reminds me of a (possibly fictional) quote attributed to Homer when he wrote the Iliad:
"Today was a good day. Today I wrote a full sentence."
So the story was written by someone without coding (Score:5, Insightful)
That article makes it seem like the author never work professionally doing any coding.
I have a reputation as a fast to deliver coder, however I have taken a while day to change a line of code, because that one line of code, can drastically change the program and affect outcomes down the line.
Changing code is a lot like those Science Fiction Time Travel stories, where someone just kills a butterfly in the past ends up destroying civilization in the updated present.
Even the best documented code, you will need to understand what it is doing and why in a bigger picture before you fix something. Often you can fix it rather quickly because you know the scope of the function, but other times it could be a bigger scope, where that result of the function calls different functions to call.
Re: (Score:2)
And don't even start on the huge amount of 'coders' that have zero mathematical background, so just write code they 'think' works instead of writing something that's actually mathematically efficient..
(Sort algorithms come to mind. Although those have been hammered out pretty solidly nowadays you still see people implementing their own because they think it's better (spoiler: It's not), without having any clue about the actual mathematics behind it.)
Re: (Score:2)
Actually, this is one of my problems with the way interviews are done all over the industry. The only answer to, 'how do you efficiently implement a balancing binary tree," should be "I find a stable and reliable library that already does it, rather than wasting my time writing it from scratch'.
Unless you're being hired specifically to create those libraries because you're working at Matlab or something, you should almost always be using built-in language or library functionality instead. Reinventing the wh
Re: (Score:2)
When I'm conducting the interview, I'm a big fan of candidates that give that answer. And then I make them implement one anyway.
Because what I'm really trying to find out is if they can write anything that makes sense, and try to get a glimpse into their problem solving. And asking something like "balance a tree" or "sort an array" is much faster than explaining a real problem.
Also, for any young'ins reading this: Talk about what you're doing and why during this kind of thing. Yeah, sorting an array is s
Re: (Score:2)
And then you have them shimmy up a tree and pick you a nice ripe banana.
Re: (Score:2)
Re: (Score:2)
(Sort algorithms come to mind. Although those have been hammered out pretty solidly nowadays you still see people implementing their own because they think it's better (spoiler: It's not), without having any clue about the actual mathematics behind it.)
One place you see this is with slightly disguised sorting operations... things that the programmer does not think of as sorts but is. The canonical example of this I have seen even experienced programmers implement (often due to code evolution) is a selection sort. They have a list and scan it to find something. Then they scan it again to find the next thing. Then they scan it again...
The initial implementation only scanned the list once for one item, and so was O(N), not too bad (though O(1) is available).
Re: (Score:2)
And don't even start on the huge amount of 'coders' that have zero mathematical background, so just write code they 'think' works instead of writing something that's actually mathematically efficient..
(Sort algorithms come to mind. Although those have been hammered out pretty solidly nowadays you still see people implementing their own because they think it's better (spoiler: It's not), without having any clue about the actual mathematics behind it.)
It depends a lot on the project but in my experience efficient code doesn't come from a background in math, it comes from a background in algorithms. Big O notation [wikipedia.org] is generally all the math you need, and then it's largely just understanding what the code is doing.
But for sure, clever code is slow code more often than not. Leveraging a library that people have been optimizing for years is usually the way to go.
Re: (Score:2)
> Sort algorithms come to mind. Although those have been hammered out pretty solidly nowadays you still see people implementing their own
Like the time when the junior programmer in our team implemented a bubble-sort over a linked list. Hello, O(n^3) complexity.
Re: (Score:2)
Or: not all code changes are equal.
That's fast for when money is on the line (Score:5, Informative)
That article makes it seem like the author never work professionally doing any coding. I have a reputation as a fast to deliver coder, however I have taken a while day to change a line of code, because that one line of code, can drastically change the program and affect outcomes down the line.
Changing code is a lot like those Science Fiction Time Travel stories, where someone just kills a butterfly in the past ends up destroying civilization in the updated present.
Even the best documented code, you will need to understand what it is doing and why in a bigger picture before you fix something. Often you can fix it rather quickly because you know the scope of the function, but other times it could be a bigger scope, where that result of the function calls different functions to call.
Couldn't agree more. I think one thing to add is that once you have users and a product makes money, the most trivial of changes becomes expensive in terms of complexity and person hours.
...sure, I can write code as fast as I can type and upload to an AWS instance. However, I don't make money off it, so it doesn't matter. Once you make money, things like legacy support and backwards compatibility are huge.
For my personal dicking around project?
Here's a classic example. I worked at a place where the product had a codename and a month before launching, they settled on the final market name. We found on reference to the code name in the on hover help text. A new employee checked out the latest version of the code, made the trivial text change and broke a major form. Even though it was just 1 word in a display, the entire page broke because he checked out the latest version...and it had a bunch of new code in it. Every code review showed a trivial 1 word change. The change was live for 10 minutes. We got a ton of calls from angry customers for the downtime.
For my current job, I've done simple 1 line changes and it's literally taken months to hit customer servers.
We have billions of dollars on the line and there are a lot of precautions.
While the process I follow is not without opportunities for improvement, you can never get a 3 line change out 1 day when money is on the line. It has to be audited for security violations. It has to be reviewed for correctness. Regression tests need to be performed to ensure there are no downstream impacts. It takes nearly a day for a machine to run those regression tests. Even when we have mission-critical emergencies, a 5 minute fix takes hours to propagate to all our servers across all zones and be verified.
The author definitely has never had experience working for a profitable company.
At Nintendo... (Score:4, Insightful)
One of their cartridge games for the N64 (no online bug updates) was being finalized (crunch time), and the special ending for collecting all stars (or whatever) didn't pop. The lead programmer made a change, went to sleep while the playtester started a marathon game on the new code for all the stars. 14 hours later, the bonus ending still didn't pop. The lead programmer says, "Oops," makes another change and says "try that." The playtester has to do it all over.
Re: (Score:2)
Seems odd if game developers don't have special versions with editable save games so you can make a quick test to see if something works, then a full run on a 'real' version to see if it works there as well.
Agile doesn't mean what they think it means.. (Score:2)
A change kicking off a whole day of continuous integration before it can merge is not surprising, though it can be frustrating and maybe relatively rare. In this case it may be more manual but the principle applies, agile doesn't mean 'merge the code fast, no matter what'. If your process is that once you've made your change you can't work on other changes while the code is being tested, well then you have a problem of some sort.
Unit testing can *help* and failing fast on unit tests can help iterate to a
Misleading (Score:5, Insightful)
This is not "a day to fight through bureaucracy to change three lines of code." This is "a day to change three lines, and properly evaluate the consequences of that change," which for something as core to their gameplay loop as the competition management, isn't really that surprising.
Agile? (Score:2)
While originally intended as a process for manglement to get extra work out of coders for free, Agile quickly turned into a way for programmers to avoid doing any work, sprint after sprint.
After sprint.
Re: (Score:2)
While originally intended as a process for manglement to get extra work out of coders for free, Agile quickly turned into a way for programmers to avoid doing any work, sprint after sprint.
After sprint.
Sprint doesn't exists anymore. You mean, ... T-Mobile after T-Mobile. After T-Mobile. :-)
Re:Agile? (Score:5, Insightful)
That's the thing. Agile (as opposed to agile) is yet another attempt to observe a highly skilled and productive group of programmers, formalize what they do and try to get a pack of chimps to become highly productive coding by numbers. Sadly though, they miss so many corner cases and how to determine when those are present, that even the original team becomes a pack of chimps if they are forced to adhere to the Agile way.
Re: (Score:2)
How so?
If programmers would not work, the project would never finish, or never had "sprint results" in production.
Agile means "let the user test." (Score:2)
This is actually good software discipline. Agile was designed for things that are easy to deploy (ie: web stuff), so the cost of fixing something is low. When the fix cost is high, testing is important.
The joys of programming (Score:3)
three lines (Score:5, Insightful)
Thus spake the master programmer:
"Though a program be but three lines long, someday it will have to
be maintained."
-- Geoffrey James, "The Tao of Programming"
Re: (Score:2)
It's amazing how true that has been in my career.
Re: (Score:2)
it's three lines of code
it shouldn't need maintenance
and yet here we are
Confusing Agile with CI/CD (Score:2)
How about 2 weeks ... (Score:2)
Way back in 1998, I worked somewhere that (initially) outsourced its HP sysadmin to another company (whose initials come before HAL). It took 2 weeks to get them to add an entry to the /etc/hosts file (seriously, literally) as all changes had to go through their "review board". After they accidentally deleted the production database -- twice -- (so much for the review board) our company cancelled the contract and brought the systems in-house for me and another person to manage. It helped that our manage
Author thinks Agile is everywhere (Score:2)
I'm guessing the author hasn't done a lot of coding for pay.
There are places Agile makes sense, but it's not everywhere - and shouldn't be. But the term does seem to mainly float around in the manager-sphere.
Re: (Score:2)
But the term does seem to mainly float around in the manager-sphere.
And often when it floats around there, it has no meaning.
I Have A Friend... (Score:2)
Re: (Score:2)
They spent 6 months testing a single line of code and when inserted, it killed service in three stateThey spent 6 months testing a single line of code and when inserted, it killed service in three states.
That's just a sign that AT&Ts software QA process is broken. A bunch of people who write and rubber stamp piles of documents without reading or understanding the actual contents. They are just going through the motions. But it sure does keep a lot of people employed.
Cargo cult QA.
This isn't ... (Score:2)
Agile isn't winning in reliability of code .... (Score:2)
I'm not a developer, so take my comments as you will. But with a long career in I.T. plus making it one of my primary hobby interests for decades, I've definitely noticed a trend towards buggier, less reliable code, since the concepts caught on for things like "Agile programming".
Seems to me, it's all about putting a priority on getting new code out the door rapidly, instead of ensuring things are done properly first. "Let the millions of users find the bugs for us when they use it!" is the new mantra fro
(Un)agile (Score:2)
We're working "agile" as well...
We are a cloud provider though. Never in my life have I been so inefficient and unmotivated at work.
It can take a day to get a code review (Score:2)
Anyplace with actual quality gates, things take time. I can make a 1 character change, I still need to have that change code reviewed, pass autoamted tests, and then hand it off to QA. The first part of that could take a day to get someone's time if the team is busy. Processes just take time, and the size of the change is not always the determining factor in how long it takes- in fact the smaller it is the more likely something else is to be the bottleneck.
Only one day? (Score:2)
A whole lot of orgs are very slow to make changes. One day to make a change isn't strange at all.
Agile is a generic bunch of idealistic phrases by a dozen dudes at a whiteboard. Agile does not provide any practical solutions to common problems. It's just bullshit that people throw around to justify not using rigor or due diligence, or to avoid writing documentation, or having a change control process.
And this article is a joke in 3 words (Score:2)
bad heading (Score:2)
I'm not programmer, but we are talking about an EA (Score:2)
sports title right? So, my guess is those 3 lines of code are to change the date to the current year and pull a roster of new player names so they sell the same garbage for $80+ again.
Extremely out of context (Score:4, Informative)
Did anyone actually read the original article? Not the linked neowin post, but the devtails one neowin links to? It's a post explaining how EA managed to get rid of extremely long iteration times. I don't have much love to spare for EA, but this post is just ridiculous.
Has nothing to do with agil (Score:2)
As other posters have pointed out: If those are critical lines of code, where a change requires a lot of testing, then a day is fine.
I want to make an additional point: This has nothing at all to do with whether the team is "agile" or not. Agile means working iteratively, in short cycles. Changing those three lines of code in one day could well be part of an agile cycle.
If the guy is actually an experiences developer, then he is clearly used to working on small or new projects. Maintenance on big, estab
In an game changing an percentage can take an lot (Score:2)
In an game changing an percentage can take an lot of play to see if it works / if the game play is good with that change.
And in an sports game that may be you need to play an full season to test it
And still it doesn't move (Score:2)
In the 1970s, there was a book called "Why Nothing Works". Their poster child was Xerox, which, at one point, needed the sign off of something like 50 managers to make a small change to a copier machine.
This was a vastly different company from the one that took the stock market by storm a few decades earlier.
Re: (Score:2)
Thus spake the Master Programmer:
"Let the programmers be many and the managers few -- then all will be productive."
THis sort of thing is not new... (Score:2)
Re: (Score:2)
The secret is that routine is actually written in Whitespace [wikipedia.org].