Can ISO 29119 Software Testing "Standard" Really Be a Standard? 152
New submitter yorgo writes The International Organization for Standardization (ISO) will soon publish part 4 of a 5 part series of software testing standards. According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation." However, many in the testing community are against it. Some wonder how the ISO/IEC/IEEE achieved consensus without their input. James Bach speculates that exclusion helped build consensus. Others, such as Iain McCowatt, argue that something as variable as software testing cannot be standardized, at all. And others believe that the motive behind the standards is not increased quality, but economic benefit, instead. Michael Bolton explains "rent-seeking" as he builds on James Christie's CAST 2014 presentation, "Standards – promoting quality or restricting competition?"
A comprehensive list of many other arguments, viewpoints, and information has been collected by Huib Schoots. Opponents of ISO 29119 have even started a petition aimed at suspending publication of the standard. Even so, this might be an losing battle. Gil Zilberfeld thinks that companies will take the path of least resistance and accept ISO 29119.
So, where do you stand? What constitutes a consensus? Can a standard be honored without consensus? Can an inherently sapient activity, such as testing, be standardized, at all? What is the real purpose of a standard? Will companies acquiesce and adopt the standard without question?
A comprehensive list of many other arguments, viewpoints, and information has been collected by Huib Schoots. Opponents of ISO 29119 have even started a petition aimed at suspending publication of the standard. Even so, this might be an losing battle. Gil Zilberfeld thinks that companies will take the path of least resistance and accept ISO 29119.
So, where do you stand? What constitutes a consensus? Can a standard be honored without consensus? Can an inherently sapient activity, such as testing, be standardized, at all? What is the real purpose of a standard? Will companies acquiesce and adopt the standard without question?
Who cares (Score:1)
Considering ISO certification costs money and most companies refuse to spend money on stuff that isn't ISO 9001 then very few will probably get it. Most dedicated software houses I've encountered seem to prefer getting the TickIT certs instead of ISO certs for actual code certification but nobody bothers with the testing side of things.
Re: (Score:3)
Re: (Score:2)
"Who cares" has to be the most useless argument on this page.
The businesses with the most money will obviously attempt to get certification for a bullet point. The people who make decisions based on bullet points care. The people at those companies who have to implement those bullet points care.
More importantly, the customers who have to spend more money care. If it is spread across enough customers that the financial impact is negligible, the customers don't care and the business doesn't care. It all w
Standards (Score:3, Insightful)
Re: (Score:2)
At least where I work, most likely is that it will be more paper work to get done - never mind that we already have too much paper work to do. Like us in development, the testing people will make whatever they can conform to this new standard, then file waivers for the rest.
Re: (Score:2)
Are rules for some and suggestions for the rest of us. The IEEE can put a standard on cleaning the toilet.
At least where I work, most likely is that it will be more paper work to get done
Well it is normal practice to not consider cleaning the toilet until the occupant has finished their paperwork.
Re:Standards (Score:5, Insightful)
It's not quite that simple - customers can require the certification. If a customer is sizable enough, the suppliers will have comply to stay in business.
For real world examples, look at the power that the state of Texas has wielded in text book requirements and the state of California with warning labels.
Re: (Score:2)
Re: (Score:1)
Can see it now: (Score:5, Insightful)
MBA CEO: I want our new product to be QA'd according to ISO 29119 before shipping.
Project Manager: Good idea, but that will add some time and overhead cost to my budget.
MBA CEO: Never mind, just ship it.
Re:Can see it now: (Score:5, Informative)
MBA CEO: Never mind, just ship it.
More likely response: "Figure out how to get it done within the existing budget and schedule."
Re:Can see it now: (Score:4, Funny)
Re: (Score:2)
And if anything goes wrong, we'll crucify the QA people that we never gave enough testing time to in the first place.
Re: (Score:2)
You still have QA people? What luxury!
Re: (Score:2)
Re: (Score:2)
More than more likely response: Figure out how to get us compliant so our sales guys can use that ASAP.
Also, don't delay our shipment because those schmucks paid for non-compliant widgets. Delays for no revenue or no value add in the next rounds of sales hurt my bottom line.
It's like you ascribe the next-to-worst attributes to a CEO but don't go all the way and see it realistically the way a CEO would.
Which Michael Bolton? (Score:4, Funny)
Michael Bolton explains "rent-seeking"
The no-talent ass clown known for his God-awful "music" or the one who hates him??
Re: (Score:2)
You jumped my stuff.
Re: (Score:2)
Re: (Score:2)
But but ... how are we supposed to live without you, now that we've been loving you so long?
Dude, seriously, it must be suck to evoke such a reviled musician every time someone hears your name.
Re: (Score:1)
Re: (Score:2)
LOL ... well, tell me how you feel [azlyrics.com], time is on my side, [azlyrics.com] it's just a feeling [azlyrics.com], but I almost believed you [azlyrics.com].
This is the internet, everybody's crazy [azlyrics.com], don't like it, walk away [azlyrics.com].
From now on [azlyrics.com] you'll know that forever isn't long enough [azlyrics.com] until you hear another Michael Bolton joke.
Just remember to love somebody [azlyrics.com]. The one thing [azlyrics.com] to remember, this is the time [azlyrics.com] you're asking yourself Why me? [azlyrics.com]
So, before you turn a whiter shade of pale [azlyrics.com], I wanna hear you say it [azlyrics.com] ... If I could [azlyrics.com] go the distance [azlyrics.com], That's life! [azlyrics.com]
All the best" [azlyrics.com].
There's two
Re: (Score:2)
As this particular thread is off-topic anyway: Thanks for your blog, Mr. Bolton! It's always an interesting read and I try to point new testers to it at least once to get some exposure to a usually unknown view on testing.
Wrong focus? (Score:3)
... According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation."...
If I were to jump upon a standard for testing software, the fact that it is "internationally agreed" is way down on the requirements, yet it seems to be mentioned as the main feature here.
Re: (Score:1)
Re: (Score:3)
This shit matters if you are selling things internationally. It makes it difficulty for country A to refuse imports of county B's software because it wasn't tested to their local testing standard. If it was tested to an ISO standard it'll do and the WTO will be on their ass if they try to claim otherwise.
That doesn't mean the specs are any good. They aren't. But there is a very real interaction with international trade regulations.
Re:Wrong focus? (Score:4, Insightful)
You are incorrect. ISO standards are recognized by multiple countries. You can Google the full list.
The main advantage of any kind ISO is that it enables your software to get on government procurement lists. This doesn't impact small shops, they don't generally have resources or organizational maturity to sell to governments. There are multiple international, proprietary, and country-specific standards (e.g. ISO, FIPS, Common Criteria, PCI-DSS). International means you only have to go through certification once, not once per country that you were doing before the standard was adopted.
Re: (Score:1)
Re: (Score:2)
Compliance establishes baseline level of competence. As a system integrator, I have seen numerous examples that fall under gross negligence category. As a result, I strongly believe that even ineffective standards testing with associated increased overhead is better than a status quo.
Re: (Score:2)
Re: (Score:2)
If I were to jump upon a standard for testing software, the fact that it is "internationally agreed" is way down on the requirements, yet it seems to be mentioned as the main feature here.
I think it may be incidental as a result of the standard being published by ISO/IEC/IEEE. I have seen very few marketing talking about the the international standards. The only time internationalisation is mentioned at all is when the local standards disagree with what the rest of the world is doing and cause grief with the procurement / installation of equipment.
There is only really three questions that should be applied to any standards:
1. Do we have a legal obligation to follow it.
2. Did our clients requ
Shades of 2167 (Score:4, Insightful)
This sounds like the 21st century version of 2167.
Re: (Score:2)
But, I've always assumed that the function of government specs was to achieve precisely that.
I mean, a quality standard defined by a government committee can't actually be expected to have been designed to product actual quality outputs, can it?
I've just assumed they were there to give the bureaucrats something to tick off on their checklist, even if it has nothin
Re:Shades of 2167 (Score:4, Interesting)
In the late 80s and early 90s I was involved in 2 projects run under MIL SPEC 2167, which was supposed to ensure product quality. Both were epic disasters. IMHO, 2167 pretty much guaranteed mediocre at best software, taking 3x longer to do, at a cost at least 6x of non-2167 This sounds like the 21st century version of 2167.
MIL SPEC 2167, iirc, deals with documentation and deliverables. The actual software development process was "guidelined" by some other MIL SPEC. With that said, those were supposed to act as guidelines for a waterfall methodology (which surprisingly, it can actually be used in some contexts, or subverted into a spiral/iterative process.)
I worked at a defense contractor not long ago, and alas, the software projects were horribly run. But I always saw that it was the execution, not the specs per say that was the culprit for each and every single snafu/fubar hybrid I encountered. That and management, and life-long-career jockeys from the punch-card era dictating how to write software, and department infighting.
It's just like CMM/CMMI - A CMMI level X or Y only indicates that, to some degree, an organization a) has a formal process, and b) follows such process.
It doesn't indicate that the process is good - it doesn't even guarantee that *it is not shit*.
What it does, is that it helps an organization guarantee that its constituent parts know what activities to do under what circumstances and tasks in a business lifecycle. And that helps an organization improve things (assuming said organization has the necessary technical wherewithal and culture.)
In private business and with defense contractors, it is the companies who fail to execute (let's think of how many companies ditch source control to become *agile*!) particular practices. Defense contractors have a lot of leeway in negotiating and tailoring the actual execution of a process. Typically, they do not do it because they suck (and for a lot of other political and financial motivation$.)
Re: (Score:2)
Per se.
Do try not to write things you've only heard spoken.
Re: (Score:2)
Per se.
As a person whose first language is not English, I thank you for your gratuitous lecture.
Do try not to write things you've only heard spoken.
Oh uhmmm, m'kay?
Re: (Score:2)
"What it does, is that it helps an organization guarantee that its constituent parts know what activities to do under what circumstances and tasks in a business lifecycle." That claim is unsupportable, since it ignores the role of tacit knowledge; it ignores all the things that the process manual doesn't say; and it ignores all the ways in which the process manual overstructures the work.
That is why I said this (re-quoting myself in bold/underline below):
"What it does, is that it helps
I guess the following I'm going to say was supposed to be implicit, I will have to be explicit. Only in slashdot.
Of course it ignores tacit knowledge and all the things that the process manual doesn't say. It has to, because attempting to do is impossible in the general case. A process is not supposed to be all encompassing and inclusive. It is supposed to operate at a meta level, to provide structure around activities. It is supposed to be more strategic than tactical. Very few things in a process would cut down to the bare metal of day-to-day operations.
When a process, formal or informal becomes more tactical than strategic, that is when it falls into the realm of micromanaging. Once you have micromanaging in place, then you cannot use such a process to improve things because of all the other dysfunctional social/political forces that cause the process to become micromanaging in the first place.
Again, a capability model is not concerned if your process is geared towards continuous improvement or micromanaging. It is only concerned with the degree in which a) you have one, and b) that you follow it.
If you have a process that is functional, and your organization is functional, and that it follows said process, IT WILL HELP said organization with its improvements.
I should not have to spell that out. It should be self-evident. It should also be self-evident that if any of these pre-conditions are met, then all bets are off (and that such a situation is not a capability model's concern.)
As James C. Scott points out in /Seeing Like a State/, this is exactly the reason why work-to-rule campaigns can be as effective (that is, more disruptive) than a strike.
Yes, you are describing a dysfunctional situation, a combination of a dysfunctional organization, system or set of players and a a dysfunctional process open to abuse. None of that is a capability model's concern, nor are inevitable consequences of having a process.
Automated test in is a minimum (Score:3, Interesting)
Unit testing is step one in any decent software development and I will never enter into or manage another project without unit tests being a critical component of the project. I'll just hire a QA guy to unit test all my code... I don't want to do it haha.
Second, there is absolutely nothing which can't be automatically tested too. When you write code, GUI, Web, Command Line, message based, etc... An automated script to test the code is critical. There are tools for it.
Everything should be tested automatically... That even includes memory leaks when exiting. I would never hire someone even for a C#, Java, Python or even PHP position who doesn't write code which cleans up properly after itself (even if that means correct use of the garbage collector).
I have worked on several multi-million line commercial applications, some with 500 million+ users. I have never seen a piece of code which could not be properly tested using the right tools. That can even include small embedded systems where we would have to actually implement a QEMU module or three.
So... Quit your bitching and write a test suite.
Re:Automated test in is a minimum (Score:5, Interesting)
You would love the control system software we use at work... (world leading platform for the industry).
No revision control. You have 'current' revision. That is it.
Integrated code editor that has no syntax highlighting.
Patches to the system will break components that are not considered 'core'. Which forces updates of ALL components in the system. This has lead to bugs persisting at sites for years with no patch because nobody wants to fix bugs when it costs tens of millions of dollars to do so.
No automatic testing. Of anything. When we update a component everything has to be tested manually. Someone will sit for 2 weeks and check every state of GUI symbols for the whole HMI. Oh joy...
If you change ANYTHING in code, you can no longer connect to controllers to view live data. You need to do a download to the control with the code changes before live data can be observed. This means that as soon as you make changes, you lose the ability to view the old code running. There is no way to have both a 'online capable' version of the code and a changed codebase in the same control system. We operate two separate virtual environments and switch VLANs or just move a cat6 when testing...
This is for oil rig control systems. There is no automated testing of any kind, even for critical emergency shutdown systems. Every test is done manually.
The ESD systems are usually a complex matrix of casues and effects with hundreds of inputs, hundreds of outputs... This is all tested manually as the software does not support any reasonable simulation of the controller input/output systems.
Enjoy that little gem.
Re: (Score:3)
Sounds like you have a poor choice of control system.
Our control system is the exact opposite:
Control graphics run locally on the workstations which means they can be tested away from the control room with live equipment and then push the completed graphics to other stations.
Loop simulation can be performed online for changes to control system parameters. Feed it a bit of historical data so it builds a model, then update the parameters and simulate, then push the result back out to the control processor.
Com
Re: (Score:2)
Which control system is used depends on a lot of things, and it may not be reasonable to pick one based on ease of development.
Re: (Score:2)
Very true, my point was more that it is a moving trend in the world of control. The vendors are actually hearing our calls for a more developer and more maintenance friendly system. Even our emergency shutdown system has gone from offline logic changes only, to limited online logic changes, to online firmware updates, to completely online full system upgrades.
The industry on the whole is changing. The problem is for old plants these changes will take MANY years to filter through.
Re: (Score:2)
I'll just hire a QA guy to unit test all my code...
Actually, that's how the testing should be done. Give the requirements to both teams - testers and developers. Developers design/write the product code. Testers design/write the tests. Then let the testing begin. Problems entered into issue tracking. Both teams fix their respective problems. Retest. Repeat as needed.
Unfortunately, many companies fail to adequately fund testing so devs end up writing tests, which, in turn, catch fewer problems
Re: (Score:3)
That's not going to work, because you'll never be able to economically write a requirements document so complete that the behavior is so well defined that you can get meaningful test coverage from it.
To get that kind of completeness you'd have to code the entire software in MSWord, which is a terrible programming language, and without ever testing it along the way.
Testing needs to be a continuous process as part of software development, not something that happens parallel or afterwards.
Re: (Score:2, Insightful)
Unit testing is white box testing and it's okay to get a tester to develop the tests as long as there's feedback from the developer about how the white box code is supposed to behave (internal behavior).
Does the average dev have the skill set/ mind set of a tester? No. Even Microsof
Re: (Score:3)
FTFY.
If a developer doesn't have the mindset of a tester, perhaps they should find something else to do (maybe the local Walmart needs a greeter?) or be assigned support "on-call" more regularly so they can experience being waken at 3AM due to someone failing to design, implement, and test well. If someone doesn't have the mindset of a tester and their title is something like Senior Software Developer or higher, there is a serious ti
Re: (Score:2)
Qualified senior developers I know do some amount of all three and all have done a substantial amount of all three during various points in their career.
If you're considering just a "coder" (working, presumably, from some detailed design spec), that's probably a different matter (I've never worked anywhere that a "coder" job role existed), but a "dev" (a.k.a. "developer") is a much broader role than "coder" (and pays much better - so, yes, I probably pay them three times what I would pay a "coder" if I coul
Re: (Score:2)
Unit testing legitimately comes under both job descriptions, though
1) the coder really understands the meaning of 'unit' in a particular context, and
2) the tester is the subject matter expert on testing.
The two need to have a dialogue. If they aren't talking, there are bigger problems.
Re: (Score:2)
In a static typed language unit tests are pretty pointless. You only have them because your developers are using cut/paste to much and accidentally hit return when a line was selected (and delete it by that).
You are far better off if you write user acceptance tests on use case or story level.
I have worked on several multi-million line commercial applications, some with 500 million+ users. I have never seen a piece of code which could not be properly tested using the right tools. That can even include small
Re:Automated test in is a minimum (Score:4, Interesting)
In a static typed language unit tests are pretty pointless.
Because static typing catches all bugs? That must be some statically-typed language that I've never seen. Unit tests are perhaps marginally less necessary than in dynamically-typed languages, but they're still necessary. Test-Driven Development is a life saver regardless of your toolset.
I can write you in 40 lines a function which you wont be able to automatically test. It only needs some nested loops and an 'if' cascades with loops inside.
There's nothing untestable about such a function. Basic code coverage tools will even identify any branches within the code that aren't taken, so you know to look for ways to write test cases that cover those branches. What's harder is ensuring coverage of all of the issues that could be provoked by different input data, even after you've covered all of the paths. With experience you learn to do that quite effectively, though.
Sure: you should refactor that into a lot of small functions, containing only a single loop or a single 'if'.
FTFY. Change it to "must" if I'm your code reviewer.
You lost quite some credibility with using terms or sentences like That even includes memory leaks when exiting. and "... even if that means correct use of the garbage collector ..."
Unfortunately a exiting program can not leak memory
Sure it can. If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak. Again, there are good tools to help you find these leaks, like valgrind memcheck.
you don't use a garbage collector, it runs in the background. You can parametrize it perhaps ... but thats it. (and please don't tell me you are doing System.gc() in Java programs at "random" intervals)
And yet you can still have leaks in garbage-collected environments, and there are ways to test for them. It's a bit more complex than in non-GC'd environments, but it can -- and must! -- be tested if you want to have reliable software.
Re: (Score:2)
Rofl.
Acacademic half knowledge with complete wrongs: If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak
No it has not. Where should the memory go to after the program has exited?
Technically you can test a 40 lines mess of loops and if cascades, practicaly you can't ... or how likely is it that you can prove me in a reasonable time that a certain branch in an if - else inside of a cascade of nested loops and if's is executed with meaningfull data in your test?
Re: (Score:3)
He's saying you can detect runtime memory leaks at exit time.
The memory won't be leaked anymore post-exit, and the leak doesn't matter at exit time, but that's when it is possible to deterministically detect memory leaks over the course of the software running.
Re: (Score:2)
Nevertheless he is wrong, even if your interpretation of his words is right. :) how should that work?
In C/C++ like languages no one is going to free/delete any memory at the point of exit(), you simply call exit() and are done with it.
but that's when it is possible to deterministically detect memory leaks over the course of the software running.
No it is not
Same in garbage collected languages. It is close to impossible to free all memory in a garbage collected environment ... you programmed the whole applica
Re: (Score:3)
No it is not :) how should that work?
The memory allocator can keep track of all the memory that is allocated (maybe you do this with a special build, maybe your default allocator does this).
Provided your heap isn't actually corrupt, (which tooling can also help detect with things like canary words and on special builds with extreme one-page-per-allocation policies, but is not totally detectable), your allocator can have a record of every allocation that does not have a corresponding free by the time of exit, no matter whether it's because of r
Re:Automated test in is a minimum (Score:5, Interesting)
Rofl. Acacademic half knowledge with complete wrongs
Dude. I've been a professional software developer for 25 years, and am a respected engineer at one of the biggest names in software. You use my software daily.
If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak No it has not. Where should the memory go to after the program has exited?
Well, obviously it'll all be returned to the system after exit. The point is to check AT EXIT. If there are any blocks still allocated, you've screwed up.
Technically you can test a 40 lines mess of loops and if cascades, practicaly you can't ... or how likely is it that you can prove me in a reasonable time that a certain branch in an if - else inside of a cascade of nested loops and if's is executed with meaningfull data in your test? Especially if I have written the function and you want to write the test?
I've done it many times. Just check to see which branches aren't being executed and work out what needs to be done to execute them.
Though it's much, much better to refactor the function first.
The rest I leave as it stands if you like to argue about how likely it is that a single compilation unit has a bug that is not dicovered by a functional user acceptance test ... unit tests without user acceptance tests or integration tests are pointless.
I've found thousands of bugs with unit tests that were not discovered by functional tests, integration tests, or user acceptance tests. In fact, unit tests are the most likely ones to find thing like subtle boundary condition errors which are hard to reproduce and are the source of nasty, rare bugs that are seen in production only once in a while.
The next thing is you tell me to test getters and setters ...
Typically those get exercised by integration tests... and it is a good idea to have coverage of them at some point in your automated suite, because the whole point of getters and setters is that they may someday become non-trivial. Writing tests for them at that point isn't as good as already having tests in place, because you want to verify that your new implementation doesn't have subtle differences in behavior.
But you will figure that soon enough when you have 100% code coverage and still have bugs and wonder why
No one is claiming that unit tests are sufficient, only that they're necessary.
Btw, I never was in a team that had a memory leak in a GCed language.
Then you haven't worked on many non-trivial systems.
Re: (Score:2)
No one is calling free/delete before exit() on all still living objects to show that he has no memory leak.
a) Yes, there are people who absolutely are doing that, particularly if they have nontrivial destructors (whether or not that's a good idea on a heap-allocated object is a different discussion).
b) You don't have to call free/delete/whatever before exit to show no memory leak. You just have to find that there's no *unexpected* allocations. A leak can reasonably be defined as memory that is unexpectedly still allocated at exit-time.
Just an excercise 'I know where all my pointers are'?
Yes. Because if you don't know where your pointers are, that means you le
Re: (Score:3)
Re: (Score:2)
The point was more that I tried to educate you.
But with your academic position you are on the safe site, a little bit at least. However you waste resources. I assume you have still many bugs in your code which you can not 'find' with unit tests. You can only write a unit test later after you have fixed the bug to ensure it does not creep in again. :)
Your stance is illusioning yourself, bad that you obviously did not try to follow my arguments, otherwise you would agree. But perhaps you do so in 10 years
You
Re: (Score:2)
The next thing is you tell me to test getters and setters ...
You damn well better test the getters and setters. In my experience they are usually the buggiest part of a class. To save time, sooner or later you will cut-and-paste the previous getter/setter pair and modify the name ... while forgetting to update the variable name behind the API, leaving it side-effecting something else. Now you have a landmine waiting to strike on the rare occasion you set that field. And woe betide you if the setter performs verification on the value, which you will probably get
Re: (Score:2)
It is much quicker to let the IDE generate the getters and setters.
I don't use cut/paste at all when coding. Autocompletion in Idea Intelliy or Eclipse is simply faster and less error prone.
Testing setters only make sense when you know that you have to test them, like when nulls are not allowed or the setter actually has side effects.
Most bugs regarding setters and getters are caught by compiler warnings, like passing a long in to a setter taking an int.
If your coworkers abuse copy/paste and make such error
Re: (Score:2)
Sure it can. If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak. Again, there are good tools to help you find these leaks, like valgrind memcheck.
Really can you point to any contemporary operating system that would NOT free all the memory allocated to a process when it exists? I guess you might mean if your process asks other "servers" to do things like say just exists without closing database connections etc, the other process might not free resources associated with yours but that is not the same thing as a memory leak.
Re: (Score:2)
Really can you point to any contemporary operating system that would NOT free all the memory allocated to a process when it exists?
Obviously not. If there were such an OS it would be broken.
The point is that if you're managing your memory well, by the time you exit every allocated block will have been freed by your code. Yes, this is a little more work than appears to be absolutely necessary, but when that function which allocated a block which you expect to be released at exit, and which you know is only going to be called once, ends up being called in a loop a few years later, the maintainer who makes that change will hate you. And
Re: (Score:2)
First off, I would love it if people took unit testing more seriously and automated it! In my view that helps greatly for getting robust software. However, not for all tests automating is the answer. Especially when you get to acceptance testing (where we validate, rather than verify) or when you do integration testing for systems that communicate with other systems it is not a silver bullet. Aside from feasibility, as automating is time consuming, there are more drawbacks. Automating all tests means assumi
Re: (Score:2)
The automated tests can and will miss things that are plain obvious to human testers.
True dat. The solution is that for every bugfix submitted there is also an automated test to verify it stays fixed.
Automated test suites are not static. They should grow as the project matures and users/developers gain experience with it.
Re: (Score:1)
I agree almost 100%, let's say 95%. Automated testing is fantastic. It's not everything though. You can't automatically test for "does it annoy and/or confuse the operator". For that, you need a human to test it. Also I think there are going to be aspects of the test that could be automated and you just didn't think of them. Acceptable performance falls under this category somewhat. An automated test won't tell you that something takes too long to render unless you know what "too long" is for a human
Re: (Score:3)
Asserts are fairly useless as a testing tool. Their excessive use can be extremely annoying as they clutter up the code and that clutter can increase the chances of introducing bugs.
However, responsible use of asserts can be quite useful in debugging - esp. when a customer is encountering a problem that is difficult to recreate (or, perhaps, nearly impossible from a practical standpoint because they won't give you access to their confidential data/environment). In those cases, running a "well asserted" debu
Re: (Score:1)
but how many tests? (Score:2)
Whilst unit testing is a good thing in general it's actually hard to know how many tests to write. Whilst it is in theory possible to write tests for most things, it's completely infeasible to a priori write tests every for possible interaction with any reasonably sized bit of software. In practice everyone ships code with gaps in the test suite.
There are contexts in which we should be formally proving the correctness of code. This tends to be a niche thing tho' because it's hard and time-consuming.
Re: (Score:3)
I love testing. Honestly, if forced at gunpoint to give up testing or version control, I'd be hard pressed to pick. Testing means I can update an innocent-looking line of code without worrying whether it's going to break some arcane business logic that a customer depends on, and that I can gut and reimplement an API while having a reasonable expectation that consumers will keep working afterward. I'm a huge testing advocate.
But.
I have no desire whatsoever to conform to an ISO document about their idea of th
Re: (Score:2)
I work on code that uses a MFC GUI and spits out gcode for CNC machine tools. While there's a lot in it that can be unit tested, and we have a lot of unit tests, I'd like to see a testing structure that would automatically test everything from user manipulation to gcode production. (Seriously, I'd like to see it. I've asked elsewhere to see if there's such a thing. If somebody could find such software, I'd heartily recommend it here.)
This sounds... familiar.... (Score:4, Funny)
SSL, anyone? (Score:2)
With SSL, we all wanted the security; but everyone wanted it to be cheap, and wanted to avoid a monopoly over certificate authorityship. So, what did we get? A mass of CAs, many painfully shoddy, who w
just like ISO 9000, that worked well! (Score:2)
Re: (Score:2)
In RL quality saves money.
Unfortunately many small (and also big shops) have not realized this yet.
Re: (Score:2)
You are very right of course.
But that does not change my point: if you can run your business in a way that you produce quality without having enormous overhead, then you safe money bottom line.
If such a habit is established in your organization, also new projects get to marked speedy.
The problem with introducing processes and QA etc. is that the introduction is to complex, slow and expensive that it costs you dearly and not only money but time. That is why 'agile methods' are en vogue. They try to get peopl
Re: (Score:2)
ISO9000 is a really sad joke. If you have a shitty process that produces poor quality, you can pass ISO9000 just fine. From that point on, it will make sure you never accidentally produce a mediocre or (god forbid) a good product.
Re: (Score:2)
... but spending money and time does not guarantee quality.
Nor does a certificate that proves that you spent time and money on paperwork.
It will "catch" on (Score:1)
Ohh, and they also forgot t
Re: (Score:2)
Wait... I thought Healthcare.gov [healthcare.gov] was coded long ago. Are you working on version 2?
I see a bunch of whiners (Score:3, Informative)
It seems as if their chief complaint is that they were not asked to provide input, and the personal communications with members of the committee didn't go anywhere. That's not how the standards process works (I'm speaking from the IEEE perspective, anyway; don't know how ISO works)... your organization (at least from the IEEE end, this is open to pretty much anybody that can muster up the nominal dues) signs up to be on the standards committee, you pay a nominal fee to be included in the working group, and Pow! Your organization is now a full voting member for the standard.
If you don't sign up for the working group, then it should be no surprise that your input is considered entirely optional and/or ignored entirely.
In the first article, the author describes a management course where a group was supposed to form a consensus. He complains that he disagreed with everyone else, wouldn't change his mind (because of his self-proclaimed "high-standards"), and was therefore excluded from the final output from the group, which then was reported to be a consensus. He disagreed that there was a consensus at all, since he didn't agree with it. That's not how "consensus" works; it does not mean that everybody will be satisfied with the outcome, or even want to be associated with it. He goes on to complain that the ISO process requires "consensus", but since he, and like-minded individuals, disagree with the standard, it should not be cleared as a standard.
Again, not how consensus works. In a consensus process, the majority approve of whatever the final output is, and the objections of the dissenters are noted and made available as part of the standards record. You can look on the website of pretty much any standards organization and access drafts, comments, meeting minutes, presentations, the whole works. This full record can help potential adopters of the standard decide if they want to utilize it or not.
Re: (Score:1)
Re: (Score:1)
Then participate! (Score:2)
Firstly, I'm not going to answer stupid leading questions. What is this, some kind of sound-bite-driven political debate?
If you don't like the way the standard is going, you form an organization of like-minded individuals and join the working group. Spread amongst a group of people, the costs are not that extreme, nor the commitment that dire.
And I don't the ability of education providers and consultants being able to advertise "We teach/use the XYZ Standard" as being some sort of nefarious plot. If you
Curse my typos! (Score:2)
*And I don't see how the ability*...
*ever*
*not reached (or failed to be reached)*
You really don't get it... (Score:2)
First, I refused to answer those questions not because I bow down to suspect authority. (Where did THAT come from?) I refused to answer them because they are stupid. Of course the answer is "no", but that answer establishes nothing, because the questions imply conclusions that themselves are up for debate. If this is the way you and your like-minded compatriots engage in debate, no wonder you can't get anybody to take you seriously.
And I keep seeing this claim that the ISO process is set up to "suppress
Re: (Score:2)
Re: (Score:2)
That's not how the standards process works (I'm speaking from the IEEE perspective, anyway; don't know how ISO works)...
Typically when a multi-agency standard is published then one agency will do the lifting and the others will change the title and publish away with a unified number scheme. You'll find one of those agencies (I bet it will be the IEEE) has the working group and the other two agencies will have a couple of members on the review panel and that's it.
ISO/IEC 29500 (Score:1)
ISO? Who are they?
Re: (Score:2)
International Social Outcasts.
Can it be a *useful* standard (Score:3)
Re: (Score:2)
Consensus? You have to be kidding... (Score:2)
The free dictionary (by Farlex) defines consensus: 1. An opinion or position reached by a group as a whole.
That's very democratic. Unfortunately, reality is not democratic.
Software testing is designed to unveil real vulnerabilities and errors in a complex system. Having a bunch of people hold up their hands and say, "Is this a problem?" is flatly ludicrous. In point of fact, it's the error that isn't noticed by the majority that constitutes the deepest problem. Remember the Columbia shuttle? A group of
Only $775 to read the first 3 parts (Score:1)
First 3 parts of 29119 cost $775, the total for all the parts when they are finished will probably be around $1200. This seems to be more about money than anything else.
Of course you can have a standard (Score:2)
You may have to extend it in some cases, but that's normal.
On standards (Score:3)
The best known standard quip about standards itself has multiple versions and attributions. How meta:
"The nice thing about standards is that you have so many to choose from." - Andrew S. Tanenbaum, Computer Networks, 2nd ed., p. 254 [wikiquote.org]
"The nicest thing about standards is that there are so many of them to choose from." -- Ken Olsen [brainyquote.com]
“The wonderful thing about standards is that there are so many of them to choose from.” -- Grace Murray Hopper [goodreads.com]
See also:
Obligatory (but who set that standard?): xkcd : Standards [xkcd.com]
Why are there so many plugs and sockets? [www.iec.ch]
‘Mediocrity finds safety in standardization.’ -- Frederick Crane
‘It is not enough that X be standard, it should also be good.’ -- Rob Pike (Window Systems Should Be Transparent)
The two above can be found on the cat -v page on standards" [cat-v.org]
"Standards are like toothbrushes. Everybody wants one but nobody wants to use anybody else’s." -- Connie Morella [niso.org]
On "concensus" (Score:2)
CAPA vs. Barrier to Entry (Score:2)
Most people don't understand the compliance. There's good and bad, but there's no going back once your industry (candle makers, software writers, barbers, whoever) adapts a standard it invariably becomes a tool of an authority.
Good: What I like about it is that our certifications increase accountability by encouraging recording mistakes. The "routine" of flagging mistakes and finding root causes and formalizing "corrective and preventative action" has been good and improved our company.
Bad: These stand
Re: (Score:2)
Re: (Score:2)
You have to pay.
Re: (Score:1)