Clashing Scores In the HTML5 Compatibility Test Wars 203
Andreas(R) writes "Microsoft has published a set of HTML5 tests comparing Internet Explorer 9 to other web browsers. In Microsoft's own tests, IE9 performs 100% on all tests. However, the Internet Explorer 9 HTML5 Canvas Campaign has published results that show that Internet Explorer gets 0% on all their tests." The results reported here are selected with tongue in cheek: "Therefore, we'll also present shameless results from tests which have been carefully selected to give the results that the PR department has demanded."
New MS, Same as the old... (Score:5, Insightful)
Here's how to solve the impasse (Score:5, Insightful)
Well, let knowledgeable slashdotters point us novices to a set of a "standard" HTML5 test site to which we can run and establish the fact.
Ohh wait, I forgot that there is yet to be any agreement on the HTML5 standard itself! This is why I think Apple is just bluffing with their campaign against Flash. It also demonstrates the weaknesses we all have to work around.
Clearly... (Score:2, Insightful)
Clearly, the independent, third-party tests are flawed. Microsoft would never create a biased benchmarking test to promote their own product.
Seriously though? The only people that understand what HTML5 is and what these results actually mean are going to understand that it is complete nonsense.
No shipping IE results (Score:4, Insightful)
TFA: "The first table is a summary of the test results with the May 2010 IE Platform Preview and each of the major shipping browsers running on Windows."
So...IE8 isn't a "major shipping browser" that runs on Windows?
If IE8 scores so terribly that Microsoft is embarrassed to post its scores, that's fine, but it would be less dishonest and more informative then to include recent betas of their competitors' browsers in addition to the latest shipping version.
Re:Build Your Own Test (Score:5, Insightful)
Indeed. However, Microsoft has a poor record of interoperability which only improved recently. So it needs to regain trust. The way to regain trust is to actually improve interoperability and standard conformance, no mere marketing and public affairs campaign. Real credibility stems from real achievements. I am sure Microsoft is able to become an interoperability leader.
Re:Build Your Own Test (Score:2, Insightful)
First off if this is a technical discussion, we should probably be talking about layout engines -- not browsers.
Meh. If I code a website, I want to know which browsers can show it correctly and which can't. I don't care about whatever layout engines the browsers use, that's pretty irrelevant for me. (Okay: I don't really care even about the browsers, I care about users. Certain demographics tend to use certain browsers, etc... But as most of the statistics about the subject are about browser shares, not layout engine shares, I see no reason to switch to talking about the layout engines there.) If I want to choose a new web browser for my personal use, I care about how well it can display sites, I don't care about the layout engine underneath it.
The only people for which the actual layout engines are relevant are browser developers. For everyone else, that is irrelevant.
Re:Do we have any *real* test? (Score:5, Insightful)
Re:New MS, Same as the old... (Score:3, Insightful)
Hell, why do that. Just put out a press release saying you won and the other side is lying.
Re:IE has 100% compatability... (Score:5, Insightful)
The advertisement doesn't claim anything about compliance with anything. It claims that IE9 passes 100% of the tests labelled "HTML5" that Microsoft has constructed.
It doesn't claim that those tests either represent the whole of the HTML5 spec or any draft thereof, or even that they test behavior required by the spec or any draft thereof, or even -- except by implication -- that passing them indicates behavior that is acceptable under some draft of the HTML5 spec.
Huff, Huff and gufaw! (Score:4, Insightful)
Re:It's also worth mentioning... (Score:5, Insightful)
The Difference (Score:5, Insightful)
Here's a difference that everyone should note. When the later Acid tests were formulated they were written by Webkit and Gecko developers and were specifically biased against those engines. If one of the two did not fail, it didn't go in. That way it motivates them to improve. When MS writes a test suite it's biased in favor of their engine, so they can claim to be "ahead" and have no motivation to improve. It's an excellent example of who values technical excellence and who values marketing.
Re:The Difference (Score:5, Insightful)
Sounds to me more like all parties are doing test driven development.
I think the difference here is that the Acid tests were published before anybody went and got 100% of them. But I'd bet that Microsoft wrote these tests back when IE9 didn't pass them, then made IE9 pass them, THEN released the tests.
Re:Microsoft's tests seem legit (Score:1, Insightful)
Did any of you 'holier than thou' slashdotters actually evaluate the tests that Microsoft published before ripping them apart? They seem legit so far, and they are pointing out issues that SHOULD be fixed in the other browsers as well.
If so, then good for them. It would be a dramatic change in behavior, given the complete disregard for standards in previous versions of IE.
There is an obvious methodology problem with this test that makes its result questionable: They are testing an unreleased product (IE9) against the shipping version of other browsers. That is like bragging that a PC that will ship in nine months is faster than one that has been on sale for a year. Other browsers are actively working on the same issues, and will be better by the time any real users get IE9. Firefox and Chrome are open source, and provide bleeding edge builds that they could have tested. They chose not to test the shipping version of their own browser.
Being skeptical that the company with the worst track record in standards compliance has completely changed based on a benchmark of their own choosing, comparing an alpha product against stable versions of competing products, is not "holier than thou". It is common sense.
Re:Do we have any *real* test? (Score:5, Insightful)
While you are sitting back and waiting for someone else to write these comprehensive test suites the only entity really taking the W3C Test Suite projects seriously at all is Microsoft. They've submitted thousands of test cases for CSS2.1 and have been working to submit hundreds of tests for CSS3.0 and HTML5. These are the very tests that are being disparaged here. It can be claimed that Microsoft is stacking these tests intentionally, except that these tests are publically available for comment and scrutiny. The Test Suite projects are open for submissions by others as well so Google and Mozilla and anyone else is free to submit tests of behaviors that IE9 fails and that their browser passes. The more comprehensive these test case projects are the more everyone benefits by having a real target to implement.
ACID is not an authority. They cherry pick a relative handful of problematic yet pointless tests and construct something cute. ACID3 tests about a hundred different things from a mix of technologies. The W3C Test Suite for CSS2.1 currently has nearly 8,000 tests. Which do you think is more comprehensive?
Re:Do we have any *real* test? (Score:5, Insightful)
Well, that's the problem with HTML. The W3C doesn't create an acceptance test, so there's really no objective way to measure how compliant a particular browser truly is. People love to use the ACID tests, but ACID tests only a small portion of the relevant standards. And the portions tested aren't even the major, important parts; ACID tests for very obscure, esoteric parts of the standards.
On one hand, you can look at the ACID tests and say 'well, at least it's an indication of interest in conforming to the standard.' But is that true either? ACID tests have become another marketing point: 'my browser got to 100% compliance before your browser.' Aiming for 100% on the ACID tests doesn't necessarily indicate a desire to be highly compliant, it indicates a desire to score 100% on the ACID tests.
You could perhaps consider the instantaneous behavior of the tests: how compliant various browsers are upon release of the new test. There's a certain logic to that; developers which are truly interested in compliance, and not just marketing, will do well in a previously unseen test. But ACID tests aren't developed in isolation either. They're politically justified, an effort to encourage compliance, and as such they test for specific behaviors which major browsers were getting wrong (i.e., a browser could be 99.9% compliant, and ACID would target the 0.1% they get wrong).
So to answer your question: No. There's no comprehensive compliance/acceptance test for any of W3C's standards, so don't expect one either. The only evidence of compliance is anecdotal, and the plural of anecdote is not data. Microsoft's test results are completely unsurprising and generally meaningless for anybody familiar with normal development practices, and W3C standards, but it's a nice indication that they're aiming for at least some level of standards-compliance in IE9.
Re:IE has 100% compatability... (Score:3, Insightful)
The page clearly gives the impression that IE is more compliant than other browsers in general. There are multiple articles and comments all over the web that clearly show that this is the impression most people get when they see the page. Microsoft, however, chose not to fix their page, so it is still as misleading as it used to be.