Forgot your password?
typodupeerror
Firefox Internet Explorer Microsoft

Clashing Scores In the HTML5 Compatibility Test Wars 203

Posted by timothy
from the jake-the-explainer-steps-in-early dept.
Andreas(R) writes "Microsoft has published a set of HTML5 tests comparing Internet Explorer 9 to other web browsers. In Microsoft's own tests, IE9 performs 100% on all tests. However, the Internet Explorer 9 HTML5 Canvas Campaign has published results that show that Internet Explorer gets 0% on all their tests." The results reported here are selected with tongue in cheek: "Therefore, we'll also present shameless results from tests which have been carefully selected to give the results that the PR department has demanded."
This discussion has been archived. No new comments can be posted.

Clashing Scores In the HTML5 Compatibility Test Wars

Comments Filter:
  • by calmofthestorm (1344385) on Thursday June 03, 2010 @06:13PM (#32452276)

    ...with MS HTML# 5.0

    • by Fluffeh (1273756) on Thursday June 03, 2010 @06:45PM (#32452622)
      Wouldn't this be a great opportunity for one of the internet watchdogs to make a suit against MS here? HTML5 hasn't been agreed to yet, here is an advertisement saying that IE9 is 100% compliant.

      That's obvious false advertising isn't it?
      • by DragonWriter (970822) on Thursday June 03, 2010 @06:53PM (#32452690)

        HTML5 hasn't been agreed to yet, here is an advertisement saying that IE9 is 100% compliant.

        The advertisement doesn't claim anything about compliance with anything. It claims that IE9 passes 100% of the tests labelled "HTML5" that Microsoft has constructed.

        It doesn't claim that those tests either represent the whole of the HTML5 spec or any draft thereof, or even that they test behavior required by the spec or any draft thereof, or even -- except by implication -- that passing them indicates behavior that is acceptable under some draft of the HTML5 spec.

        • by Fluffeh (1273756) on Thursday June 03, 2010 @06:59PM (#32452756)

          It doesn't claim that those tests either represent the whole of the HTML5 spec or any draft thereof, or even that they test behavior required by the spec or any draft thereof, or even -- except by implication -- that passing them indicates behavior that is acceptable under some draft of the HTML5 spec.

          In Australia (where I am from) an advertisement needs to either have a disclaimer (normally small text at the bottom) if there is vagueness about what it is saying, or what the advertisement says needs to be taken at face value - meaning "what it implies".

          Surely the US would have that too?

        • Re: (Score:3, Insightful)

          by notrandomly (1242142)

          The advertisement doesn't claim anything about compliance with anything.

          The page clearly gives the impression that IE is more compliant than other browsers in general. There are multiple articles and comments all over the web that clearly show that this is the impression most people get when they see the page. Microsoft, however, chose not to fix their page, so it is still as misleading as it used to be.

      • by mcgrew (92797) *

        The suit would have to be brought against them by a competetitor, since customers can't sue suppliers for false advertising. Or somone from somewhere other than the US would have to sue. I wish we had something like Britain's ASA.

    • by Shin-LaC (1333529) on Thursday June 03, 2010 @06:48PM (#32452650)
      And the award for Best Internet Explorer goes to... Internet Explorer 9!
  • Build Your Own Test (Score:5, Informative)

    by eldavojohn (898314) * <eldavojohn AT gmail DOT com> on Thursday June 03, 2010 @06:14PM (#32452288) Journal
    First off if this is a technical discussion, we should probably be talking about layout engines -- not browsers. Secondly their HTML5 capabilities are well documented [wikipedia.org]. You can come up with whatever perventage you want from those charts as some things (Video) might be deal breakers compared to others (MathML).
    • by Elektroschock (659467) on Thursday June 03, 2010 @06:25PM (#32452398)

      Indeed. However, Microsoft has a poor record of interoperability which only improved recently. So it needs to regain trust. The way to regain trust is to actually improve interoperability and standard conformance, no mere marketing and public affairs campaign. Real credibility stems from real achievements. I am sure Microsoft is able to become an interoperability leader.

      • by Jah-Wren Ryel (80510) on Thursday June 03, 2010 @06:40PM (#32452558)

        I am sure Microsoft is able to become an inoperability leader.

        Fixed that for ya!.
        Yeah it's old, but it's good.

      • by poetmatt (793785)

        what have they exactly improved with interop? I've never seen them any different than normal, aka zero interop unless mandated by courts.

        • by dAzED1 (33635)

          well yeah, but considering how lawsuit-happy we increasingly are, then hey...it's not why they did it, but that they did it...right?

          • by poetmatt (793785)

            har har. making 1% of your stuff interop versus the 99% incompatible isn't really a step forward. That's like open sourcing your shoelace while all the rest of the clothing is still proprietary.

        • what have they exactly improved with interop? I've never seen them any different than normal, aka zero interop unless mandated by courts.

          IE8 is as fully CSS2.1-compliant as any browser. You no longer need to write separate stylesheets for IE8 in practice, you just have to make sure not to use CSS3 features (it only implements a couple). IE9 improves standards compliance in a host of ways. None of this is court-mandated, it's probably just Microsoft realizing that they'd like web developers not to loathe them with the fury of a thousand blazing suns.

      • by rolfc (842110)
        Of course they are able to become an interoperability leader but that would ruin their business idea.
    • Re: (Score:2, Insightful)

      by Anonymous Coward

      First off if this is a technical discussion, we should probably be talking about layout engines -- not browsers.

      Meh. If I code a website, I want to know which browsers can show it correctly and which can't. I don't care about whatever layout engines the browsers use, that's pretty irrelevant for me. (Okay: I don't really care even about the browsers, I care about users. Certain demographics tend to use certain browsers, etc... But as most of the statistics about the subject are about browser shares, not layout engine shares, I see no reason to switch to talking about the layout engines there.) If I want to choose a n

      • Meh. If I code a website, I want to know which browsers can show it correctly and which can't.

        That is disgraceful.

        Code to standards. Let the poor browsers fail.

        Microsoft created this situation, you should expose their failings, not paper over them.

      • You do realize that significant numbers of browsers share the same layout engines ? So if your site works in the layout engine - it will work with *all* the browsers that support it (give or take a tiny bit).
        For example Google-chrome and Safari both use a variant of the KHTML engine (known as WebKIT - KTHML itself is used by Konqueror and several related KDE projectS), as does any embedded browser using default QT4 features. There are literally dozens of browsers that use the gecko engine from firefox. In

    • Re: (Score:2, Funny)

      You can come up with whatever perventage you want from those charts as some things (Video) might be deal breakers compared to others (MathML).

      i avoid high schools to keep my perventage low.

    • by DFJA (680282) on Thursday June 03, 2010 @06:38PM (#32452536)
      I never deal in perventages, it just seems...well, a bit percerted.
    • Re: (Score:3, Informative)

      by matthewv789 (1803086)
      Here is a better listing of specific capabilities for different browser versions: http://caniuse.com/ [caniuse.com] It's still unclear how competitive IE9 will be at the time it comes out, but it will clearly be a huge leap forward from IE8. (The big problem will be getting all the existing IE6, 7, and 8 users to migrate to IE9 or other modern browsers.)
  • Yeah that's certainly entertaining and all but...do we have any real tests with real results?
    And by "real" I mean tests that includes all HTML5 specifications...
    • by thoughtsatthemoment (1687848) on Thursday June 03, 2010 @06:38PM (#32452532) Journal
      No, at least not before we have a real HTML5 spec.
    • by jpmorgan (517966) on Thursday June 03, 2010 @08:46PM (#32453646) Homepage

      Well, that's the problem with HTML. The W3C doesn't create an acceptance test, so there's really no objective way to measure how compliant a particular browser truly is. People love to use the ACID tests, but ACID tests only a small portion of the relevant standards. And the portions tested aren't even the major, important parts; ACID tests for very obscure, esoteric parts of the standards.

      On one hand, you can look at the ACID tests and say 'well, at least it's an indication of interest in conforming to the standard.' But is that true either? ACID tests have become another marketing point: 'my browser got to 100% compliance before your browser.' Aiming for 100% on the ACID tests doesn't necessarily indicate a desire to be highly compliant, it indicates a desire to score 100% on the ACID tests.

      You could perhaps consider the instantaneous behavior of the tests: how compliant various browsers are upon release of the new test. There's a certain logic to that; developers which are truly interested in compliance, and not just marketing, will do well in a previously unseen test. But ACID tests aren't developed in isolation either. They're politically justified, an effort to encourage compliance, and as such they test for specific behaviors which major browsers were getting wrong (i.e., a browser could be 99.9% compliant, and ACID would target the 0.1% they get wrong).

      So to answer your question: No. There's no comprehensive compliance/acceptance test for any of W3C's standards, so don't expect one either. The only evidence of compliance is anecdotal, and the plural of anecdote is not data. Microsoft's test results are completely unsurprising and generally meaningless for anybody familiar with normal development practices, and W3C standards, but it's a nice indication that they're aiming for at least some level of standards-compliance in IE9.

      • by Psaakyrn (838406)

        On hindsight,
        1) Why can't the tests be created with a random generator? Create webpage, internal logic calculates image which should be generated, test image generated with screenshot (or better, pull the image from the drawing cache).
        2) Along with Javascript/CSS, shouldn't it be possible to test with a range of values anyway? Granted a full test may take a significant amount of time (since you have to test multiple values at the same time), but for a full test, I don't think this would be an issue. An adde

        • Re: (Score:3, Interesting)

          by koiransuklaa (1502579)

          1) Why can't the tests be created with a random generator? Create webpage, internal logic calculates image which should be generated, test image generated with screenshot (or better, pull the image from the drawing cache).

          "internal logic", huh? Do you realize that means a complete implementation of all the relevant standards with no faults. If we could do that, maybe we should put it in a browser...

      • Well, that's the problem with HTML. The W3C doesn't create an acceptance test, so there's really no objective way to measure how compliant a particular browser truly is.

        An official test suite will be developed for HTML5. It's a huge amount of work, though. Microsoft is contributing tests as they add features to IE9, which is great. Of course they pass all the tests; they're probably designing the tests as part of their implementation, as part of an automated test suite for IE9. But it's great that they're helping out by contributing their internal tests to the public for use in a comprehensive cross-browser test suite.

        The only obnoxious thing here is that they're pos

  • by thestudio_bob (894258) on Thursday June 03, 2010 @06:19PM (#32452344)
    When you can't beat em, change the rules.
  • by bogaboga (793279) on Thursday June 03, 2010 @06:19PM (#32452348)

    Well, let knowledgeable slashdotters point us novices to a set of a "standard" HTML5 test site to which we can run and establish the fact.

    Ohh wait, I forgot that there is yet to be any agreement on the HTML5 standard itself! This is why I think Apple is just bluffing with their campaign against Flash. It also demonstrates the weaknesses we all have to work around.

    • Re: (Score:2, Informative)

      by maxume (22995)

      There is plenty of agreement in the html5 draft, lots of it is not controversial.

      There certainly is not complete agreement.

    • Re: (Score:2, Informative)

      by GoochOwnsYou (1343661)
      There isn't full agreement but most of it is pretty complete. The only non nitpicking issue that people cant agree on is video/audio. Microsoft and Apple want push h.264 into their browsers and push h.264 as a de-facto standard so they advocate against defining a codec in HTML5 (an open standard). Of course they dont support anything else in IE and Safari(for HTML5 video tags)

      The other camp: mostly the open source community push for Ogg containers (Theora/Vorbis), despite h.264 is a superior codec Micr
      • Re: (Score:3, Interesting)

        by Simetrical (1047518)

        There isn't full agreement but most of it is pretty complete. The only non nitpicking issue that people cant agree on is video/audio. Microsoft and Apple want push h.264 into their browsers and push h.264 as a de-facto standard so they advocate against defining a codec in HTML5 (an open standard). Of course they dont support anything else in IE and Safari(for HTML5 video tags)

        If you think audio and video codecs are the only part of the spec that's controversial, you clearly don't follow the HTMLWG mailing list. There are 29 open issues [w3.org], and many of them have been hotly debated. So have lots of other issues that weren't formally raised to the tracker.

        The video codec issue actually was resolved long ago – the spec just doesn't say what codecs are required, and no one is really objecting to that. Mozilla, Opera, and Google support open codecs, but none has suggested that

  • Clearly... (Score:2, Insightful)

    by JansenVT (1235638)

    Clearly, the independent, third-party tests are flawed. Microsoft would never create a biased benchmarking test to promote their own product.

    Seriously though? The only people that understand what HTML5 is and what these results actually mean are going to understand that it is complete nonsense.

  • by ottffssent (18387) on Thursday June 03, 2010 @06:24PM (#32452390)

    TFA: "The first table is a summary of the test results with the May 2010 IE Platform Preview and each of the major shipping browsers running on Windows."

    So...IE8 isn't a "major shipping browser" that runs on Windows?

    If IE8 scores so terribly that Microsoft is embarrassed to post its scores, that's fine, but it would be less dishonest and more informative then to include recent betas of their competitors' browsers in addition to the latest shipping version.

  • In other news, something surprising happened.

    Coming up next: are angry hobos stalking your Facebook account? What you don't know might eat you. Film at 11.

  • Sex analogy (Score:3, Funny)

    by Anonymous Coward on Thursday June 03, 2010 @06:27PM (#32452424)

    That's like saying slashdotters are 100% successful sexually.

    If the tests that include the opposite sex are excluded.

    • Fuck that. Car analogy.

      It's like testing several cars with similar performance on a very long, winding racetrack covered with obstacles and comparing their lap times to that of another car that drives around a very short and straight loop.

      It's easy to say who gets first place when you get to choose the conditions of winning. Doesn't matter, though, the point here isn't to make Internet Explorer more popular, it's to make Microsoft look like they're competing fairly and remaining relevant.

      • Re: (Score:3, Funny)

        by Spatial (1235392)
        Where's Bad Analogy Guy when you need him?
        • Re: (Score:3, Funny)

          by geekoid (135745)

          His Analogy would hit that bullseye, then the rest of the dominoes will fall like a house of cards. Checkmate.

          • by dangitman (862676)

            His Analogy would hit that bullseye, then the rest of the dominoes will fall like a house of cards. Checkmate.

            Game, set and match.

        • by Minwee (522556) <dcr@neverwhen.org> on Thursday June 03, 2010 @07:59PM (#32453286) Homepage

          Where's Bad Analogy Guy when you need him?

          Well, Bad Analogy Guy is kind of like a car. And the radio only gets two stations on AM, but there's an eight track with a copy of "Journey's Greatest Hits" stuck in it. If you look at it that way then this discussion is something like an eight hour drive from Tulsa, OK to one of the Portlands. I can't remember which one, but it's eight hours away by car. Now the car has wood grain paneling on the right side and some kind spray-on granite countertop on the left, so the driver can lean out of the window and chop tomatoes as long as the passenger leans over to take the wheel.

          The rest of us are the two pedigreed schnoodles sitting in the back seat, trying to eat bacon and egg sandwiches.

          Does that answer your question?

          • by Aphoxema (1088507) *

            It's all so clear now... my days of nihilism are finally over...

          • by macshit (157376)

            ...
            The rest of us are the two pedigreed schnoodles sitting in the back seat, trying to eat bacon and egg sandwiches.

            Does that answer your question?

            You forgot the freaky right-wing rant!

      • by zooblethorpe (686757) on Thursday June 03, 2010 @06:49PM (#32452658)

        That's like saying slashdotters are 100% successful sexually.

        If the tests that include the opposite sex are excluded.

        Fuck that.

        We have a winner! :D

        Cheers,

  • by The MAZZTer (911996) <megazzt@gmail . c om> on Thursday June 03, 2010 @06:31PM (#32452452) Homepage

    ...that they benchmarked IE trunk against OLD versions of other browsers. They didn't even use Chrome 5.0!

    In some places it's a significant difference. [withinwindows.com]

    I also did some benchmarks of my own on non-Microsoft controlled sites. See the first comment on that page for results. Suffice it to say IE9 has improved since IE8 but still has a ways to go.

    • by fermion (181285) on Thursday June 03, 2010 @07:21PM (#32452996) Homepage Journal
      I think the point of the article is that no one implements all HTML standards perfectly. Chrome is an immature browser based on one of the newer rendering engines, so we expect it to mature rapidly, but hardly can expect it to match it's cousin Safari in most areas, thous we expect it would in a short time.
      • by CAIMLAS (41445)

        Chrome is an immature browser based on one of the newer rendering engines,

        Uh, what? Webkit [wikipedia.org], the basis for Chrome, has been around since 1998 (then KHTML). As long ago as KDE 2, Konqueror (using KHTML) was a usable browsing alternative (2001?) and was better than Firefox for some time.

        Apple took KHTML some time ago, forked it for Safari, and a lot of those forked changes got merged back into the engine. Google has since forked the project again, but the engine changes are still evolving and improving.

        Honestly, Webkit is significantly more mature than the other browser engines now:

      • Re: (Score:3, Informative)

        by RAMMS+EIN (578166)

        ``Chrome is an immature browser based on one of the newer rendering engines, so we expect it to mature rapidly, but hardly can expect it to match it's cousin Safari in most areas, thous we expect it would in a short time.''

        I think that depends on how you look at it. Chrome's rendering engine is based on an older version of the rendering engine from Safari, which is in turn based on KHTML from Konqueror, which was forked from khtmlw in 1998, making it about as old as the Gecko engine used by Mozilla. While C

      • by Webz (210489)

        I would hardly call Chrome immature. It has had since its inception features that other browsers are just dying to imitate now. Per process tabs, plug-in sandboxing, site specific browsing, silent updates... And it came out of left field! It's a fast browser popular among the technically literate. And it has its own marketing campaign.

        Not immature.

  • HTML5TEST (Score:3, Informative)

    by v1 (525388) on Thursday June 03, 2010 @06:47PM (#32452644) Homepage Journal

    http://html5test.com/ [html5test.com]

    things like this will have to do until we see something like ACID support HTML5.

  • by Dahamma (304068) on Thursday June 03, 2010 @06:55PM (#32452702)

    Discussion of test results

    Based on the tests that we have performed, it is very clear that there is a very big difference between the best and worst browsers. Therefore we can only conclude that the results are valid and true.

    Now if that isn't a rigorous application of the scientific method I don't know what is!

  • by NicknamesAreStupid (1040118) on Thursday June 03, 2010 @07:02PM (#32452790)
    Microsoft is 100% Microsoft compatible (restrictions and exclusions apply).
    • by xs650 (741277)

      Microsoft is 100% Microsoft compatible (restrictions and exclusions apply).

      Many restrictions and exclusions, such as using the same version.

  • The Difference (Score:5, Insightful)

    by 99BottlesOfBeerInMyF (813746) on Thursday June 03, 2010 @07:32PM (#32453082)

    Here's a difference that everyone should note. When the later Acid tests were formulated they were written by Webkit and Gecko developers and were specifically biased against those engines. If one of the two did not fail, it didn't go in. That way it motivates them to improve. When MS writes a test suite it's biased in favor of their engine, so they can claim to be "ahead" and have no motivation to improve. It's an excellent example of who values technical excellence and who values marketing.

    • Re:The Difference (Score:5, Insightful)

      by Your.Master (1088569) on Thursday June 03, 2010 @07:43PM (#32453174)

      Sounds to me more like all parties are doing test driven development.

      I think the difference here is that the Acid tests were published before anybody went and got 100% of them. But I'd bet that Microsoft wrote these tests back when IE9 didn't pass them, then made IE9 pass them, THEN released the tests.

      • I think the difference here is that the Acid tests were published before anybody went and got 100% of them. But I'd bet that Microsoft wrote these tests back when IE9 didn't pass them, then made IE9 pass them, THEN released the tests.

        Maybe, but there's no way to know that and there's no pressure on them to fix what's still broken because they haven't released a list of what's broken and what it is their goal to fix. So whether MS wrote tests they knew they'd pass or wrote tests then fixed them before publishing them, either way they're not in the same boat and not focused on technical excellence so much as marketing.

    • Re:The Difference (Score:4, Informative)

      by UnknowingFool (672806) on Thursday June 03, 2010 @07:57PM (#32453264)
      At best this demonstrates misrepresentation. MS is like a student who says that they passed did better in Math than their peers. They even have a handy chart. What MS doesn't tell you is that they only tested specific skills like quadratic equations. What missing from this is that Chrome/Opera/Firefox/Safari took the whole Math test from algebra/geometry/trig/calculus/etc while MS only attempted the handful of questions it knew it solve.
    • Well, what do you expect? Microsoft has always been a marketing company first and foremost.

    • Re: (Score:3, Informative)

      by notrandomly (1242142)
      If I am not mistaken, Acid3 was created by an Opera employee at the time (who now works for Google), and was meant to be extremely tough to pass for all browsers.
  • The reason why most tests failed with browsers other than IE:

    1st) Since HTML5 is still in a very early state, many browsers (AKA Webkit, Gecko, Presto) used prefixes for most tags and CSS properties. Example: round borders is -moz-border-radius in gecko, and -webkit-border-radius in Chrome. Some latest versions have taken some out of beta and also read border-radius, but most still don't. IE obviously uses border-radius, and that's why other's don't work.
    2nd) The JS is tricky at best. Go and check it out. Lots of lines of code to perform a simple task, and those lines are carefully selected to fail in other browsers. I downloaded the tests, and they work on ALL browsers (I tested Chrome, Firefox and Opera, all on GNU/Linux, all on their latest version). That JS was crafted to fail on all browsers and work only on IE
    3rd) I took the time to run the source of many of their scripts through the W3C validator. Most scripts have several warnings, some errors, etc. They DO NOT VALIDATE.
    4th) The tests aren't really HTML5. Only the HTML5 tests are actual HTML5, the others are XHTML 1.0 strict ... except they are not, because they use HTML5 styles and tags, and they do not validate. Validator says: The document located at was tentatively checked as XHTML 1.0 Strict. This means that with the use of some fallback or override mechanism, we successfully performed a formal validation using an SGML, HTML5 and/or XML Parser(s). In other words, the document would validate as XHTML 1.0 Strict if you changed the markup to match the changes we have performed automatically, but it will not be valid until you make these changes.

    It's microsoft ... never forget about that. This is business as usual.

  • There was a time when tabbed browsing was a new idea. If I remember correctly, IE was the last browser to get on board with it. In fact, I'm pretty sure that IE is the last browser to get on board with just about everything. Eventually, though, IE does come on board with everyone else (it may take years, but hey - possessing the lion's share of the market share has its advantages). I don't see any reason to believe that this will be any different with HTML5. And then (as now) the so-called 'Browser War
  • by foniksonik (573572) on Friday June 04, 2010 @07:31AM (#32456906) Homepage Journal

    MS needs to fuully separate the render engine fron the browser in a way so that they can be updated individually. There are features in the browser which users may like or dislike, may require or not. Let the browser handle things like plugins just provide hooks for compositing in the render engine. Let the browser handle security, etc. Let the render "render".

  • Who the hell wrote these stupid tests? Whoever it is was a fucking moron.

    I went down the list and picked the first one that Firefox supposedly failed on... “Call select() on a text field.” Firefox fails this test? What the fuck? I’ve used select() on text fields many a time and Firefox supports it just fine. Something’s fishy. So.......... I hit up the test page [microsoft.com] and read the source code.

    If there’s a javascript error, the test obviously fails. A bunch of the attributes of window

    • The Insert Node Into Selection [microsoft.com] test is equally retarded.

      They create a new <div> element and insert it at the end of the selection in an existing <div>. In other words, they generate this:

      <div id="div1">some text<div>new text</div></div>

      ...then they check to see if the value returned by selection.toString() is equal to “some textnew text”.

      Morons. It shows up as this:

      some text
      new text

      ...and that’s what toString() gives you, too.

  • by gig (78408) on Sunday June 06, 2010 @01:19AM (#32473490)

    Microsoft should shut up and ship.

FORTRAN is a good example of a language which is easier to parse using ad hoc techniques. -- D. Gries [What's good about it? Ed.]

Working...