Forgot your password?
typodupeerror
Databases Government United States

NYT: Healthcare.gov Project Chaos Due Partly To Unorthodox Database Choice 334

Posted by timothy
from the even-if-it's-perfectly-nice dept.
First time accepted submitter conoviator writes "The NY Times has just published a piece providing more background on the healthcare.gov software project. One interesting aspect: 'Another sore point was the Medicare agency's decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.'" The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project.
This discussion has been archived. No new comments can be posted.

NYT: Healthcare.gov Project Chaos Due Partly To Unorthodox Database Choice

Comments Filter:
  • follow the money (Score:5, Insightful)

    by ganjadude (952775) on Sunday November 24, 2013 @11:23AM (#45506927) Homepage
    Who owns this company?

    how much do they contribute to XXX???

    There has got to be some reason that this DB that ive never even heard of (and i work with DBs, its not my main point of work but I know my way around DBs) got the gig over the more established players.

    or, perhaps they went with it because it is less known and therefore reduce the risk of known attacks in other DB systems?
    • by Anonymous Coward on Sunday November 24, 2013 @11:33AM (#45506971)

      from wikipedia,

      The company’s main product, MarkLogic Server, is an XML based database management server.

      This sentence has basically summed up the entire problem.

      • by Desler (1608317) on Sunday November 24, 2013 @11:54AM (#45507077)

        They should have just used MongoDB. I hear it's web scale. *ducks*

      • WOW! No wonder they're having trouble. This has to be the worst idea for a database I've ever heard of. It definitely explains the problems they've been having.

        • by mlts (1038732) * on Sunday November 24, 2013 @01:09PM (#45507505)

          For them, it is a wise choice. It brings job security using a non mainstream RDBMS, which means that the data would take man-years to be migrated to Oracle, DB/2, or even MS SQL server which may not have the horsepower of a SPARC or POWER7 box, but one can always partition and cluster the RDBMS. It also is expensive to keep maintained, so it provides additional job security, as the expertise for this architecture is rare.

          Great win for the contracting company. Big loss for everyone else.

        • by PapayaSF (721268) on Sunday November 24, 2013 @01:35PM (#45507653) Journal

          It definitely explains the problems they've been having.

          But this is only part of it. Check out this diagram. [nrcc.org] I'm no expert, but look at all the government systems in the upper left that Healthcare.gov is supposed to communicate with, in real time. And on the right, 50 state Medicaid systems. And at the bottom, all the insurance companies. Far, far simpler IT projects have failed. This site will not be ready by the end of the month as promised, and there is a good chance that it will never work as planned.

          • Did you notice the box in the lower right that shows they have a plan to manage the serfs?

            Where do they come up with these things?

            • The chart doesn't show "employee or consumer" connecting to the portal / web site, instead connecting directly to the administration interface, customer service, and financial services (last two of which, based on the gradient, I assume have shared responsibilities between carriers and states, which implies another unspecified interface..).

              Maybe they just didn't think the web portal was important, because the consumers would hook directly (via implanted electrodes with network port? I assume that would be

          • Re:follow the money (Score:5, Interesting)

            by artor3 (1344997) on Sunday November 24, 2013 @05:02PM (#45509291)

            The chart's hosted on the National Republican Congressional Committee's website. I would take it with a heaping tablespoon of salt, if I were you. It's say to say that chart was designed to look as scary and confusing as possible.

      • by sycodon (149926)

        XML soo sucks for db applications.

        • by EMN13 (11493)

          And you base this on what? Did the spagetti-monster tell you JSON was technically a better fit, or that XQuery doesn't work?

          • I don't mind XML for stuff like Docbook ... but for databases what's the point? XML is an editing format with some down and upsides, using it as a storage format for data never edited in XML is just retarded.

            At best it just provides a useless intermediary format which doesn't affect usability ... at worst you try to use XQuery to actually perform searches on large volumes of data.

          • by Chris Mattern (191822) on Sunday November 24, 2013 @12:56PM (#45507445)

            In fact, all of those would suck donkey balls. That is because XML, JSON and XQuery are all *data-interchange formats*. Using them to run your database internals is like using a screwdriver as a hammer.

          • The only reason people store non key field data in one big XML blob is so they can add fields without having to go through useless, ossified DBAs.

            I'm not saying all DBAs are useless and ossified, I'm saying using XML blobs is a workaround for slow, control freak DBAs.

            After this is done, others work with the databases and continue the practice without ever knowing it's root. In some cases the DBAs gain control of the XML schema and the circle is complete.

            • by Aryden (1872756) on Sunday November 24, 2013 @03:03PM (#45508327)
              The reason we are control freaks is because when some moron wants to make a change and breaks the whole system, we are the ones that catch the blame and have to try to fix it if we don't get shit canned for it. You ever tried to re-key millions of records across hundreds of tables? You ever had to spend 3 days restoring db's from backup and having a site outage for 3 days because of it?
      • It's even more basic than that. According to TFA the design goal was: "creating a cutting-edge website that would use the latest technologies to dazzle consumers with its many features. "

        In other words, even if they had gotten this thing up and working as they wanted, right on time, it would have been an accessibility nightmare that would never work unless you have a brand new computer configured with no security anyway.

      • by mosb1000 (710161)

        The problem is this:

        CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.

        Government officials like issuing "cost plus" contracts because they allow them to remain in control of the work while not doing any of it themselves. They probably wanted to use it because they had used it in other projects in the past. If the'd just issued a fixed pice contract and said "get it done" the contractor would have select

    • Re: (Score:2, Insightful)

      by WWWWolf (2428)

      Never attribute to malice that which can be explained by incompetence.

      To me, this sounds like a pretty open and shut case of "Hey, I've heard that these 'NoSQL' database thingies are trendy these days. Let's use one of those!"

      There's a difference between using fun, exciting new technologies and learning something new while doing that... and doing a project which stays in schedule and budget, is based on technology you already know thoroughly, and on which people's lives can depend (well, indirectly).

      • by uradu (10768)

        You're exactly right! I've done tons of "exploratory" coding over the years myself, either using some new techniques or new products, just for the fun of it or to learn something new. But that would always be on small, low visibility projects where the consequences of potential poor performance or other issues would be insignificant. To tread new ground on something so big with national visibility is foolish. You'd want the most well established and known reliable and performant tools and techniques possibl

    • There has got to be some reason that this DB that ive never even heard of (and i work with DBs, its not my main point of work but I know my way around DBs) got the gig over the more established players.

      or, perhaps they went with it because it is less known and therefore reduce the risk of known attacks in other DB systems?

      Perhaps either is the case: Obscurity or donations... A third option is that it provides some feature that doesn't exist in Oracle/SQL Server/Cassandra. However in digging into their web-site it looks like it is some sort of wrapper/hybridized product that mates to Hadoop, which would make sense given the vast volume of data you're talking about managing for the Federal exchange, which I believe services 36 states.

      But with that said, I've never heard of it either.

    • by MyDixieWrecked (548719) on Sunday November 24, 2013 @12:07PM (#45507161) Homepage Journal

      I used marklogic when I worked at a previous job and after learning how it worked and understanding it better, it made our jobs incredibly easy. It just had a serious learning curve.

      Marklogic is a nosql db, that uses XML for its object format and xquery for its query language. This thing is NOT mongodb. It actually works really well and allows for complex data modeling with the ability to do joins and have transactional isolation in making changes to the data as well as a really solid content processing framework with pipelining and all that jazz.

      Now, I can't imagine a reason for using marklogic, or any non-relational db for a project like this. The only clue is that marklogic has a lot of government contracts; mostly for the military. So maybe that's why it was used. But the fact that they chose a database system that they weren't experts in for a project that had so much visibility speaks volumes on how mismanaged this whole project was.

      • The only agencies that would need NoSQL type support are the ones spying on us. My take is that the insurance industry said they would only communicate in EDI-XML formats and somebody in management though that meant buying an XML-based system to be "compatible".

      • This thing is NOT mongodb. It actually works really well and allows for complex data modeling with the ability to do joins and have transactional isolation in making changes to the data as well as a really solid content processing framework with pipelining and all that jazz.

        So is SQL attached to any reasonably good content management framework.

      • by Nerdfest (867930) on Sunday November 24, 2013 @12:24PM (#45507259)

        To me, the fact that it was a major problem is an indicator of bigger problems. With systems like this, the database should be just a simple repository for persisting an retrieving domain models. If they want to do any non-trivial reporting on it the data should be replicated to a reporting repository. Treating the database a a simple persistence repository makes most database operations trivial, and better yet, decouples the database from the business, meaning that if you couldn't even manage the simple operations, replacing the layer is fairly trivial.

        The problem is when people think the database is the starting point for a system, not just a tool to be used to support what your business logic is doing. It sounds like that's what happened here.

        • by Nerdfest (867930)

          I should add that with many languages and frameworks (Java & Spring/Hibernate for example), "we have machines for that". Only the fine tuning and performance tweaks need to be done my hand. Manually coding an data access layer is almost a "make work project" that's frequently a source of errors in what should really be managed by configuration.

      • by countach (534280)

        If a database is good, it should be possible to use it for ALL problem domains.

        Of course, by this criteria, a relational database is NOT "good".

        But I would have thought an object database is good, and would work in any situation. Unfortunately, they never seemed to really take off.

        Anyway, I'd rather be using an object database for complex data of ANY kind than some crazy XML database. I can sort of see why a health application that might have tons and tons of fields might be tempted to use an XML storage, b

    • by jythie (914043)
      It is also possible that someone high on the chain was simply enthusiastic about it for no apparent reason. Several projects I have worked on used inappropriate databases because someone with enough technical knowledge thought it would be a good idea and their persistant insanity was enough to get it chosen.
    • http://www.marklogic.com/what-is-marklogic/marklogic-hadoop/ [marklogic.com]

      This seems like the relevant, interesting/crazy part. Whether it's the right choice or not, I leave to more seasons database professionals than myself.

    • Who owns this company?

      how much do they contribute to XXX???

      "XXX"?

      You mean "Obama and other democrats", right? Because this system is their baby.

      It's no mystery who "XXX" is; no need for a variable.

      I've noticed a Responsibility Diffusion Field around this whole thing ...

  • Noobs? (Score:5, Insightful)

    by Anonymous Coward on Sunday November 24, 2013 @11:26AM (#45506941)

    FTA: "An initial assessment identified more than 600 hardware and software defects — 'the longest list anybody had ever seen,' one person involved with the project said. "

    Strikes me as none of these people seemed to have ever worked on large projects before.

    • Re: (Score:3, Funny)

      by Anonymous Coward

      You don't understand. That's not "600 bugs in the software" that they are talking about. "Thousands of bugs in software" is just one of 600 items that was a defect. Others included "Not enough servers", "Data connection to limited in capacity and speed", and apparently "No hablo espanol".

      • I'm pretty sure he understands. Like he said, it seems to me as though those people have never worked on large projects before.
    • Re: (Score:3, Insightful)

      by larry bagina (561269)
      I counted 279 defects. (60 in the senate, 219 in the HoR). Had but 1 in the senate or 4 in the house been fixed, all this would have been averted. My kingdom for a horse, indeed.
  • NIH syndrome (Score:4, Insightful)

    by Gothmolly (148874) on Sunday November 24, 2013 @11:28AM (#45506953)

    You could take a handful of proven DB technologies such as Oracle/DB/MSSQL, throw a web (Apache/IIS) and app (.Net/WAS/Jboss) front end to it, and it would work. Why did these guys fuck up the whole thing? It's like the scene in The Fountainhead when the second-rate architects smash up the plans and add their own stuff, "to express their own individuality". This could have been a solved problem - hell, it WAS a solved problem.

    • Re:NIH syndrome (Score:5, Insightful)

      by Okian Warrior (537106) on Sunday November 24, 2013 @11:52AM (#45507065) Homepage Journal

      Why did these guys fuck up the whole thing?

      All actions by Federal government players seem rational when viewed from their point of view.

      The problem with most analysis, here or on other sites across the web, is in the assumptions. Specifically, if you assume that the government actions are for some benefit to the people, or efficiently completes some task (which is assumed to benefit the people), then nothing they do makes sense. That assumption makes for good outrage which can attract readers, but it doesn't answer the question of "why".

      When government actions are viewed in light of more obvious motivations, everything it does makes sense. You start to see them not as an pack of incompetent, bureaucratic screw-ups, but as a self-sustaining, self-absorbed, engine of corruption.

      I'm not aware of *anything* the federal government's done in the past two decades that's been in the interests of the people. All the useful stuff, the stuff you would want to keep across a reboot, was several decades ago.

      I don't know the specifics of why an obscure database was chosen over the obvious players, but the reason wasn't "best utility for the people". Look for another reason.

      • Re:NIH syndrome (Score:5, Insightful)

        by jythie (914043) on Sunday November 24, 2013 @12:17PM (#45507221)
        Sad thing is, much of the behavior one sees out of federal contracts is due to taxpayer groups demanding anti-corruption measures. A great deal of the bureaucracy comes directly from people complaining about waste and demanding a complex audible process.

        I would actually put forward that the problem is not the government, our government is pathetically weak in many ways. The problem are all the private groups flexing their muscles and making the government dance. Most of the people I have worked with on these kinds of projects are good people who just want to get stuff done, but they are weighted down by a nearly impossible set of requirements, many of which are mutually exclusive, dropped on them by outside groups who mainly are interested in showing off their power to their own people and have no investment in the project itself being successful.... and often have an active interest in the project failing since they can always use that as fuel for more personal power.
        • Surely it's more due to lobbyists corrupting the anti-corruption measures demanded by taxpayer groups.

    • We were looking at what we could figure out about the architecture of healthcare.gov, and one problem is that it looks like it's using Oracle Identity Manager to manage the permissions of what users can/can't see. That means that OIM is burned in - and it's probably brutally slow, since every time you need to check a permission you go through OIM.

      I'm not positive that's the case, but it fits given what pieces of the architecture I've seen. It would also explain why the system doesn't perform - permissions c

    • It is a solved problem if and only if you limit database operations to today. Then you can use today's best software and practices.

      But this is too big a system to be taken through the revisions that are needed every 7 to 10 years to keep a DBMS up to date. For one thing, once it is in full operation it cannot be taken off line for a weekend upgrade. That adds a lot of headaches as you can count on always having to support simultaneous database operations by two or more different versions of software: the p

    • by mlts (1038732) *

      You hit the nail on the head. There is just so much expertise at building a mainstream Web stack and backend database, that it is almost acting in bad faith not to take advantage of it.

      Then there is the security aspect. You can throw money to buy the skills needed at Oracle to have it locked down. Same with Windows, Linux, Solaris or AIX for the OS, Apache or IIS for the webserver, and a backend platform of choice.

      Using something so off the beaten path may just mean that something can't be done on the pl

  • MarkLogic = NoSQL (Score:5, Interesting)

    by NeverWorker1 (1686452) on Sunday November 24, 2013 @11:35AM (#45506979)
    A little googling turns up that MarkLogic's offering is NoSQL. Without getting into the whole SQL/NoSQL debate, I can't help but noting that this is clearly relational data with a fairly limited number of records (clearly there can't be more than 300M people listed) and for which ACID is (or should be) a major concern.
    • by mobby_6kl (668092)

      But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?

      • Re: (Score:2, Funny)

        by Anonymous Coward

        They should have used MongoDB, because MongoDB is webscale.

      • by Assmasher (456699)

        AKF cube.

        Healthcare.gov isn't a particularly difficult scaling problem to solve. You can 'swim lane' the hell out of it.

        It's not like it's a social network where people need to be constantly sharing things and an arbitrarily large (potentially huge) group of people need to be notified of everything each of them does/doesn't do...

      • by DarkOx (621550)

        But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?

        I don't know if you are kidding or not but I'll try anyway.

        The US population might hit 1 billion inside the period where something form of the ACA is intact in a recognizable way. Unlikely but its possible. Using partitioning schemes it should be fairly simple to scale on traditional database technologies because for most queries you don't need to know about Abby to handle a transaction regarding Bob. There are some where you do but even those are mostly simple 1:1 update operations, you might need to

      • by swilver (617741)

        It would scale just fine, assuming you make proper indexes for your queries.

        Since the general use case is probably gonna be something like "pull up records for patient that is sitting infront of me and edit them", I think an index lookup + atmost a couple of dozen records pulled from related tables will be handled just fine.

        Any other use cases that involve statistical analysis on all data can be handled by copying it to specialized DB's every few weeks.

      • Re:MarkLogic = NoSQL (Score:4, Informative)

        by gmuslera (3436) on Sunday November 24, 2013 @12:36PM (#45507323) Homepage Journal
        Google and Facebook seem to be doing pretty well having that order of users. Facebook even uses simple mysql replication and their pool scanner [facebook.com]. And health system is not a social network, there is no so much connection between different users.
      • by mcgrew (92797) *

        You make little sense there. By the time there are a billion people in the US we'll either be single-payer like the intelligent, civilized world has, or we'll go back to "no money? Let 'em die."

      • by DaveAtFraud (460127) on Sunday November 24, 2013 @02:41PM (#45508121) Homepage Journal

        But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?

        Current population of the U.S. is a little over 300 million. That includes children, people who have company provided health insurance, etc. who don't need to access HealthCare.gov so that the number of users of HealthCare.gov is expected to be about 30 to 45 million people.

        The only way HealthCare.gov would need to support 1 billion or more users would be if we inflicted it on the China or India. That could lead to war or poor technical support and customer service. Although the later does raise an intereting question of who does someone at a call center call when they need technical support? Likewise, the population of the planet is between 7 and 8 billion. For the 1,000 billion users are you planning on also providing health insurance to E.T?

        Cheers,
        Dave

    • Re:MarkLogic = NoSQL (Score:5, Interesting)

      by Anonymous Coward on Sunday November 24, 2013 @11:47AM (#45507045)

      A customer at work has a MarkLogic database. I don't know what version it is, but I assure you IT IS HORRIBLE. It's like... an XML database or some shit. Just awful, and extremely confusing to use.

      A couple years ago I was asked to do automatic database backups. The only documentation I could find for backing up the database requires logging into a web interface I had no idea existed on an obscure port and clicking a backup button. I literally had to write a perl script to fake a browser and do this just so they could get a database dump.

      Do not use this product. Please.

    • Oh, but they claim to be ACID complaint: http://www.marklogic.com/blog/can-you-pass-the-acid-test/ [marklogic.com]
      lol

    • by MyDixieWrecked (548719) on Sunday November 24, 2013 @12:16PM (#45507215) Homepage Journal

      Marklogic, afaik, is the only acid compliant nosql solution that exists.

      • by xaxa (988988)

        Marklogic, afaik, is the only acid compliant nosql solution that exists.

        Neo4j is an ACID-compliant graph database (so it's NoSQL, but that word is a bit pointless).

        It's also the most useful non-relational DBMS: relationships become first-order entities, having a type and properties.

        http://www.neo4j.org/ [neo4j.org]

  • by ilsaloving (1534307) on Sunday November 24, 2013 @11:40AM (#45507007)

    And here is a fantastic example of what happens when hype trumps common sense. NoSQL is the new hawtness, and apparently the dumbasses running the project wanted to be part of that. Now MarkLogic, and NoSQL in general, will have a massive blow to their reputations, and it's unknown how badly this will hurt them.

    As someone who has done databases for a long time, I have very little respect for NoSQL, but that is mostly because everyone keeps trumpeting it as a killer of traditional databases. There are scenarios where NoSQL systems are an ideal fit. However, NONE of those scenarios require data to be very reliably stored in a guaranteed and predictable way.

    If you don't get your tweets or your friends facebook posts as soon as they are posted, no one will really care. But for something as truly important as health insurance coverage? Are you f__king kidding me? And that's just from a reliability standpoint. Nevermind the fact that NoSQL is currently at the wild west stage where nobody is compatible with anybody else, there is nothing resembling a standard set of APIs between products, making it very difficult to develop expertise.

    Sounds like the Gov was just begging for problems.

    • Just had this debate with a current project with some wanting to use a NoSQL solution for the whole thing. Problem is most of the data is relational and I stuck my foot down and said we're using PostgreSQL for anything that needs to be retained. That mean users accounts & transaction records, and really all the data is relational.

      Now there are other elements, like the chat system and distributing JSON strings to large numbers of persistent clients that seem a perfect fit for a NoSQL database. Since t

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      The agency imposing their "unconventional" choice of DB, and this sentence:

      "They ordered CGI technicians to drive from their offices near Dulles International Airport in Virginia to the agency headquarters near Baltimore to review their code with government supervisors."

      sum up everything that went wrong.

      Basically you had a bunch of arrogant idiots who thought themselves experts (which they may well be, _in the healthcare domain_) who not only ignored the views of the _software/IT_ experts ("technician" my a

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      NoSQL predates SQL because we were storing stuff back then to. Relational databases are nice for consistency, but you sacrifice a lot of control and speed improvements for that consistency. A good NoSQL database and a good NoSQL programmer will always outperform SQL based applications. However, it means you need to have programmers capable of operating at that level AND you need to have solid strongly enforced conventions.

      I also think that you are largely misreading the problem. Being able to get a quote an

      • Everything you say is true. However, they didn't call it NoSQL back then. Back then, they were called 'proprietary data stores', all of which ultimately became utterly inflexible in how they dealt with data because the products would be originally designed to satisfy one problem (or set of problems), but then when a new requirement came down the line that needed to manipulate the data in a different way, you had to go through contortions trying to make things work.

        That was the whole reason SQL systems wer

      • Can someone please upvote the parent post? It's actually very interesting, and it's worth reading about this MUMPS system. I had never heard about it myself and it looks interesting.

    • by anon mouse-cow-aard (443646) on Sunday November 24, 2013 @12:07PM (#45507157) Journal
      Why do people insist that NoSQL means losing data and inconsistency? What you are losing with NoSQL real-time consistency horizontally across large number of instances spread over multiple data centres. When you are doing a transaction, and it should be "eventually consistent." meaning on the order of minutes. So if someone, somewhere else, who you do not know about and are not interacting with asks about your data, it might be a few minutes old. ACID makes it so that random person will get an upto the milli-second accurate answer. That makes transactions orders of magnitude slower, and much more complicated to scale. When you relax that constraint and let the people doing the transactions (the person, and the providers they are dealing with) get the right answer immediately, but just post the transaction so that the backups etc... get informed in due course, you gain much simpler scaling. The choice of NoSQL vs. SQL is not about the importance of the transaction. It is about application scaling and design, and the biases (tastes?) of the team doing the work.
    • by Assmasher (456699) on Sunday November 24, 2013 @12:23PM (#45507251) Journal

      I ran into this recently at a company whose new head of engineering was talking to me (an outsider) about the technology problem they had to solve and I thought it sounded very traditional and simple except they'd need to carefully plan for horizontal scaling.

      Basically a potentially huge number of devices (in the range of millions) would be reporting in periodic data that had to be stored and potentially evaluated in real-time. The data was quite easily swim laned by geolocation and the data had no appreciable inter-related significance. So basically, one piece of data from one device had nothing to do with any other device's information except in the general sense that can come from a more heuristic correlation of their data.

      I should mention that the new engineering head and I had already (together) handle a situation very similar to this at a previously successful software company.

      Well, the new engineering head had inherited an external architect who had different ideas. All of these different ideas involved things like Cassandra over Hadoop, AMP/Spark, BDAS. He showed me a diagram of the technologies he wanted to integrate and I'd never heard of almost half of them (and I deal with scaling issues all the time), this diagram had about 15 different technologies stacked together. It was crazy - all to solve a relatively simple data volume problem.

      Almost needless to say, I advised otherwise, and afaik they're going the bid data way because it will make it easier for them to pull in VC money since shockingly few VC's actually evaluate technology before they put money in (I do this for VCs also, and other VCs wonder how I get paid to do this, lol.)

  • by CyberLeader (106732) on Sunday November 24, 2013 @11:41AM (#45507013) Homepage

    "Some people, when confronted with a problem, think 'I know, I'll use XML.' Now they have two problems."

    -JWZ

    MarkLogic is an XML database, not a relational database, so if your data primarily consists of XML content then it's the right tool for the job. Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.

    MarkLogic itself is fine if your data fits neatly into an XML schema, but with healthcare.gov that tree is probably enormous and hard to optimize for DB activity.

    • by Tailhook (98486) on Sunday November 24, 2013 @04:48PM (#45509199)

      Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.

      It was the Medicare folks that imposed MarkLogic over the objections of the prime contractor. Calling Medicare a "vendor" is a bit of a stretch. Like non-IT staff calling themselves "customers" of IT in a corporation, as if they have a choice of IT departments.

      Another sore point was the Medicare agency’s decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.

      The CGI argument was that it would slow work. I have absolutely no doubt about that. Every NoSQL system is it's own distinct thing with unique APIs, protocols, conventions, quirks, etc. This is fine when you are dealing with a limited group of good developers that can be expected to rapidly climb a learning curve. This is not fine when you are the top contractor in a herd of contractors, populated by a vast number of mediocre programmers that last learned anything new in the late 90's.

      In that case the correct choice is to select the most ubiquitous technologies. A boutique NoSQL XML-base is not appropriate, even if it is excellent, which MarkLogic may be. I don't know. I just know CGI was 100% correct in objecting to it. CGI's fault in this case was its failure to win that argument.

      Not that it matters much. The project was doomed in any case — political agenda driving pie-in-the-sky requirements using crony contracting — Fucked at birth, as they say.

  • Well they did have the flashiest ads which used lots of buzzwords.
  • by Murdoch5 (1563847) on Sunday November 24, 2013 @11:52AM (#45507067)
    Why not just use MySQL, MongoDB or MariaDB? At least use a database system that has good support, an easy learning curve and loads of followers. That beings said, proper testing would easily of mitigated this entire issue.
  • Just when I think.. wow.. you couldnt screw this up any worse... they step up and prove me wrong.
  • by vrhino (2987119) on Sunday November 24, 2013 @12:20PM (#45507231)
    I worked as a contractor developing a system at FDA. It lasted for 5 years. Inside the Beltway, it's pretty much the same all over. Dysfunctional communications and ridiculous breakdown of authority not corresponding to lines of management. No accountability. Project management requirements that have never been followed by any project. No commitment to the output of requirements gathering. No requirements change control. No performance engineering. Inadequate testing. No acceptance process by the government. IT groups with oversight for contractor output that have never written a line of code. All in all, pretty sick and ugly. Prior to my project there were 5 failed attempts. My project followed PMI practices, worked them hard and succeeded.
  • by JoeyRox (2711699) on Sunday November 24, 2013 @12:38PM (#45507339)
    From the summary:

    "The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project."

    Unless that complexity was necessary to solve a problem, then it is in fact bad.
  • by Cr8zyguy (3442613) on Sunday November 24, 2013 @01:08PM (#45507497)

    It is NEVER the tool, but who and why it was chosen.

    Furthermore, MarkLogic is a legitimate NoSQL vendor with strong ACID.

    So the question is "how does MarLogic screw up the site", without the answer to that question, we should all refrain from pointing finger to merely a small piece in a huge software project.

    NoSQL is NOT the new hotness, it's been out there for at least 5 years and many successful projects are using them, so for the ones that things NOSQL shouldn't be used, wake up and breath some fresh air.

  • So the private-owned company has "officials". Cool...
  • I get there must be tons of complexity in managing healthcare.gov site interacting with all necessary stakeholders... Must be quite a lot of different databases, systems, operators to say nothing of complexity of working the actual problem space. I can understand how there might be glitches that cause wrong rates and plans to be communicated.

    What I am still puzzled by are the "waiting rooms" with 40k people waiting to use a site. What the hell can justify it being so computationally expensive to spit out

  • by BaronM (122102) on Sunday November 24, 2013 @08:34PM (#45510713)

    I've used it, personally, to implement a public-facing website. That site endured the dreaded 'slashdot effect' several times. No failures.

    When implemented properly, Marklogic is damn near unkillable. It will slow down, it will reject connections when queues are full, but it will not fall over. Naturally, this assumes proper underpinnings and capacity calculations. With Marklogic, those are actually documented.

    Mandatory disclosure: I do not have and have never had any association with Marklogic other than a paying customer.

Forty two.

Working...