NYT: Healthcare.gov Project Chaos Due Partly To Unorthodox Database Choice 334
First time accepted submitter conoviator writes "The NY Times has just published a piece providing more background on the healthcare.gov software project. One interesting aspect: 'Another sore point was the Medicare agency's decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.'" The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project.
follow the money (Score:5, Insightful)
how much do they contribute to XXX???
There has got to be some reason that this DB that ive never even heard of (and i work with DBs, its not my main point of work but I know my way around DBs) got the gig over the more established players.
or, perhaps they went with it because it is less known and therefore reduce the risk of known attacks in other DB systems?
Re:follow the money (Score:5, Insightful)
from wikipedia,
The company’s main product, MarkLogic Server, is an XML based database management server.
This sentence has basically summed up the entire problem.
Re:follow the money (Score:5, Funny)
They should have just used MongoDB. I hear it's web scale. *ducks*
Re: (Score:2)
MongoDB only pawn in game of politics. [youtube.com]
Re:follow the money (Score:5, Funny)
They should have just used MongoDB. I hear it's web scale.
Web scale or not, I have no idea. But I do think they've got the best bug report [mongodb.org] ever in their tracker.
Re: (Score:3)
WOW! No wonder they're having trouble. This has to be the worst idea for a database I've ever heard of. It definitely explains the problems they've been having.
Re:follow the money (Score:5, Insightful)
For them, it is a wise choice. It brings job security using a non mainstream RDBMS, which means that the data would take man-years to be migrated to Oracle, DB/2, or even MS SQL server which may not have the horsepower of a SPARC or POWER7 box, but one can always partition and cluster the RDBMS. It also is expensive to keep maintained, so it provides additional job security, as the expertise for this architecture is rare.
Great win for the contracting company. Big loss for everyone else.
Re:follow the money (Score:5, Interesting)
Also, the Federal Government fscking LOVES XML. Like, a lot. The things I saw in the new "protocols" they're deploying for internatl air-traffic control data stuff... Wow...
Re: (Score:2, Informative)
And NOBODY understands it! (if you don't believe me, try extracting useful information out of SEC EDGAR). The best I've seen so far just extracts tags using regular expressions.... not actual XML parsing.
Re:follow the money (Score:5, Funny)
XML is like violence. If it doesn't solve your problems you are not using enough of it.
I'm starting to think that saying came from the government.
Re:follow the money (Score:5, Insightful)
It definitely explains the problems they've been having.
But this is only part of it. Check out this diagram. [nrcc.org] I'm no expert, but look at all the government systems in the upper left that Healthcare.gov is supposed to communicate with, in real time. And on the right, 50 state Medicaid systems. And at the bottom, all the insurance companies. Far, far simpler IT projects have failed. This site will not be ready by the end of the month as promised, and there is a good chance that it will never work as planned.
Re: (Score:3)
Did you notice the box in the lower right that shows they have a plan to manage the serfs?
Where do they come up with these things?
Re: (Score:2)
The chart doesn't show "employee or consumer" connecting to the portal / web site, instead connecting directly to the administration interface, customer service, and financial services (last two of which, based on the gradient, I assume have shared responsibilities between carriers and states, which implies another unspecified interface..).
Maybe they just didn't think the web portal was important, because the consumers would hook directly (via implanted electrodes with network port? I assume that would be
Re:follow the money (Score:5, Interesting)
The chart's hosted on the National Republican Congressional Committee's website. I would take it with a heaping tablespoon of salt, if I were you. It's say to say that chart was designed to look as scary and confusing as possible.
Re: follow the money (Score:3, Insightful)
XML soo sucks for db applications.
Re: (Score:2)
And you base this on what? Did the spagetti-monster tell you JSON was technically a better fit, or that XQuery doesn't work?
Re: (Score:2)
I don't mind XML for stuff like Docbook ... but for databases what's the point? XML is an editing format with some down and upsides, using it as a storage format for data never edited in XML is just retarded.
At best it just provides a useless intermediary format which doesn't affect usability ... at worst you try to use XQuery to actually perform searches on large volumes of data.
Re: follow the money (Score:5, Insightful)
In fact, all of those would suck donkey balls. That is because XML, JSON and XQuery are all *data-interchange formats*. Using them to run your database internals is like using a screwdriver as a hammer.
Re: (Score:2)
The only reason people store non key field data in one big XML blob is so they can add fields without having to go through useless, ossified DBAs.
I'm not saying all DBAs are useless and ossified, I'm saying using XML blobs is a workaround for slow, control freak DBAs.
After this is done, others work with the databases and continue the practice without ever knowing it's root. In some cases the DBAs gain control of the XML schema and the circle is complete.
Re: follow the money (Score:5, Insightful)
Re: follow the money (Score:5, Insightful)
Three days restoring is too long too. Unless your server is remote and upstream is super slow.
This is becoming more commonplace as companies are outsourcing or contracting out their admins and development.
But none of these things should give you permission to be a control freak. You are responsible for keeping the databases up, yes. But you are not responsible for everything that happens to them. Sometimes your dbs just get screwed up and you'll have to fix it. Just like sometimes business wants things done to the site/app that will totally jack up the fung shui i had going on in code. Nobody is purposefully trying to screw anyone over. We all want the business to grow and succeed. People make mistakes and shit happens. There's no need to also be an ass to each other on top of it all.
You'd think that, but when it's your job on the line because fucktards in management have 0 clue and assume that because it has something to do with the database and you are the admin, it's your fault. And having been subject to intentional sabotage, and seeing it done numerous times, it exists and happens.
Re: follow the money (Score:4, Interesting)
For those wondering about the link between MUMPS and government healthcare, my vague recollections from years ago when I worked as a developer for health + insurance software: the old MUMPS language included its own looks-like-all-in-memory database system (essentially just a recursive map of string to object, either a value or another map -- the JSON comparison is fair) which made serialization simple. The language got used to build some early health IT systems, including the one for the VA (VistA) and its IHS derivative (RPMS). That stuff's available for free, by the way, through FOIA. The projects have sufficient inertia that they still use the same data-store (at least at the API level). InterSystems Caché, for example, is a MUMPS-compatible database with some relational features (and SQL parsing) thrown on top. They bill themselves as post-relational, but yeah, it's a network database pretending to be a relational database.
It kind of makes sense to continue using network databases for health data -- in a privacy-conscious world, it's not insane to isolate patient data into a document-oriented storage system, because you're not planning to relate data willy-nilly. We were somewhat frustrated that the HL7 interchange format tended to assume things were hierarchical, where we had seen potential graphs and coded for them -- but nobody wanted our better-related data. They prefer to re-enter the data in each place, and prevent things from being synchronized -- it protects the data from unexpected changes. So if all the systems and agencies you're integrating with have this attitude anyway, and you're constantly worried about data-interchange, I can see how you might come to the conclusion that a document-oriented, XML-backed storage engine would be a good idea.
Not the entire problem. (Score:2)
It's even more basic than that. According to TFA the design goal was: "creating a cutting-edge website that would use the latest technologies to dazzle consumers with its many features. "
In other words, even if they had gotten this thing up and working as they wanted, right on time, it would have been an accessibility nightmare that would never work unless you have a brand new computer configured with no security anyway.
No. (Score:3)
The problem is this:
Government officials like issuing "cost plus" contracts because they allow them to remain in control of the work while not doing any of it themselves. They probably wanted to use it because they had used it in other projects in the past. If the'd just issued a fixed pice contract and said "get it done" the contractor would have select
Re: (Score:2, Interesting)
Because relational dbs are old and stuff. We don't tie onions to our belts anymore either, grandpa. NoSQL and XML are the future! A hipster I met at Starbucks told me so.
Re: (Score:3, Funny)
Was this hipster drinking coffee or serving it?
Re: (Score:2, Insightful)
Never attribute to malice that which can be explained by incompetence.
To me, this sounds like a pretty open and shut case of "Hey, I've heard that these 'NoSQL' database thingies are trendy these days. Let's use one of those!"
There's a difference between using fun, exciting new technologies and learning something new while doing that... and doing a project which stays in schedule and budget, is based on technology you already know thoroughly, and on which people's lives can depend (well, indirectly).
Re: follow the money (Score:3, Insightful)
You're exactly right! I've done tons of "exploratory" coding over the years myself, either using some new techniques or new products, just for the fun of it or to learn something new. But that would always be on small, low visibility projects where the consequences of potential poor performance or other issues would be insignificant. To tread new ground on something so big with national visibility is foolish. You'd want the most well established and known reliable and performant tools and techniques possibl
Re: (Score:2)
There has got to be some reason that this DB that ive never even heard of (and i work with DBs, its not my main point of work but I know my way around DBs) got the gig over the more established players.
or, perhaps they went with it because it is less known and therefore reduce the risk of known attacks in other DB systems?
Perhaps either is the case: Obscurity or donations... A third option is that it provides some feature that doesn't exist in Oracle/SQL Server/Cassandra. However in digging into their web-site it looks like it is some sort of wrapper/hybridized product that mates to Hadoop, which would make sense given the vast volume of data you're talking about managing for the Federal exchange, which I believe services 36 states.
But with that said, I've never heard of it either.
Re: follow the money (Score:5, Insightful)
I used marklogic when I worked at a previous job and after learning how it worked and understanding it better, it made our jobs incredibly easy. It just had a serious learning curve.
Marklogic is a nosql db, that uses XML for its object format and xquery for its query language. This thing is NOT mongodb. It actually works really well and allows for complex data modeling with the ability to do joins and have transactional isolation in making changes to the data as well as a really solid content processing framework with pipelining and all that jazz.
Now, I can't imagine a reason for using marklogic, or any non-relational db for a project like this. The only clue is that marklogic has a lot of government contracts; mostly for the military. So maybe that's why it was used. But the fact that they chose a database system that they weren't experts in for a project that had so much visibility speaks volumes on how mismanaged this whole project was.
Re: follow the money (Score:2)
The only agencies that would need NoSQL type support are the ones spying on us. My take is that the insurance industry said they would only communicate in EDI-XML formats and somebody in management though that meant buying an XML-based system to be "compatible".
Re: (Score:2)
This thing is NOT mongodb. It actually works really well and allows for complex data modeling with the ability to do joins and have transactional isolation in making changes to the data as well as a really solid content processing framework with pipelining and all that jazz.
So is SQL attached to any reasonably good content management framework.
Re: follow the money (Score:5, Insightful)
To me, the fact that it was a major problem is an indicator of bigger problems. With systems like this, the database should be just a simple repository for persisting an retrieving domain models. If they want to do any non-trivial reporting on it the data should be replicated to a reporting repository. Treating the database a a simple persistence repository makes most database operations trivial, and better yet, decouples the database from the business, meaning that if you couldn't even manage the simple operations, replacing the layer is fairly trivial.
The problem is when people think the database is the starting point for a system, not just a tool to be used to support what your business logic is doing. It sounds like that's what happened here.
Re: (Score:2)
I should add that with many languages and frameworks (Java & Spring/Hibernate for example), "we have machines for that". Only the fine tuning and performance tweaks need to be done my hand. Manually coding an data access layer is almost a "make work project" that's frequently a source of errors in what should really be managed by configuration.
Re: (Score:2)
If a database is good, it should be possible to use it for ALL problem domains.
Of course, by this criteria, a relational database is NOT "good".
But I would have thought an object database is good, and would work in any situation. Unfortunately, they never seemed to really take off.
Anyway, I'd rather be using an object database for complex data of ANY kind than some crazy XML database. I can sort of see why a health application that might have tons and tons of fields might be tempted to use an XML storage, b
Re: (Score:2)
Re: (Score:3)
http://www.marklogic.com/what-is-marklogic/marklogic-hadoop/ [marklogic.com]
This seems like the relevant, interesting/crazy part. Whether it's the right choice or not, I leave to more seasons database professionals than myself.
Re: (Score:3)
"XXX"?
You mean "Obama and other democrats", right? Because this system is their baby.
It's no mystery who "XXX" is; no need for a variable.
I've noticed a Responsibility Diffusion Field around this whole thing ...
Re:follow the money (Score:5, Informative)
I'm guessing they were lowest bid.
Bids? What bids?
There was no bidding.
http://nypost.com/2013/11/01/obama-donors-firm-hired-to-fix-web-mess-it-helped-make/ [nypost.com]
----
"WASHINGTON â" A tech firm linked to a campaign-donor crony of President Obama not only got the job to help build the federal health-insurance Web site â" but also is getting paid to fix it.
Anthony Welters, a top campaign bundler for Obama and frequent White House guest, is the executive vice president of UnitedHealth Group, which owns the software company now at the center of the ObamaCare Web-site fiasco.
UnitedHealth Group subsidiary Quality Software Services Inc. (QSSI), which built the data hub for the ObamaCare system, has been named the new general contractor in charge of repairing the glitch-plagued HealthCare.gov.
Welters and his wife, Beatrice, have shoveled piles of cash into Obamaâ(TM)s campaign coffers and Âapparently reaped the rewards.
Beatrice Welters bundled donations totaling between $200,000 and $500,000 for Obamaâ(TM)s campaign during the 2008 election Âcycle, according to campaign- Âfinance data compiled by Center for Responsive Politics.
SICK CALL: Obama bundler Anthony Weltersâ(TM) firm owns the company picked to repair the health Web site.
The couple then became top donors for Obamaâ(TM)s inauguration festivities, kicking in $100,000 out of their own pockets and bundling another $300,000 from friends and business associates, according to the center.
The investments quickly paid off for Beatrice Welters. The Obama administration tapped her in 2009 for the plum job of US ambassador to Trinidad and Tobago, which she held through last November.
The couple have been frequent guests at the White House.
Visitors logs show at least a dozen visits between the two by the end of 2012, the most recent information available.
The entire Welters family has gotten into the donation game.
The Welters, along with their sons, Andrew and Bryant, have contributed more than $258,000 to mostly Democratic candidates and committees since 2007.
Whatâ(TM)s more, UnitedHealth Group is one of the largest health-insurance companies in the country and spent millions lobbying for ObamaCare.
The insurance giantâ(TM)s purchase of QSSI in 2012 raised eyebrows on Capitol Hill, but the tech firm nevertheless kept the job of building the data hub for the ObamaCare Web site where consumers buy the new mandatory health- Âinsurance plans.
QSSI has been paid an estimated $150 million so far, but officials couldnâ(TM)t say how much more the company might collect on the Ârepair contract.
By all accounts, the data hub has run smoothly while many other components of the Web site have failed.
Meanwhile, tempers flared among Obamaâ(TM)s Democratic allies over the disastrous rollout of the presidentâ(TM)s signature initiative.
âoeIâ(TM)m extraordinarily frustrated,â said Sen. Jeff Merkley (D-Ore.) Âafter top Obama-administration officials gave Senate Democrats a private briefing on the state of the Web-site repairs.
He said they were losing confidence the site could be quickly fixed.
âoeI donâ(TM)t think thereâ(TM)s confidence by anyone in the room. This is more of a show-me moment,â said Merkley."
----
This thing was never meant to function in any event. They didn't pay attention to the details because they didn't matter.
It was designed to fail so that the "Holy Grail" of single-payer government-run healthcare could be rolled out as a "fix". As a bonus, they get to hand a big bag of taxpayer's money to their "bag-man".
Just watch. Single-payer will be the "fix" insisted upon.
At least we won't have so many foreigners coming to the US for our excellent health care any more.
Strat
Re: (Score:3)
Yes, we're talking about huge players and huge deals.
No, we're talking about an Obama campaign contribution bundler getting a sweet payback at taxpayer expense to create something that's not designed to work in the first place. Not that that kind of thing is out of character for this administration [cough]rape-A-scan[cough].
But I'll take your bet that this is all a conspiracy to bring about single-payer healthcare. You've made a prediction, now we'll sit back and observe you being wrong.
What conspiracy? It's no secret. Here's one of the architects of Obamacare openly stating it, in a manner and tone suggesting that anyone would be stupid to NOT believe Obamacare is a path to single-payer government health care.
Jacob Hacke
Noobs? (Score:5, Insightful)
FTA: "An initial assessment identified more than 600 hardware and software defects — 'the longest list anybody had ever seen,' one person involved with the project said. "
Strikes me as none of these people seemed to have ever worked on large projects before.
Re: (Score:3, Funny)
You don't understand. That's not "600 bugs in the software" that they are talking about. "Thousands of bugs in software" is just one of 600 items that was a defect. Others included "Not enough servers", "Data connection to limited in capacity and speed", and apparently "No hablo espanol".
Re: (Score:2)
Re: (Score:3, Insightful)
NIH syndrome (Score:4, Insightful)
You could take a handful of proven DB technologies such as Oracle/DB/MSSQL, throw a web (Apache/IIS) and app (.Net/WAS/Jboss) front end to it, and it would work. Why did these guys fuck up the whole thing? It's like the scene in The Fountainhead when the second-rate architects smash up the plans and add their own stuff, "to express their own individuality". This could have been a solved problem - hell, it WAS a solved problem.
Re:NIH syndrome (Score:5, Insightful)
Why did these guys fuck up the whole thing?
All actions by Federal government players seem rational when viewed from their point of view.
The problem with most analysis, here or on other sites across the web, is in the assumptions. Specifically, if you assume that the government actions are for some benefit to the people, or efficiently completes some task (which is assumed to benefit the people), then nothing they do makes sense. That assumption makes for good outrage which can attract readers, but it doesn't answer the question of "why".
When government actions are viewed in light of more obvious motivations, everything it does makes sense. You start to see them not as an pack of incompetent, bureaucratic screw-ups, but as a self-sustaining, self-absorbed, engine of corruption.
I'm not aware of *anything* the federal government's done in the past two decades that's been in the interests of the people. All the useful stuff, the stuff you would want to keep across a reboot, was several decades ago.
I don't know the specifics of why an obscure database was chosen over the obvious players, but the reason wasn't "best utility for the people". Look for another reason.
Re:NIH syndrome (Score:5, Insightful)
I would actually put forward that the problem is not the government, our government is pathetically weak in many ways. The problem are all the private groups flexing their muscles and making the government dance. Most of the people I have worked with on these kinds of projects are good people who just want to get stuff done, but they are weighted down by a nearly impossible set of requirements, many of which are mutually exclusive, dropped on them by outside groups who mainly are interested in showing off their power to their own people and have no investment in the project itself being successful.... and often have an active interest in the project failing since they can always use that as fuel for more personal power.
Re: (Score:2)
Surely it's more due to lobbyists corrupting the anti-corruption measures demanded by taxpayer groups.
Auth problem - Oracle Identity Manager (Score:3)
We were looking at what we could figure out about the architecture of healthcare.gov, and one problem is that it looks like it's using Oracle Identity Manager to manage the permissions of what users can/can't see. That means that OIM is burned in - and it's probably brutally slow, since every time you need to check a permission you go through OIM.
I'm not positive that's the case, but it fits given what pieces of the architecture I've seen. It would also explain why the system doesn't perform - permissions c
Re: (Score:2)
It is a solved problem if and only if you limit database operations to today. Then you can use today's best software and practices.
But this is too big a system to be taken through the revisions that are needed every 7 to 10 years to keep a DBMS up to date. For one thing, once it is in full operation it cannot be taken off line for a weekend upgrade. That adds a lot of headaches as you can count on always having to support simultaneous database operations by two or more different versions of software: the p
Re: (Score:2)
You hit the nail on the head. There is just so much expertise at building a mainstream Web stack and backend database, that it is almost acting in bad faith not to take advantage of it.
Then there is the security aspect. You can throw money to buy the skills needed at Oracle to have it locked down. Same with Windows, Linux, Solaris or AIX for the OS, Apache or IIS for the webserver, and a backend platform of choice.
Using something so off the beaten path may just mean that something can't be done on the pl
Re: (Score:3)
Can someone explain to me how, for a project like setting up exchanges for Obamacare, NoSQL database systems is a rational choice? There are SQL-based systems like Oracle and MSSQL that can certainly handle recordsets of that size,and with that level of activity, and give you dialects of SQL sufficiently close to the norm that anybody with a reasonable level of RDBMS experience should be able work with it.
Re: (Score:3)
Well, the obvious answer is that someone involved has goals other than the site succeeding, such as lining their pockets or even outright sabotage. After the government shutdown fiasco, the latter sadly can't be ruled out; the US is basically at Civil Cold War at this point.
MarkLogic = NoSQL (Score:5, Interesting)
Re: (Score:2)
But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?
Re: (Score:2, Funny)
They should have used MongoDB, because MongoDB is webscale.
Re: (Score:2)
AKF cube.
Healthcare.gov isn't a particularly difficult scaling problem to solve. You can 'swim lane' the hell out of it.
It's not like it's a social network where people need to be constantly sharing things and an arbitrarily large (potentially huge) group of people need to be notified of everything each of them does/doesn't do...
Re: (Score:2)
But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?
I don't know if you are kidding or not but I'll try anyway.
The US population might hit 1 billion inside the period where something form of the ACA is intact in a recognizable way. Unlikely but its possible. Using partitioning schemes it should be fairly simple to scale on traditional database technologies because for most queries you don't need to know about Abby to handle a transaction regarding Bob. There are some where you do but even those are mostly simple 1:1 update operations, you might need to
Re: (Score:2)
It would scale just fine, assuming you make proper indexes for your queries.
Since the general use case is probably gonna be something like "pull up records for patient that is sitting infront of me and edit them", I think an index lookup + atmost a couple of dozen records pulled from related tables will be handled just fine.
Any other use cases that involve statistical analysis on all data can be handled by copying it to specialized DB's every few weeks.
Re:MarkLogic = NoSQL (Score:4, Informative)
Re: (Score:2)
You make little sense there. By the time there are a billion people in the US we'll either be single-payer like the intelligent, civilized world has, or we'll go back to "no money? Let 'em die."
International incident in the making (Score:4, Insightful)
But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?
Current population of the U.S. is a little over 300 million. That includes children, people who have company provided health insurance, etc. who don't need to access HealthCare.gov so that the number of users of HealthCare.gov is expected to be about 30 to 45 million people.
The only way HealthCare.gov would need to support 1 billion or more users would be if we inflicted it on the China or India. That could lead to war or poor technical support and customer service. Although the later does raise an intereting question of who does someone at a call center call when they need technical support? Likewise, the population of the planet is between 7 and 8 billion. For the 1,000 billion users are you planning on also providing health insurance to E.T?
Cheers,
Dave
Re:MarkLogic = NoSQL (Score:5, Interesting)
A customer at work has a MarkLogic database. I don't know what version it is, but I assure you IT IS HORRIBLE. It's like... an XML database or some shit. Just awful, and extremely confusing to use.
A couple years ago I was asked to do automatic database backups. The only documentation I could find for backing up the database requires logging into a web interface I had no idea existed on an obscure port and clicking a backup button. I literally had to write a perl script to fake a browser and do this just so they could get a database dump.
Do not use this product. Please.
Re: (Score:2)
Oh, but they claim to be ACID complaint: http://www.marklogic.com/blog/can-you-pass-the-acid-test/ [marklogic.com]
lol
Re: MarkLogic = NoSQL (Score:5, Interesting)
Marklogic, afaik, is the only acid compliant nosql solution that exists.
Re: (Score:3)
Marklogic, afaik, is the only acid compliant nosql solution that exists.
Neo4j is an ACID-compliant graph database (so it's NoSQL, but that word is a bit pointless).
It's also the most useful non-relational DBMS: relationships become first-order entities, having a type and properties.
http://www.neo4j.org/ [neo4j.org]
Blow to NoSQL movement (Score:5, Insightful)
And here is a fantastic example of what happens when hype trumps common sense. NoSQL is the new hawtness, and apparently the dumbasses running the project wanted to be part of that. Now MarkLogic, and NoSQL in general, will have a massive blow to their reputations, and it's unknown how badly this will hurt them.
As someone who has done databases for a long time, I have very little respect for NoSQL, but that is mostly because everyone keeps trumpeting it as a killer of traditional databases. There are scenarios where NoSQL systems are an ideal fit. However, NONE of those scenarios require data to be very reliably stored in a guaranteed and predictable way.
If you don't get your tweets or your friends facebook posts as soon as they are posted, no one will really care. But for something as truly important as health insurance coverage? Are you f__king kidding me? And that's just from a reliability standpoint. Nevermind the fact that NoSQL is currently at the wild west stage where nobody is compatible with anybody else, there is nothing resembling a standard set of APIs between products, making it very difficult to develop expertise.
Sounds like the Gov was just begging for problems.
Re: (Score:2)
Just had this debate with a current project with some wanting to use a NoSQL solution for the whole thing. Problem is most of the data is relational and I stuck my foot down and said we're using PostgreSQL for anything that needs to be retained. That mean users accounts & transaction records, and really all the data is relational.
Now there are other elements, like the chat system and distributing JSON strings to large numbers of persistent clients that seem a perfect fit for a NoSQL database. Since t
Re: (Score:2, Interesting)
The agency imposing their "unconventional" choice of DB, and this sentence:
"They ordered CGI technicians to drive from their offices near Dulles International Airport in Virginia to the agency headquarters near Baltimore to review their code with government supervisors."
sum up everything that went wrong.
Basically you had a bunch of arrogant idiots who thought themselves experts (which they may well be, _in the healthcare domain_) who not only ignored the views of the _software/IT_ experts ("technician" my a
Re: (Score:2, Insightful)
NoSQL predates SQL because we were storing stuff back then to. Relational databases are nice for consistency, but you sacrifice a lot of control and speed improvements for that consistency. A good NoSQL database and a good NoSQL programmer will always outperform SQL based applications. However, it means you need to have programmers capable of operating at that level AND you need to have solid strongly enforced conventions.
I also think that you are largely misreading the problem. Being able to get a quote an
Re: (Score:2)
Everything you say is true. However, they didn't call it NoSQL back then. Back then, they were called 'proprietary data stores', all of which ultimately became utterly inflexible in how they dealt with data because the products would be originally designed to satisfy one problem (or set of problems), but then when a new requirement came down the line that needed to manipulate the data in a different way, you had to go through contortions trying to make things work.
That was the whole reason SQL systems wer
Re: (Score:2)
Can someone please upvote the parent post? It's actually very interesting, and it's worth reading about this MUMPS system. I had never heard about it myself and it looks interesting.
Re:Blow to NoSQL movement (Score:4, Insightful)
Re:Blow to NoSQL movement (Score:5, Interesting)
I ran into this recently at a company whose new head of engineering was talking to me (an outsider) about the technology problem they had to solve and I thought it sounded very traditional and simple except they'd need to carefully plan for horizontal scaling.
Basically a potentially huge number of devices (in the range of millions) would be reporting in periodic data that had to be stored and potentially evaluated in real-time. The data was quite easily swim laned by geolocation and the data had no appreciable inter-related significance. So basically, one piece of data from one device had nothing to do with any other device's information except in the general sense that can come from a more heuristic correlation of their data.
I should mention that the new engineering head and I had already (together) handle a situation very similar to this at a previously successful software company.
Well, the new engineering head had inherited an external architect who had different ideas. All of these different ideas involved things like Cassandra over Hadoop, AMP/Spark, BDAS. He showed me a diagram of the technologies he wanted to integrate and I'd never heard of almost half of them (and I deal with scaling issues all the time), this diagram had about 15 different technologies stacked together. It was crazy - all to solve a relatively simple data volume problem.
Almost needless to say, I advised otherwise, and afaik they're going the bid data way because it will make it easier for them to pull in VC money since shockingly few VC's actually evaluate technology before they put money in (I do this for VCs also, and other VCs wonder how I get paid to do this, lol.)
Re: (Score:2, Insightful)
> MarkLogic's NoSQL managed to land a giant lucrative contract for the venture capitalists ...
+24 insightful and informative. This.
Here's another angle on it. Haven't you ever wondered why everything associated with the government takes longer and costs more?
I once worked with a guy who did government contracts all the time. He said, "you deliberately underbid* to make sure you get the deal. Just be sure to put a clause in there protecting yourself when the government (which is Rule By Committee, remembe
Re:Blow to NoSQL movement (Score:4)
MarkLogic is an XML repository, not a RDBMS (Score:5, Interesting)
"Some people, when confronted with a problem, think 'I know, I'll use XML.' Now they have two problems."
-JWZ
MarkLogic is an XML database, not a relational database, so if your data primarily consists of XML content then it's the right tool for the job. Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.
MarkLogic itself is fine if your data fits neatly into an XML schema, but with healthcare.gov that tree is probably enormous and hard to optimize for DB activity.
Re:MarkLogic is an XML repository, not a RDBMS (Score:5, Insightful)
Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.
It was the Medicare folks that imposed MarkLogic over the objections of the prime contractor. Calling Medicare a "vendor" is a bit of a stretch. Like non-IT staff calling themselves "customers" of IT in a corporation, as if they have a choice of IT departments.
Another sore point was the Medicare agency’s decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.
The CGI argument was that it would slow work. I have absolutely no doubt about that. Every NoSQL system is it's own distinct thing with unique APIs, protocols, conventions, quirks, etc. This is fine when you are dealing with a limited group of good developers that can be expected to rapidly climb a learning curve. This is not fine when you are the top contractor in a herd of contractors, populated by a vast number of mediocre programmers that last learned anything new in the late 90's.
In that case the correct choice is to select the most ubiquitous technologies. A boutique NoSQL XML-base is not appropriate, even if it is excellent, which MarkLogic may be. I don't know. I just know CGI was 100% correct in objecting to it. CGI's fault in this case was its failure to win that argument.
Not that it matters much. The project was doomed in any case — political agenda driving pie-in-the-sky requirements using crony contracting — Fucked at birth, as they say.
How cait it be the problem (Score:2, Insightful)
Open Source (Score:3)
Re:Open Source (Score:5, Insightful)
Impressive (Score:2)
US Federal Government Systems Projects (Score:5, Interesting)
Unnecessary complexity is actually bad in itself (Score:3)
"The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project."
Unless that complexity was necessary to solve a problem, then it is in fact bad.
It is not the tool, stupid (Score:3, Interesting)
It is NEVER the tool, but who and why it was chosen.
Furthermore, MarkLogic is a legitimate NoSQL vendor with strong ACID.
So the question is "how does MarLogic screw up the site", without the answer to that question, we should all refrain from pointing finger to merely a small piece in a huge software project.
NoSQL is NOT the new hotness, it's been out there for at least 5 years and many successful projects are using them, so for the ones that things NOSQL shouldn't be used, wake up and breath some fresh air.
CGI "officials" (Score:2)
Computer time..where does it all go? (Score:2)
I get there must be tons of complexity in managing healthcare.gov site interacting with all necessary stakeholders... Must be quite a lot of different databases, systems, operators to say nothing of complexity of working the actual problem space. I can understand how there might be glitches that cause wrong rates and plans to be communicated.
What I am still puzzled by are the "waiting rooms" with 40k people waiting to use a site. What the hell can justify it being so computationally expensive to spit out
Nothing wrong with the tech. (Score:4, Interesting)
I've used it, personally, to implement a public-facing website. That site endured the dreaded 'slashdot effect' several times. No failures.
When implemented properly, Marklogic is damn near unkillable. It will slow down, it will reject connections when queues are full, but it will not fall over. Naturally, this assumes proper underpinnings and capacity calculations. With Marklogic, those are actually documented.
Mandatory disclosure: I do not have and have never had any association with Marklogic other than a paying customer.
Re: (Score:2, Informative)
It was designed from the start to pave the way for single payer health care.
Re:Stop with the excuses. (Score:5, Insightful)
Bullshit. Since the law has been passed almost the entire implementation has been up to the administration. Which in case you had not noticed is lead by partisans in favor of the affordable care act.
If the GOP can be accused of anything material in terms of interfering with the implementation it would preventing the law from being amended, and if it needs amending there is your proof it was bad law to begin with. Sorry Obama owns this and really HE owns it not even the larger DNC as its HIS branch of government that has been responsible for implementation from the moment HE signed the thing.
*IF* it fails its either because it was unworkable in the first place as written and should not have been enacted or because the Obama administration failed to execute. Those are really the only honest high level conclusions.
The argument well because they had to pass the Senate bill as is it did not get the usual fixes and tweaking is why its got so many problems is also bogus. They had to pass the Senate bill by abusing the budget reconciliation process to deny the minority party in the House at the time its rights; they knew if usual procedures were followed it would never have become law. So again it either should not have been enacted or the administration has failed to execute.
Re: (Score:2)
Since the law has been passed almost the entire implementation has been up to the administration.
The implementation is, for the most part up to the administration and they certainly have screwed the pooch on this; however, the GOP can quite easily be accused of willfully sabotaging the process through the ACA lawsuits and intentional state exchange shortcomings (not to mention a myriad of other issues that relate to the shutdown, refusing ancillary funding out of spite, and issues with the Supreme Court.)
In any case, even if the GOP are being total jerks about the law, healthcare.gov requires a softwar
Re: (Score:2)
GOP can quite easily be accused of willfully sabotaging the process through the ACA lawsuits
Attempting to sabotage it maybe, succeeding no. There was never even so much as a stay issued, as far as I am aware, preventing the administration from getting started on implementation. If I am wrong on that please correct me but I don't think the court cases could have had much impact on the implementation.
The medicare expansion issue is going to create a coverage gap, in the states that did not do it. That is about the only *problem* the legal challenges successfully created.
As far as states not impl
Re: (Score:3)
Attempting to sabotage it maybe, succeeding no. There was never even so much as a stay issued, as far as I am aware, preventing the administration from getting started on implementation
Sorry, but even as an independent I would have to disagree with that. The statute provided monies for states to create their exchanges, there was no stipulated money to fund a federal exchange because it was entirely expected that states would want to control this themselves.
Suddenly GOP led states started saying "we're not building that" which makes absolutely zero sense except as a political stratagem to sabotage the process. You should note that the only two non-GOP states to leave the burden entirely
Re: (Score:2)
To be fair, the AC didn't specify Republicans and Democrats. It could have been internal groups behind the website itself that wanted to blame other internal groups, to pump up their own importance. That could be what he meant.
.
But, yeah, he almost certainly made a laughable dig at the GOP there.
Re: Stop with the excuses. (Score:2)
Uh, no the Republicans do not have clean hands in this foaled rollout.
By refusing to set up the state exchanges they shifted a massive burden to the Feds unplanned for in the law. Then, they refused to expand funding for this massive integration effort. Think about it... Most of the state exchanges are doing fine. Why? Because they only have to meet one state's requirements. The Feds were forced by Republican tactics to handle the differing requirements for 30+ states.
Re:Taxes... (Score:5, Insightful)
Re:Taxes... (Score:5, Informative)
Also the working poor - 47% pay no _income_ tax, but more than half of those people do pay payroll taxes, and even many those who don't (after deductions) still work.
They also pay sales tax, property tax, etc etc.
Re: (Score:2)
Everyone pays taxes, income or not. Buy something? Sales tax. Pay utility bills? Taxed. Buy gas? Taxed.
Re:Taxes... (Score:4, Informative)
Sorry to break it to you but that "statistic" is a bunch of BS. Everyone in the US pays taxes of some sort. I'm not particularly high on the tax scale (about $36k a year) and I estimate that i pay at least 30% of my income in local, state, federal and specific taxes (sales, gas, property, vehicle, etc). MAYBE some very poor (or very rich) people can avoid some of those taxes but EVERYONE pays at least a few of them.
Re: (Score:2)