Ask Slashdot: Should Developers Install Their Software Themselves? 288
Paul Carver writes "Should developers be responsible for installing the software they develop into production environments? What about System Test environments? I'm not a developer and I'm not all that familiar with Agile or DevOps, but it seems unhealthy to me to have software installs done by developers. I think that properly developed software should come complete with installation instructions that can be followed by someone other than the person who wrote the code. I'd like to hear opinions from developers. Do you prefer a workplace where you hand off packaged software to other teams to deploy or do you prefer to personally install your software into System Test and then personally install it into production once the System Testers have certified it? For context, I'm talking about enterprise grade, Internet facing web services sold to end users as well as large companies on either credit card billing or contractual basis with service level agreements and 24x7 Operations support. I'm not talking about little one (wo)man shops and free or Google style years long beta services."
I'm of two minds about this (Score:5, Insightful)
My answer is mostly 'no'. The only benefit I see is that developers will become more motivated to have a simpler and better installation process. And that's a pretty nifty benefit.
But I've been on both sides of the fence. I've rarely been a developer who did anything less than thorough testing before declaring something 'done'. But I know that I'm an incredible rarity in that way. And on the devops side of things, the less ability developers have to push things, the more likely decent QA will get done before stuff ends up in production. But developers frequently also give you installation instructions that are unrepeatable special case installs with rollback instructions that make no sense.
I think that one good way to balance this is to have a preliminary test environment into which developers are allowed to push things. They are given limited rights to this environment, basically just the ability to upload some software and run a deployment script. This encourages them to write functioning deployment scripts. But it prevents them from shoving things into production because it just 'has' to go out today and it's such a small, low-risk thing. Of course it'll work!
Re:Why not use tools that help do it? (Score:5, Insightful)
This question was not directed as consumer application. It is direct at Enterprise applications. InstallShield just won't do it.
Developers should not install it. Nor should they help install it. If the Configuration Management team cannot do it themselves, then they need to send it back to the developer for better packaging or instructions.
This is to the developers benefit. When new environments are set up, they shouldn't have to contact the developer to deploy the application to those new environments.
And as always. Titles with questions are typically answered with a "no".
My persepctive (Score:5, Insightful)
As a systems administrator, nothing frustrates me more than when a developer sends me an e-mail that says "install this".
First, they do not always say what the software is supposed to do, so I cannot prepare for any security requirements. I am rarely told if it needs a port opened, I have to check the security logs to see if the software is trying to communicate through the network.
Second, while I may have the ability to fix their software, I prefer not to mess with their code or configuration. Since I may not know what their software is supposed to do, I may get it running I do not know if it is operating properly.
Third, if you are asking me to install alpha or beta versions on a live system, it's usually a bad idea. I have no problem installing it on a test server or a VM, but I hate putting it on a production box.
Developers shouldn't have production access (Score:5, Insightful)
Title says it all - giving a developer access means they can deploy undocumented "hacks" and "quick fixes", usually meaning to document and normalise them later on.
Forcing the, to hand off installation and maintenance to a second team means documentation is enforced, standards are enforced and quick fixes are better vetted.
For the record, I wear both hats in different situations for different clients - as a developer I don't care about the production environment, and I like it that way. I care about the bugs the production environment uncovers, so the UAT environment should be identical, but even then the developer shouldn't have to care about it - that's the systems admin teams job.
Seperation of duties (Score:4, Insightful)
If it's enterprise class, you have a team responsible for the stability of the servers (support) that is not the same as the developers. Sometimes, you even have a team responsbile for just deployments (depends on size of the org). The developers should have access to install in DEV, but not to TEST. This allows you to test your deployment process (and backout process) before you touch your PROD environment.
http://en.wikipedia.org/wiki/Information_technology_controls [wikipedia.org]
Even if not financial in nature, one of my former employers lumped it all unde SOX compliance.
I used to think yes, but not so much (Score:5, Insightful)
No. (Score:4, Insightful)
-Development happens where they are able to test using a test environment
-Developers hand off everything to the System Admin (SA) who will install it. They then install it on a test environment as well.. If there are issues found work with the developers to solve the issue, correct the documentation and proceed to step 3.
-Install in production.
The only issue with this is step 1 and 2 can sometimes become filled with accusations. SAs think the product sucks and Developers think that the SAs are idiots who need everything spelled out for them. It becomes a lot worst when the developers are contracted out (which is common). This needs to be avoided, both parties should see themselves as working together to create a better product.
yes but there are exceptions. (Score:4, Insightful)
Creating the flawless installers costs time.
Time means money. If the software is only installed twice (on test system and production), a good flawless installer is very expensive. If it is installed a lot of times (more than 5 systems), automation of the installation will pay itself.
Generally it is good practice to keep the developers out of the production environment.
However there must be exceptions.
-Emergency fixes.
-If there is no good team of maintainers, you can actually look at the installation logs, and understands them, the developer might be the better option. A good maintainer, who only blindly runs a script is not the best option.
-Uptime might be more important that the principle.
Possible issues with letting Devs do this (Score:4, Insightful)
Re:Why not use tools that help do it? (Score:5, Insightful)
Developers should not install it. Nor should they help install it. If the Configuration Management team cannot do it themselves, then they need to send it back to the developer for better packaging or instructions.
As a (web) developper, I strongly agree.
I'll just add that the Configuration Management team should have some knowledge about the software and the environment they manage.
I've often seen software come back because sombody did'nt have a clue what their job was. ("prerequisite: apache 2.x" should be enough for anyone: I don't have time to write a doc about how to install standard software, especially when I don't know the target server configuration)
Re:Why not use tools that help do it? (Score:2, Insightful)
Yes,but not into production (Score:5, Insightful)
You should have at least 3 environments beyond the personal ones the developers use to develop and unit-test code: development common, QA and production. Development common should be where dev does integration tests, including installation. If developers are responsible for creating the installation tools, they should be doing the installations there so they can debug those tools. If someone else is responsible for writing them, the devs should be working with them to make sure the tools do what the software needs to install correctly. You can't get the installation tools right if you don't test and debug them, and when they interact with the software the developers are the best ones to figure out what's wrong and what's needed to fix it.
Developers should not be doing the installation into the QA environment. They should be handing the installation tools over to QA and letting them run then according to the deployment instructions. That's the only way to confirm the instructions were really complete and that everything works per the documentation, by putting it in the hands of people who didn't write it and letting them deploy it. That way when it comes time to deploy in production you've got some assurance that the deployment will work because it has worked before.
Now, if things do go wrong you need dev involvement. They wrote the software and the tools, they'll often recognize exactly what's going wrong where Ops and QA won't. They're also the ones most likely to be able to give you a quick fix to the problem that'll get production up and running without having to back out and try another day. If you've already invested the time bringing the systems down and deploying the new version, it makes no sense to revert to the old version and waste more downtime tomorrow re-doing the deployment if the only problem is a path in a config file having the version number in it in QA but not in production and a quick edit of the newly-deployed copy of the config file will clearly fix the problem.
Re:Why not use tools that help do it? (Score:5, Insightful)
Of course the devs need to install it into their test environment. And QA needs to install it into their test environment, but Ops needs to install it on the production servers.
Re:Developers shouldn't have production access (Score:4, Insightful)
I've been in companies that practiced it both ways.
Company A) Developers can never ever access production no matter the reason. The end result in that situation was bugs that couldn't be reproduced on the desktop or in the QA environment. The problems went on for months until I had a lucky break of a developer moving jobs into the system admin role of the production environment. When he looked things over he discovered the previous admin had not configured things in production properly. To the point of lying about it when I had sent a previous check list of things to verify. If I had access to the systems the problem would have been resolved in a few days rather than months.
Company B) Developers own the software and hardware from end to end. In my current company we have to package the software up into a deployment system and deploy it that way. However we do have full access to all the systems. Can/do we do hacks and quick fixes? Yes, if the situation warrants it. But in the end it has to get rolled into the official distribution for it to be correct. Can it be abused? Yes. But that is why the culture of the company become very important. In the end you either trust your developers to do the right thing or you don't. If the company can't trust the developers to have ownership of their code and systems.. Well then at least for me I would say I'm working at the wrong company.
FYI, I enjoy working at company B far more than I ever did at company A. Given a choice I will never go back to an environment where developers don't have access to production.
Re:Why not use tools that help do it? (Score:5, Insightful)
I'm a build and configuration manager (and a LONG long time software developer), and I completely agree.
If developers want to employ new (as in industry-wide, or new as in company-wide) technology, they are free to install, configure, test, and prove out the functionality and feasibility of the software on their own machine. But that should be the limit of their reach.
Any shared / integrated environment from QA on up to Pre-Prod and Prod, should be off-limits from a developers' sticky fingers. If there is an installation and / or configuration problem that arises with the application, AND a developer is needed for support, then they should absolutely be called in, but from a CONSULTATIVE role only. I.e., their fingers never touch a mouse or keyboard.
It's common (if not expected) for developers to build an intimate working relationship with the technology they're using. In the world of "familiarity breeds contempt", it means that a developer can overlook something that occurs "obvious" to them, but not to everyone else. How many times has a developer been "called in" to fix an installation problem, and the fix wasn't documented or proved out? (not to say that integrators are saints in this area, but they damn well SHOULD be). Or a developer has access to a test region, and just "hops onto the machine" to tweak a parameter that caused a test suite to fail?
QA (and QC Testers) need to count on the stability of a machine, it's known state. If a test is failing 100% of the time, it should keep failing. Or if it's passing 100% of the time, it should keep passing. Having the parameters of an integration machine change without the knowledge of the QA team (e.g. so that changes can be scheduled in, and updates to the testing suites can happen), then the validity of test runs is nullified, testing costs go through the roof, thus adding to the pressure to "skip the tests" and ship / deploy. Ick.
The instructions / script for installing a package on a machine should be EASILY understood by an installer who is skilled in the practice of software installation, and no more. (ie, not written for a senior engineer, not written for the janitor). Enough information to have it properly installed and configured, some basic troubleshooting, and a clear escalation path should issues not get resolved. Skip the 65 pages of configurable parameters if all the installer needs to alter are 12 parameters on the target machines. but don't skip ANY of those 12. If one is missed, find out why. If 10 extra are there, see which ones are needed for the different regions and skip the rest.
My personal line in the sand is the Developer integration area; that stays with the code monkeys. It's important to be able to test out package installs, and this is the type of machine upon which to do it (which is not to say there is a single DIT - have multiple, including one just to test out package installs if need be). QA regions and beyond are under tight control. I work in banks a lot, so Pre-prod and Prod are under a metaphorical armed guard.
Once the installation and config documentation is tested by the developers, the docs get thrown over the wall to the integration team (optimally, QA should be involved in a doc review to make sure that what's in the Doc is what is required, no more and no less). For a Waterfall / SDLC methodology, this documentation review and handover is one of the gating steps. For an Agile / Scrum / XP methology, this can be considered a single story, where the success condition of the task (story, etc) is working installation (works) and usable documentation (has been tested).
The key is not to go bat-crap crazy on it, but to ensure repeatability and workability. It would be GREAT if the install could be automated (or run unattended, or have little or no intermediate steps requiring human intervention) so as to reduce integration errors, but that is dependent on the requirements of those managing the QA regions and above.
Re:Why not use tools that help do it? (Score:4, Insightful)
If Dev and Prod aren't identical, then their Configuration Management team has failed.
They usually aren't hardware wise for budget reasons, hardly ever are configuration and environment wise (considering other running programs as part the environment) and they definitely aren't identical user base wise.
While I understand devs shouldn't modify anything in prod, denying access to the system to see in real time what the fuck is going on when something doesn't work is just asking for the solution to take 10x, frustration, finger pointing and longer hours (usually the developers and often unpaid).
Seriously, nothing wrong in telling the developer "come down with me, I'll start the damn thing in front of you, you can check a bit of the system to you go back with a clue".
Re:Why not use tools that help do it? (Score:5, Insightful)
Sorry, no. I couldn't disagree more. I've worked in places like that, were developers were unable to get close to the production servers, things wouldn't work in production (but worked fine in dev) and we were unable to do anything except send builds with more and more debug info, working late nights to get things done, with a client more and more pissed each day.
Then it turns out that contrary to what they said, dev and prod wasn't identical, in a number of important things such as library versions.
If the install team is unable to install it by fuck's sake *get a developer see the installation process by himself* so he can come back just once with all the data he needs.
Then your process is screwed up. The solution isn't to open production to developers but to get your process fixed. Part of the process must be ensuring that the production environment is regularly and consistently copied over to development to ensure the problem you describe can't happen. That's the solution.
I have no problem with giving developers read-access so they can look at the production environment to "gather data" but developers must not be able to ever install or change anything in production. That's just good sense.
Re:Why not use tools that help do it? (Score:5, Insightful)
Re:Why not use tools that help do it? (Score:4, Insightful)
depends on the software....
In terms of, for example, the web database etc. mentioned in the question you're into a lot of installing an SQL server (that's separate from a developer team generally, but it's a 30 minute job if you know what you're doing), and then building the database. Once you built the database you don't necessarily have an 'installer', you just send them an image of the whole installation (or the whole physical machine).
If you're a company that specializes in making billing software databases then sure, you might have an install script and setup parameters than your installer guys can execute.
There's sort of a hierarchy of techie people. Not everyone at a higher level can do the low level jobs, but one person at a higher level might be able to do all of the lower level stuff in 20 minutes that would take them weeks. This isn't 'consumer' product, in enterprise you might send out a 'linux guy' (who doesn't know linux scripting, that's an advanced linux guy) to do a linux install, but he can be given detailed, line by line instructions to follow from developers. In enterprise 'you' as the development firm can be responsible for getting it installed and setup. And you have co-op students who are basically high school grads following instructions for installations, or you can have a partially automated lab with software engineers running deployment tools to hundreds of machines at a time, it just depends on how big an outfit you are and the sort of customers you have.
Re:Why not use tools that help do it? (Score:5, Insightful)
I have often worked as a contractor for major corporations, and I have worked with contractors as a system administrator. I was not accusing you of anything but I was criticizing the environment you were forced to operate in. If the development environment gets so out of sync with production that installs succeed in development and fail in production, the solution is not to allow developers to change production but to correct the environment.
If production doesn't match development (and/or test) then the whole process is broken! Development is invalid and testing is invalid. Both development and test must duplicate production or absolutely nothing done in development or tested in test can be trusted.
As a developer, years ago, I actually modified a running production environment and I was very lucky that nothing went wrong. I understand that it happens and, in broken environments, sometimes it must be done, but that doesn't make it safe or right.
Re:Why not use tools that help do it? (Score:3, Insightful)
That an application "doesn't break" is only the very first test in a long list of criteria. Many applications that "don't break" will still corrupt data or cause other existing applications to fail.
Development done in a different environment that production is a waste of time and testing in a different environment is useless. Good luck with your method.