The Rise and Rise of IT Administrators 686
maffstephens writes "Have you noticed how difficult it's become to develop software? Not because software is more complex, but because there seems to be an army of administrators standing in your way - sys admins, network admins, database admins, runtime admins - the list is endless. They should be there to help us, to make our lives easier, but the reality is often very different. This thought-provoking article from Software Reality is all about the emerging culture of spiteful, dog-in-the-manger prevention amongst corporate IT administrators. Software development has become so inefficient as a result, it's no wonder so many companies are outsourcing."
Outsourcing (Score:3, Insightful)
Developers (Score:5, Insightful)
(That's all I hear these days, thank you Steve Ballmer)
As a sysadmin, the Devs need to learn how to play nice and keep the system stable. As a developer, I want total access to everything.
Solution? Developer network off the main network. If they blow it up, it's their fault and they fix it. Sounds good in theory. I think programs like Ghost will play a big role in this type of setup.
power corrupts (Score:4, Insightful)
Re:In all areas (Score:3, Insightful)
Work For Yourself (Score:1, Insightful)
I highly recommend it.
Have you ever stopped to think ... (Score:5, Insightful)
I'm a UNIX system administrator. My responsibility is to ensure my systems perform well. This includes actual performance statistics (I/O, CPU, memory), security, reliability, scalability.
It also means I need to scale up the hardware as applications grow. I need to keep tabs on what my systems are doing, and why.
I'm the "guy who gets in your way" because my responsibility is to the system, not to you.
I don't work for you. I work for the systems. They are my "customers" if you will.
Sure, I slow you down when I tell you "No, your app can't run as root."
I slow you down when I make you diagram your database so we can lay out the I/O correctly.
I slow you down when I make you tell me what you're doing with shared memory so I can tune my kernel.
I slow you down when I ask for projections over the next year so I can plan the hardware and scale appropriately.
I slow you down when I shut off telnet, ftp, r services, and every other plaintext protocol. You b*tch and moan because your expect script from 1994 needs to be rewritten, but too bad.
I slow you down when I ask for a detailed list of which ports your application uses, who they communicate with, and what IP blocks I need to permit access from.
Yep, I'm in your way.
That's my job.
And if you don't like it, well, too bad. I *DON'T* ask you why you're using C instead of Java. That's not my business.
I'm a systems subject matter expert. I don't pretend to be a code expert.
Your a coding expert. Don't pretend to be a systems expert.
Let me do my job, and I'll let you do yours. We need to work *together* and understand the interactions between your code and my systems.
Systems are NOT simple. They're very complex; you need to understand all the interactions here, from the kernel through the disk management (whether it be VxVM, LVM, or whathaveyou), through the network drives, through the firmware, through the HBA drivers...
Let me do my job. Yep, it'll "slow you down" a bit, but in the end, we'll actually have a complete SYSTEM that functions. Code, OS, hardware.
So you can't roll things out in an hour anymore. At least it works now.
No, we need more GOOD planning (Score:5, Insightful)
I've been involved with companies who spend forever planning and twice as long coding, and they still produce crap. Why? Because the design is always done by committee, so no really good ideas get out there, and the design always ends up as a preoptimized mess with a few "management-approved" ideas thrown in.
I seriously think a small tag-team (2 or 3 people) should be responsible for projects, and they should take in all of the input and recommendations, and produce a solid spec by themselves.. rather than the typical '10 departments sit around a table for 20 meetings and produce a piece of shit' method.
Delicate Balance (Score:5, Insightful)
We need more planning and less coding. (Score:2, Insightful)
Because these "coders" know little to nothing about security, I spend hours in meeting trying to drill into them why the firewall is in their best interest, and why they should be using different ports for each protocol.
If we all just sat down and coded first, our productivity would soar.
And you end up with Microsoft-like products.
Re:We Need Less Planning and More Coding (Score:5, Insightful)
Umm, first you say the problem with these administrators is their not developers, then you say we should just sit down and code?
Any good developer who paid attention in their software engineering course would know the further down the development cycle you get when you discover a problem with your specifications the more expensive it becomes to repair the problem.
Make prototypes I can see, show the basic functionality and flow of the software. But before developing any large software project one must design the specifications and requirements.
Any developer who doesn't understand this would fall into the same boat as these "non-developer administrators" in my opinion. Go pick up a software engineering book and re-read it. And make sure it's not eXtreme Programming, that book is how Windows-like disasters are made.
Re:We Need Less Planning and More Coding (Score:3, Insightful)
And despite your sniff at "enterprise architecture" let's keep one thing in mind; both the coding and the planning are there not as an end in itself, but as a means for the enterprise to make more money so that you can continue to code. If you're not doing anything useful for the enterprise, then it has a way of meting out retribution (in the form of closed companies and layoffs.)
Arrogant developer crap (Score:5, Insightful)
As a result, nearly every end user of a developed system is given attention before system administrators and operators. The secondary result is SA's and operators are left with big piles of innefficient crap to wade through, and much of the pressure of making said piece of crap work. How many folks here have had to work in huge, bloated teams of SA's all to support an ill-concieved and poorly developed (but gee whiz does it look greeeat!) product, getting paged and phoned all night to come in an slap more duct tape? How many people here have had to manage a bunch of boot-camp MCSE's trying to do 400 manual processes an hour because "that's the way it was developed"? How many people here have had to explain to a customer that some piece of code written by a fresh off the MIS degree train VB developer isn't RFC compliant and therefore 45 percent of the people in the world won't be able to interface with it? How many SAs here have had to tune the crap out of boxes and networks because a login page makes 75+ ODBC database calls? How many security consultants have had to go in and basically tell a company that they'll have to repartition and reinstall every server because someone found SQL injection in an app that required superuser privileges?
The list goes on and on. Administrators aren't there to make life easier for developers, they are there to make things work--and make them work better, more reliably, and more securely. I'd suggest that this whiny ivory tower developer wake up and realize that coporations have gotten smart to the crap he's been turning out and further realized that the people who run the stuff are just as important as the people who write it and use it.
In short, he needs to learn how to work on a team.
Developers are smart, but they aren't the top of the computing pyramid. There are many other groups of people that are just as smart in different areas.
Re:Nothing against prgrammers (Score:3, Insightful)
Cleaning up after code monkeys who wreck a production server is not fun.
Re:We need more planning and less coding. (Score:5, Insightful)
This pushes me to take responsibility for having an overall understanding of how the application fits into a larger security context, and that the application works in the real world/under load.
Only then is the app dumped onto the larger network. I think all developers should do some real-life system adminstration, and system administrators should do some development.
Nice mindset. Here's the flip side. (Score:5, Insightful)
Nice mindset there; you're a real team player. The reason we are there(network/sysadmin here) is to HELP you.
However, we're also there to make sure you don't do stupid things. While you say "these IT people get in our way", I point to a laundry list of really, really, REALLY stupid things developers have done at every company I've ever worked for; they just don't THINK about anything besides code, and they get Great Ideas without thinking through the consequences, either technology or business-wise. Some of it is just sheer laziness, and I've been faced with developers who act liked goddamn 5 year old spoiled rotten BRATS- this was particularly bad a few years ago when anyone who knew what "printf()" meant, got a 75k+ job.
Prime examples of stupid things I've seen: logging into machines using the root account because you're too lazy to use su. Or not allowing you to ssh directly into a system from your home PC without a VPN. Or yelling at you when you use temp tablespace for permanent data. Or not letting you move production functionality to some desktop system underneath your desk. Or using the database admin account for your application, instead of a seperate account?Or not implementing your latest code changes until you're willing to put down on paper that you actually did your job properly and TESTED the damn changes(do you know how many times I've seen developers just push code out without testing? Guess who gets blamed first. Guess who gets PAGED first, at 3am when it crashes. Management doesn't distinguish between a misnamed variable and a "Internal Server Error 500"; they're both production problems, and you're not in charge of production).
We're part of the team, and we're here to stay. You can either work with us, and clearly communicate to OUR supervisors(not just us) what your needs are...or you can make us the enemy, always try to do things half-assed, and get nothing done. Your choice- but management usually sides with safety, and we're the ones saying "that's not safe", and even if management doesn't side with us- when things blow up, we simply point to the emails we sent saying "that wasn't safe", and let you sweat it out while we restore from backups and clean up your mess.
Sometimes it's simply not our choice; it's "do it this way, tell Development that". You have no idea how frustrating it can be sometimes for even us- I once worked at a place where root passwords were changed on us sysadmins, and we were told "use sudo". The incompetent assholes a few levels up didn't realize that gee, guess what, if the machine crashes and fsck fails, you need the root password.
Re:We need more planning and less coding. (Score:5, Insightful)
My developers tend to want to run their web servers on port 80. I won't let them.
Why not? Because then they have to have root privs to start/stop the app.
No dice.
What's my solution?
Run the webserver on a high port (I tend to use 8000, but that's arbitrary)
Put the systems (Yes, each app has to have at least two for redundancy) behind a pair of load balancers. Let the load balancer do the work. While we're at it, make sure the load balancers have SSL accelerators too, so we can offload that from the CPUs...
Much saner architecture than letting a developer download Apache from Sunfreeware and running it on port 80.
And then people wonder why we have sysadmins?
Walk a mile in the other guy's shoes? (Score:5, Insightful)
Are there sysadmins who've never coded, not highly skilled at what they do who are a drag to work with? Of course. Sysadmins run the gamut, the best (and probably most productive) have enough coding experience to know and work with the dev side also. The very best can run circles around the average dev imx. Naturally the very best devs are int the same class.
There are just as many 'developers' who don't have the first idea how to perform adequate testing, let along consider the constraints of running in a production environment let alone writing portable, consistent or maintainable code.
The author of this article is quick to bitch about a sysadmin losing his working files. Sure it happens. What the hell is with a developer who doesn't bother to keep any working copies of 2 weeks work? (In my own time managing a corp. network I'm pleased to be able to say I had exactly one instance of unrecoverable data loss -- where two users hadn't realized that NFS did not provide pc clients with any form of locking)
As a distribution maintainer (lunar) [lunar-linux.org] I see several OSS packages a week breaking reasonble build schemes or changing thier tarballs, (breaking MD5/PGP checks) without updating version info. So I'm sorry but there are no shortage of sloppy developers out there.
In my own engineering practice I've found over the years that all work goes better if the people doing it know they'll be held accountable for it over the long haul. Too many devs are allowed to get away with a 'throw it over the wall' mentality, going on to the next project and never having to deal with some of their cruft. Of course the same logic applies to the sysadms, I've seen lots of the behaviors the article rants about it but I gotta say ranting and pointing fingers ain't the solution.
Not sure where you're from but... (Score:5, Insightful)
I don't know what planet you or the author of that article live on, but I've seen a steady increase in the quality of software development and maturity in the development models used in the last decade. This may have slowed our development a bit, but you can see the results in our defect find rates. We're delivering a much better product than we used to.
Rather than just hacking out some code, doing a perfunctory test, and throwing it over the wall to be released, our developers are actually managed these days and do this cool thing called "planning." Yes, they actually investigate, propose, design, implement, and test code on a schedule. We even have teams dedicated to testing the systems to make sure they work. Oh, the horror!
In a decent software lab, which I consider mine to be, most of our management is also made up of engineers who rose through the ranks. These people know their stuff and trust the engineers beneath them.
In my area at least, we've also seen a large increase in the complexity of systems as well. No longer are our engineers programming a lone application to slap on a PC or a server. Our projects are large and distributed across multiple networks and servers. We have to traverse firewalls and worry about security trust domains and lots of other things that nobody cared about a decade ago.
I think that this increase in complexity of projects is likely responsible for the entire list of negative consequences that the article attributes to 'role fragmentation'. The only one I'd leave out is "de-skilling of the workforce". That may have been true in 2000, but the layoffs of the last few years have forced everyone to do work that was once done by multiple people.
All of that requires more attention to detail, and requires more effort to get right. I don't see that as good or bad, it simply is. Get used to it and stop whining about having to actually plan something and coordinate with others.
- Necron69
What is wrong with outsourcing? (Score:5, Insightful)
Re:Have you ever stopped to think ... (Score:3, Insightful)
"Don't get paged."
My customer is the system, my objective is the availability of said system.
Like you said, most developers have no idea really what goes on behind the scenes. They don't understand why building a cluster is difficult, let alone what quorum is, failure fencing algorithms and the like.
They have no idea why it's perfectly OK for a cluster node to shut itself down, given the right circumstances
But I digress
The article sounds like a guy who's never worked in a real enterprise shop, and is upset because they didn't give him admin rights on his PC
Re:Lots of admins here (Score:5, Insightful)
to say any different reveals your ignorance about either field.
Re:Have you ever stopped to think ... (Score:5, Insightful)
I can't believe that even needs to be explained.
Not a rant against administrators... (Score:3, Insightful)
Just as low quality developers with no sensitivity for production issues cause problems for talented admins, low quality admins with no knowledge of development cause problems for the developers. Talented administrators help your development team build bad-ass production ready apps and don't get in your way.
Mostly though, it's IT management and corporate higher ups that have created this sprawling bureaucracy, for a variety of reasons. The admins would love to change it, but really have no say.
As with anything, hire talented people and things will run more smoothly (as long as you don't shackle them with process developed by and for the untalented people
I programmed for 10 years ... (Score:3, Insightful)
And yeah, there are BOFHs. Even sysadmin run into them themselves if the organization is large enough.
Because Administrators are Responsible (Score:5, Insightful)
Because Administrators are the ones who have to deal with the most headaches. I quit administrating and switched to development because of the complete lack of control I felt. The bulk of my admin was on Windows NT servers. A bad patch or rogue program caused grief, which I was expected to fix. Because of largely closed-source development environments, that meant flailing around in the dark trying endless shotgun approaches: patch, reboot, test, change, reboot, test, reconfigure, test, blow out OS, reload, test...on and on and on. Meanwhile developers would say "just get my database up and running! I don't care about _your_ problems".
Unix is the best environment I know for Administrators. It slowly nudges them towards programming because of the close relationship of scripting and automation. Admins grow to become programmers. NT on the other hand, is a completely non-sensical environment because it's prodiminatly adminsitrated through application layers; no programming knowledge required.
The old addage of freedom and responsibility applies. The more responsibility you have, the more freedom you should have. The less responsibility you have, the less amount of freedom is tolerable. Since a lot of admins work with closed-source products, they do not have the freedom to fix or investigate problems the way their open-source counterparts have, and therefore are given the responsibility without freedom.
Largely, I agree with the article's points, but I think the blame goes beyond the administrator. I think it belongs squarely in the lap of the commercial software industry. Then answer: open-source.
Re:In all areas (Score:4, Insightful)
Re:Have you ever stopped to think ... (Score:1, Insightful)
There is no need for anyone to have to put a project on hold to wait for software to arrive from a vendor or be installed by the IT group. What should have happened is the software was ordered when the project was in the planning stages so it would have arrived and been installed on the developer machines before it was needed. But (yes, I'm going to blast the developers) rare do they think ahead.
Better planning will also avoid some of the problems nbvb brought up. There are very few times when a process should ever be considered for root access. If it is not a system process, it doesn't need root, period! Security is a huge deal this days and you need to plan the project with that in mind. Take the time to develop the software so it runs without root. Database diagrams are essentail to maintaining the system and the software. As for naming goes, like everything use names that make sense, please! The administrator who limits names to archaic 8 character names, better have a damn good reason!!!
As for all you who think "development" is about coding, i have to disagree. True development work is about designing a system. Anybody who understands a programming language should be able to write a well designed program. Which is not to say the understanding the programming language is trivial, but rather the process of actually coding should consume a whole lot less time than actually designing the application.
Feel free to blast away on the last point. I know many will disagree as it flies directly in the face what some thing XP should be.
On to the authors point about virus scanners. It sounds like the virus scanner should be setup differently and the policy on the server definitely needs to be changed. However, as an system administrator will tell you, the are a must these days. Unprotected machines getting tagged by viruses will cause (and have) more downtime for the company than having it running and preventing them.
The other thing system administrators have had to learn to deal with is the law. Licenses for software must be purchased if you are using the software. Viruses spoofing emails from the company or from company computers (everyone say SPAM) is a legal liability to the company.
That's enough ranthing for now.
I think the point is (Score:5, Insightful)
That there is a reason why alot of admins are paranoid about giving anyone , not just developers, control of their box. The case in the article was an extreme example, but I couldn't help but wonder "What did some developer do years ago that completlty hosed everything?"
The back up situation at the place in the article sounded outrageous. The author had every right to be angry about that.
As far as the firewalls go..if there is a security breach, the developers would not get sacked and new abused ports are discovered. Users find ways of clogging everythign up with Yahoo! IM going through port 80, outside KaZAa users from Brazil suddenly thing that you have LTR Return of The King hidden somewhere on your network or script kiddies from Korea sudenly decide to scan port 1021 all day long...In other words, there are lots of reasons to change the configuration of a firewall daily (unless disconnected from the outside completlty..but no users want that).
Like NetNinja said, cleaning up after them is a nightmare, plus the admins are liable for the mess , not the developers. Communication between groups is the key.
Re:No, we need more GOOD planning (Score:1, Insightful)
Re:Excuse me? (Score:3, Insightful)
if engineering was like software engineering, you'd go into an architect's office, and say "great job on that suspension bridge. Whip up a motorcar engine for a universe where an electron is +5 charge"
Re:We need more planning and less coding. (Score:1, Insightful)
Inept developers on par with inept admins (Score:3, Insightful)
That being said, The two jobs are really intertwined: admins should be contributing to the design stages of software just as developers should be keeping in mind deployment, administration and security factors when they're writing code. Sadly, I've met people on both sides that basically say "not my job" and move on. Those are the ones that cause problems.
BTW, I don't have a lot of pity for the author. He tries to make a point by saying his roving profile got deleted and it caused him to lose 2 weeks of work. Let me just say in my experience I have never gone that long without a commit. The rule I've always gone with is that if the build isn't broken it gets committed. There's really not an excuse, even in early development phases, to not be comitting often. This just sounds lazy.
Derek
Re:We need more planning and less coding. (Score:5, Insightful)
The ideal project group contains about 3 people, all of them code, all of them plan, all of them test, and all of them administer. The individuals may be responsible for a specific area, say security, but they all know and share in the design.
The main problem is that technical people don't know enough to serve in more than one capacity, and the formal corporate structures enforce this division, through their arbitrary classification of techs based on which ersatz MS diploma they possess. Administrators think that software development is separate from security administration, when in fact these are just two chapters in the same book. If developers know nothing about security you end up with MS Outlook, if security admins know nothing about software then you still end up with MS Outlook.
There are only two different job classifications in IT, the Hacker and the User.
Re:Nice mindset. Here's the flip side. (Score:2, Insightful)
Re:In all areas (Score:3, Insightful)
Our school's sysadmins were crap. No really. The secretarial server - with all the confidential student and teacher data - had the unicode bug and they refused to fix it, can't remember why. Eventually, a friend and I got bored of seeing all our personal details on show (unencrypted SIMS) and went round and fixed it in 2 minutes.
Our head of department once gave me a lecture over playing Flash games online cos they "could be virus-infected". If there's a way that this is possible, someone please tell me.
There was no defence against a simple promiscuous sniff, let alone ARP cache poisoning. This was a relief as, when the Head of IT "reconfigured" the email server to run on the wrong port, then left for a day's conference, one of my friends was able to reroute the school's mail via his laptop and send it on to the new port.
School IT staff are only in it cos they can't get jobs elsewhere...
The article's author is confused. (Score:5, Insightful)
The reality is that the authors problems are due to inept individuals and the corporate bureaucracy that keeps these inept individuals in place. The problems are not simply admins vs. developers. This is no different than any other profession.
There are countless bad administrators out there. Many/most do not deserve the title of Administrator. But, at the same time, there are just as many developers out there that should not be allowed near a keyboard and yet they are forcing new "applications" down end user's throats on a daily basis, "applications" that reduce productivity due to bad design and processing inefficiency, buggy and untested code, and a total lack of understanding of the business process.
There are far too many inept individuals on both sides of the fence. It is not about admins vs. developers.
One more thing, the author seems to understand that J2EE is a bad idea so, why does he continue to develop with it?
The Chicken or the Egg? (Score:4, Insightful)
In the end, if there need to be so many admins, it's because the software demands it, not the other way around. And I tend to think admins are pissy because developers are increasingly giving them crap to administer.
What is that whining noise? (Score:1, Insightful)
The FA is just a bunch of whining from this particular person's viewpoint. Big deal. Does this rate a slashdot story? No.
The reason we have division of labor in big environments these days should be obvious. When your company handles millions of dollars they don't want the sales guys doing the budgets, they have dedicated people for finance. Specialization. If you want to wear all hats then find a small company where you can do everything at once and work 90 hours/week.
I should point out that 10 years ago it was much more practical for people in even medium-sized companies to be part-time admins because the systems weren't so FRIGGIN COMPLEX! When you've got 3 times as many systems now, and heftier software, and more patches and security threats to keep up with.... What might have been a 40-hour programming week with an extra 5 hours of admin thrown in 10 years ago, would now be 40 hours plus 40 hours. Most places I have worked as admins have been only too happy to offload this annoying tedium of system management so they could get real development work done.
And now, back to the whining....
The Answer to this Problem (Score:5, Insightful)
but it seems that the author missed several of the intervening years that have led to the current situation.
In the beginning, the developers also were required to administrate the machines they were developing on and for. Shortly after that, as there were more deployments, there were folks who's primary task was system administrator, and they would perform development tasks according to the needs of the organization and in order to make thier own jobs much easier, then came the network administrators, who would also develop software according to the needs of the org, and in order to make thier own jobs much easier. Then it all went to shit as the marketing department realized that there was money to be made, they began asking for needless software with dubious need and poorly thought out devlopment requirements that could be used marketing fodder. The administrators became notorious for (rightly) defending defending thier turf and saying "not on my network. not on my system."
So the role of developer was born, a person with skill in writiong code with the willingness to write program asked using whatever programing language specified without any objection whatsoever, regardless of the technical merit of the spec, the need for the program, or the overall effect on the system, as long as they were paid. All applications were written in whatever the language of the day happened to be, and fufilled the purpose of whatever the flavor of the month dioctated. What had been elegantly designed systems that specifically fufilled the needs of the user using existing tools (most often transparently) whenever possible, using custom (or community) designed software whenever necessary, and requiring the least amount of system rescources possible, now became incomprehensible morasses of rediculously complex dependancies, multiples of propietary protocols that replicated each others capabilities but were "incompatible" with systems that served the exact same purpose, huge collections of libraries all addressing the same needs and differing only in what would justify the high cost of the (propietary) product, and an absolute disregard for any sense of of efficient and elegant network, system, or application design.
The design process has been divorced from the persons who use the apps, maintain the systems, and have the best knowledge of the needs of the given organization. Software development is now managed by sales people, marketing divisions, and corporate executives, most of whom have little real knowledge of the IT feild other than what they read in Gartner's artcles, and will accept the advice of a "consultant" before even considering asking one of thier own employees. These are the people who believe that the best developers are teenagers, that ".net" is the "way of the future", and that when a sales person tells him that thier product achieves something never before accomplished, or that it provides capabilities available nowhere else, they believe this.
Now the developers are crying that they don't have domain administrator rights on the network, or that they cant write to directories that they have no reason to be writing to in the first place. They bitch when the network has been infected by yet another virus, but complain when the administrator strips all VB script attachments from thier emails. They bitch about how much work they have, about thier hours, and about thier pay, but drop to thier knees for any manager that brings them yet another impossible to implement product idea or project that serves very little purpose (other than as something that might sell). They bitch that the admins are fscking around all day without understanding that this means all is well on the network and the admins have done thier jobs well.
This is the problem in the development world, and it is being addressed by Open Source. True, there may be some job loss a
Seems a bit superficial (Score:4, Insightful)
However, there's a real need for administrators, and increasingly so, as the systems get bigger. I'm the lead DBA for a app development staff of 25. Do I get off on holding the keys to the databases? No, but we ran for a long time with developers as sa. The bottom line was that there were a lot more problems than there are now that I locked the dbs down. I also realize that that puts a greater burden on me to not be a bottleneck in the development process.
That said, the problem is not in the fact that you have administrators, it's that you sometimes forget what your role is. Developers forget this all the time, that they are supposed to be responsive to users. Administrators forget that they are supposed to be responsive to users. Customer service forgets that they are supposed to be responsive to customers. I have to occasionally step back and remind myself of the fact that I am there to support the developers. But to think that this is only something administrators are prone to is to try to single them out as being exceptionally sinister. It's just human nature, and we all have our way of sometimes screwing the people we are supposed to be helping. Contact your local congressman for more details.
Every day, I spend time configuring the system and developing policies that give the developers the greatest freedom possible while maintaining stability. And in general, the developers appreciate the effort. Why? Because each one knows that, while he is sometimes hindered by my policies, he benefits greatly by everyone else following them. And therein lies the whine of the selfish developer. He wants everyone else to follow the rules to make his life easier, but he doesn't want to follow them himself.
System built by developers (Score:4, Insightful)
Here's a quick idea:
Server was floor standing in a wiring closet at the location the developers worked in. Not, in a rack at the corporate data center (with redundant power, switches, etc).
The server had 10 (!) 100BT NICs. They were all teamed and run into a 10/100 hub on the floor that then had one uplink at 10BT.
Server had a very nice 6 channel RAID controller that was completely unused. Instead the hard drives were connected to the internal SCSI and software RAIDed.
Moral to this story: Yes, developers and admins should work together, but each should respect the other in their field of expertise. If the admin tells a developer something is a bad idea, they probably have a reason for saying that.
Configuration management: A disapearing role? (Score:3, Insightful)
Maybe the problems with keeping development, deployment, and systems administration in sync would be helped by that level of glue.
The result would be that the fragmented systems administration level would be simplified since they wouldn't have to deal with configuration issues except where they impacted ... well ... systems administration.
Re:No, we need more GOOD planning (Score:5, Insightful)
In the "new age", where everyone and their brother got into computers because "that's where the money is", there are a number of real problems.
1) The average skill level has greatly diminished. Thus, "the masses" have to be partitioned narrowly as they cannot, generally, operate on a big picture. So there are far fewer people who can actually plan, as that demands broader skills.
2) Once you discover the need to partition on narrow skillsets, every partition comes with an automatic presumption of expertise vis a vis all others.
3) Once you have "experts", you presumably want them to assert that authority. Thus you end up with network "experts", database "experts", web hosting "experts", ad nasium. Each, by definition, opertating in a clueless vacuum due to organizational structure, and the original reality that that structure was created to address the narrowness of skill found in most modern day technology people.
4) Now that you've accepted a lower skillset per capita, and tanked up your organization with same, you created yourself a self-fulfilling prophecy. Each "expert" will basically refuse to co-operate with any higher functioning integrating authority out of self-preservation. Human nature will refuse to hire, cooperate, or contribute (to the maximum extent possible) to anyone that might threaten their status.
In "the old days" the model you suggested was pretty much the norm. Execpt there was no need for 10 departments. Generally there were 3, Businss Analysis, Development, and Operations.
Systems were, generally, "standardized" becuase the universe of potential staff was small and maximalizing technical diversity was not in anyone's best interests. There was no need to compete with your fellow technologies, there were few of them, and they all were in the buisness as a result of a legitimate calling.
Today, hacks are the norm. There are a number of dubious stratigies employed to remain competative, none of which includes standardization or hiring "the best (and possibly better) people for the job.
I'm an old-timer, watched it happen. I sit here, unemployed, today becuase I felt it important to hire "the best people", even if they were "better". Net-net, after 10 years, they still have jobs maintaining a system I designed and built for pennies on the dollar, played a role in moving from 80 to 250 million dollars a year, and enabled a host of standardized technologies like TCP and web. See, "we don't need generalists anymore" (they are "disruptive") was the reason everyone who actually contributed true-IP for this company was let go.
From now one... I'm dumb, er "focused" in Corporate speak, narrowly skilled, and will never hire anyone that's bright enough to span technical disiplines. I'm just sorry I didn't get that message soon enough.
Re:No, we need more GOOD planning (Score:5, Insightful)
Re:We Need Less Planning and More Coding (Score:3, Insightful)
What most coders don't understand is that I have to support several thousand workstations and servers. Every few weeks, some application appears on the network that I've never seen and never heard of. I have no idea what it does or what resources it needs. It has no documentation nor any interface the likes of which I've ever seen. All I know is that I have an irate user on the phone who needs it to work now, ten other users in queue calling about ten other apps or scripts and all they know is that the developer said to call the helpline if there were any problems before leaving to the next shiny project.
If anything is going to be used on the network, I want it to be rock-solid, have a consistant interface (and ascii data dumps for crying out loud - I'm so tired of trying to perl together obscurely formatted excel spreadsheets "dumped" from databases to be used by HPUX users), be well documented (including how to diagnose its failures not just a list of what cool stuff it theoretically does). I want there to be a plan for how it is going to be deployed, how it is going to be supported and how to uninstall it when it's discovered to be full of unrepairable security holes.
Personally, I feel like too many coders and developers and planners who complain about admins are just the next generation of people who didn't want to add comments to their code.
(OK, I feel better now)
Re:We need more planning and less coding. (Score:1, Insightful)
Usually require special priveleges to bind to.
Now fortunately, the company I was at gave developers full control of the test machines.
The finished product was given to folks like this guy who then ran it on whatever the hell port they want.
By giving us control of the hardware to get our jobs done, we didn't run into guys like him.
Re:We need more planning and less coding. (Score:2, Insightful)
Developers don't need root access. Simple.
For what? Give me one good reason why.
It's not a control thing. It's a reliability thing.
Any downtime is not tolerated. For every minute my production machines are down, we're losing hundreds of thousands of dollars. Really.
Why should I let anyone except someone who doesn't know exactly what power they wield have it?
I'm not getting thrown anywhere. My systems, company-wide, had the best availability numbers for the first 11 months of this year. And all of last year.
That's because I coordinate with my development teams, and we're in sync on any projects going on.
And if you want to compare us against other customers, Sun came in and did an evaluation of our systems as per our Platinum support agreement. Our RAS profile score was better than *ANYONE* in the United States.
So yeah, maybe I don't let the developers have free reign, but we also have the best-performing, most available systems around.
Re:Developers (Score:3, Insightful)
Since you've never been a sysadmin, you really don't know what you're talking about, do you?
Sysadmins not only attempt to keep systems up and running, they also have to ensure compliance with the company's security policy, acceptable usage policy, touble-shoot user calls when an individual can't access a website (not the corp. site), configure a multitude of disparate hardware to play nice with shoddily developed software, cater to management whims, concoct solutions when things go TU, etc...
I've done it.
Now I am a Test Lead in a Gov't software development environment and trying to help the Dev team understand that their apps aren't going to make it out the door without some semblance of compliance is the biggest battle I have on all projects.
Just a little bit of understanding of the constraints we all have to work within would make things incredibly more efficient.
Bad Experience != Administrators Bad (Score:3, Insightful)
I read this guy's article, and I feel his pain. He is clearly suffering in a bad corporate environment.
What he has done is scapegoat administrators, when what is happening is not a problem with administration, but with senior management.
When there are too many administrators, when they are not doing their jobs well, when they are deploying bad tools, when they are poor communicators or worse, obnoxious... when, overall, policy is bad, and the work environment is imperilled, look up, my freinds, to the head office. The fish rots from the head.
The idea that administrators might be a "problem" is novel just because normally in a bad shop with failing senior management you just get bad developers... bad everybody. Because hiring standards are shit, because bad behavior and bad performance doesn't get anyone fired.
I actually do see specialization as a problem - but as an unavoidable one. As our work has grown in complexity over the years, specialization was inevitable, and we are just seeing the tip of the iceberg. But let me speak from the vantage point of a relatively well-managed shop. Here is what work is like for me.
We do have an IT group that "runs the desktops," making sure email gets delivered and wrangling windows and fileservers and so forth. They lock down the machines of quite a few people, but they are flexible and friendly - and developers (and anyone else who has a need) get full privileges on their own machines, and even on development servers if necessary.
Our backups run, consistently, on time, and well.
We have a Unix ops team (that encompasses our DBA) that is 2nd to none in my experience. They will ride your ass hard for anything you want to do in production - as they should. It's their ass if those machines go down, and that 2nd set of eyes has caught many a terrible thing before it escaped our development and staging environments. When we are testing something for scalability, they are in there with us, unleashing Solaris or Oracle wizardry, literally coaching better code out of developers, and generally making magic happen.
That in itself is a good snapshot of specialization working as it should - not insular, but aware of what's on the boundaries of what you own, and working as a team.
We have a complex security architecture that simply works. They don't make semi-montly changes to it, because they did it right the first time. DBA staff is a collaborator in your database design - my god, I can't imagine these people holding _us_ up. Its almost always them waiting for us to carry our leg of the relay race.
Our long-suffering QA department is strikingly overqualified and polite to a fault when we destroy their schedule and then ask ridiculous feats of endurance from them. Even when they catch some pretty dumb stuff on our part, which they do quite often.
We use good tools, often open source, but not out of doctrine - our choices won on their merits. Management is ready to throw out anything if it doesn't work. We are platform and vendor agnostic - advocates and zealots get put in their place.
We are very productive, and although we are far from perfect and are not all of us dream-team members, we do extremely good work. Our company is so spoiled, and I think many who are relatively new have no idea how lucky they are.
And why is all this? One simple reason. We have a good CTO, and he's hired and promoted middle-management who are good engineers. And that, pretty much, is all it takes.
The trouble in big companies is that you have absentee management who pick senior technology leadership without any sense of how to gauge talent, and pay no attention to their performance, almost literally until (sometimes even after) it has burned the company down. The old white men are by and large still prime representatives of the previous generation in terms of their ignorance of technology, and every generation's aristocracy has the same habit of being in flight over the Swiss Alps while the house is on fire.
One man's opinion.
Admins Missing the Point (Score:3, Insightful)
I understand that those DBAs who understand the details of the database engines are worth their weight in gold.
My impression is that he is not talking about the sort of admin that is likely to be reading Slashdot on a Saturday. He should have repeated this statement in each section, to make it clear that there are good and valuable admins in every sector.
It is my experience that we are now 60% over-administrated.
This is also a bit too understated. He implies at a few points earlier in the article that he works for large enterprises. If you've worked in a large enterprise, you know that in such places, the paper-pusher admin to skilled admin ratio is 60/40 on a good day, going downhill, with a gale-force tailwind.
The people he's attacking are (for example) the sorts that engage in "security by chewing through the wires" - putting a firewall at every major network nexus, shutting down all traffic, and demanding written justification and properly red-tapified authorization for every open port. Don't get me wrong - default deny at the perimeter is a must, and default deny on some nexuses is the right choice. OTOH, for example, a default deny firewall between the developers and the appservers has a very real cost (EG: waiting six days for paper to clear before being able to turn on JMS). He's not even saying this cost can't be justified - he's just saying that cost has to be assessed and charged to the administration budget, and it is currently charged to development:
the true cost of administration must be accounted for when totalling up the cost of any project.
The point he's making is not that administration is bad, but that because management has lost it's grasp of development, and because they can grasp the paperwork-and-authorization oriented style of administration, management has given administration more power than is optimal. There's a balance that must be struck between wild-eyed developers and stodgy administrators - safety and speed are both valuable, and they are naturally at odds. In major enterprises, the balance is askew and getting worse, because the practice of software development is evolving so rapidly. Likewise, administrators are quick (and right) to point out that in smaller companies the balance is askew in the other direction.
I think his main point is that bad admins are a bad thing, and that management often sees bad admins as good admins, because bad admins generate more sturm und drang. "If people are complaining about things being shut down, there must be some security goin' on. If they're not complaining, what did we hire these guys for?"
So unruffle your feathers - if you're not allowing your developers to host outside accessible websites on their desktops, he's not talking about you.
OTOH, if you don't know enough SQL to understand a script that has been submitted, and you reject it because it is not indented properly, remain ruffled - you are the problem.
Comments on an ignorant rant (Score:3, Insightful)
1) The list of so called administrators here is ridiculous, you can't include network/system administrators in with package vendors(what ever the hell they're supposed to be) or blame the whole thing on J2EE because it has architecture problems, that's not admins fault. As a side note, forcing developers to do things they don't want to(document code, plan before they start etc) is a necessary evil since most coders including myself don't want to do these things.
2) The idea of developers doing their own administration is to be honest laughable, if this guy thinks that administration slows down development imagine what it would be like if the people who were supposed to be coding were doing it, even if it didn't take all their time they'd be focusing the results on their needs not on the needs of the secretary next door.
3)As for getting root permission on anything or being able to install your own software. Being able to code does not make you qualified to run a system even an internal test system(assuming you want it connected to the internet). Even on a windows box giving root access to anyone(even developers) can be a serious nightmare. In my experience fixing the computers of people who know something(ie developers) is much worse than doing it for people who don't since other than the usual crap they don't futz with their pc's much. Supporting a machine with random software and random configuration is hard enough when it isn't mission critical, ask tech support.
4)As for Roaming profiles, the reason people set them up is because that way, when you hose your machine(which 90% of people will do either because of ignorance or bad luck) they can just roll you out a new one without having to recover the data from your old one.
That said any system administrator who tells you "it's only two weeks worth of work" under any circumstance beside an act of god(there is no reason you shouldn't be able to get your data back but you can't) should be canned and even under the act of god circumstances they should be apologetic.
incompetence is the only problem (Score:5, Insightful)
The only real problem in the article is incompetent administrators and incompetent management. Interestingly there are no incompetent developers in his world.
Delaying a multi-million dollar project is never okay for a competent admin. Competent management would never allow such a thing to happen either, or such an admin to remain employed. Missing backups should be an instant pink slip too.
Unfortunately most developers are no more competent than most management and most admins. Most people are mediocre by definition. Paranoid admins are no worse or more common than managers who don't do anything but protect their fiefdom, or developers who know nothing but driving a particular gui.
In technology we'd all be better off if we understood computer fundamentals better, but we can't all do that. Very few of my developers have any acquaintance with microcontroller programming, but studying that is a part of my understanding of how computers work. Most of them have never touched Unix, or any free tools either as far as they know, but knowing those makes me a better admin.
Dealing with multiple administrators is a pain. Modern systems are large and complex, and complexity increases exponentially with size. You're going to have to deal with multiple administrators, and modern projects need a project manager.
I worked in the office of the network architect for a fair sized company, and he spent hours at a time on calls making the network work, sometimes days. He also implemented a VOIP architecture that saved the company hundreds of thousands of dollars in the first year. None of the developers would have been able to do his job. They didn't have the technical expertise, nor would any of them have been willing to troubleshoot the global WAN around the clock.
One director who wanted the NetArch to make a client's VPN to work right now, threatened to call the CEO because it didn't. Never mind that the client had refused to give us any of the information we had requested (weeks before) to make it work. We had done the best we could, and the final problem turned out to be one with the client's configuration, which we weren't given any information on. One client finally agreed to call their tech support, who pointed out that they needed to use a special acces point coming from NATted environments. Simple for them, impossible for us.
A friend worked in a company without a dedicated admin. This startup full of brilliant coders got their FTP server cracked, and someone downloaded all of their work. Maybe it was one of their competitors, maybe not.
Backups are a pain, and none of the development servers I've seen were actually backed up regularly by the developers. These were the same folks who insisted on running Visual Source Safe without licenses, and just didn't have an administrator, so when they had to fix the database or roll back to an earlier version so that people with Macs could use it, they were out of luck. They never bothered to learn to use their tools, so they encountered the same problems over and over again.
When they wanted firewall holes for AudioGalaxy, or for me to give them software that they were unwilling to buy, or when they opened another virus infected email, I was at fault. When the dev servers failed, even though they existed solely to give the developers total control, it was my fault. When the VP wanted a deleted email back, we had backups though, unlike the development servers maintained by the developers themselves.
I may sound cranky, but I was always cheerful & respectful of my developers. I knew that they just wanted to do their jobs, and didn't know how things worked.
The author reports that many developers feel that administrative burdens are halving their development efficiencies. That's meani
Re:We need more planning and less coding. (Score:3, Insightful)
1) thinks 'password' is a good password
2) thinks telnet is better than ssh
3) likes world writable permissions on the deployed code
I've worked with developers who have said or done these things. These developers also designed code that was inherently insecure and was exceeding hard to secure 'after the fact'.
At work, *nix dev boxes are locked down almost as tightly production systems. This way, the developers know what kind of permissions their code will have when it is deployed in production.
I'm sure some of the developers would be more productive with root access. But I doubt the team as a whole would be any more productive.
Good points.. but (Score:2, Insightful)
Re:We need more planning and less coding. (Score:5, Insightful)
If idiots could fly, you'd be an ass-tro-nut!!!
Why Admins are Firm: Dump and Run Deveopers (Score:5, Insightful)
The developers then work on their little projects, get them finished, and DUMP IT ON THE ADMINS and RUN AWAY (off to the next exciting project). Yes, the developers are typically able to coble together their cool Java Beans J2EE to
Unfortunatly, for the sake of the users and the corporation, the Admin ocassionly has to say, "No". Believe me, I'd like to let everyone install EVERYTHING and play all day and all night. Innovation, innovation, innovation to the extreme. And that security stuff, who needs it. Just turn off all that annoying ActiveX security for the ease of development. We wouldn't want to slow anybody down. Also, disable all the Java security and let applets connect to and from anywhere. Ditch it all, after all the world is a safe place. There aren't any bad people out there creating hostile ActiveX and Java applications, plus the firewall and virus checker will save us, right?
Systems must function for the sake of the customer and the customer's business. For this to happens, there must exist a well understood development framework and compotent people to manage and maitain the framework. Without this, you are in the wild-wild west of systems and are heading for the land of perpetual system rewrites
I am an ex-developer admin so I work with my developers to help them logically design their applications and database objects. I even tune their SQL, but I resist letting them go off in "cool" direction of the week, just so they can add a new line to their resume and dump yet another one-off application on me which is guaranteed to misbehave and be nearly impossible to upgrade.
Re:We Need Less Planning and More Coding (Score:5, Insightful)
And every developer with experience knows that if you took your client, beat their head against the wall for weeks, then finally cracked it open with a mallet, you're not going to get a specification to pop out. Even with the prototypes for demonstration purposes, if your boss/contract/whoever (how often are YOU in a position to do this) clamp down on the specifications at some point in the development cycle, your development cycle will never leave the "chasing new specifications" stage.
Of course, its even worse when you're doing an in-house project and your boss is the one who decides that it needs to reach a "stable" point... but, "Oh, by the way, we haven't used it for two months because the address on the bills it prints is a quarter inch off from the window on the envelope. We really need to work on this billing part of the system." Thanks a lot, boss!
Re:We need more planning and less coding. (Score:3, Insightful)
I think there is an issue to think about here:
You need to adequately communicate the simple fact that though you are forcing a developer to run a server under formally more limited restrictions which is in their best interests to minimize risk, they are still responsible for unholy code (the kind that lives on forever because it can't get hacked). The "if you don't understand it, that's not my problem" comment is uncalled for and is precisely what this article is all about.. don't be an asshole.
Re:In all areas (Score:3, Insightful)
See the problem with these people though is that they are idiots. You get that in any field. They were probably ex-teachers who took 2 or 3 courses on learning Microsoft Office and suddenly got thrust into the job of being the network administrator. At least, that's how it was 15 years ago when I was in school. The Basic programming teacher was also the LAN admin. He wrote his passwords under the keyboard so we could conveniently install software without his knowledge.
bah! (Score:3, Insightful)
This is like a Ford exec writing about how Chevy sucks. Taken with a grain of salt, it is.
Re:We Need Less Planning and More Coding (Score:3, Insightful)
So developers will keep doing what the author of the article describes: working around the bullshit to actually get things done.
This is exactly on target. Other replies here are bashing your comment, but this is definitely true. Frankly if we didn't circumvent the red-tape bullshit at my company, we would cease being productive.
The bottom line is that to be efficient and productive you need to operate under the radar of the IT overlords who have this obsessive vision of making all of the computers in the company 100% identical. The truth is that the needs of my group are NOT the same as the needs of every other group in the company. If I need a certain tool, and IT doesn't support it, I will do whatever it takes to get it, install it, and use it.
Don't believe me? well our group is in the rather fortunate position of being in a different city than the rest of the company. Since we are only a handful of people, and since we (thankfully) report to essentially the top of the food chain (directly to executive staff), we get to operate in semi-autonomous mode. Officially we are under the IT overlords thumb. However the reality is that we do our own system admin, despite the fact that it goes totally against the corporate rule. If we need another tool, we just go get it. If we need another machine, we acquire it - one way or the other. We don't expect support from the "mothership" because we know there is essentially no one there who has a clue.
Bottom line, productivity skyrocketed. We are beating time-to-market against every other group in the company. We are going to beat the time-to-revenue record of any previous product (ie. making the most money from the product in the shortest amount of time since the project started).
If the IT overlords would understand that their purpose is to give whatever is needed as soon as its needed, to facilitate and accelerate project development, instead of giving red tape and bullshit because they don't want to do any real work, the rest of the company would be better off also.
What a hatchett job... (Score:3, Insightful)
I currently look after a team about 100 developers, testers and technical writers. The situation I walked into was one where each software group had control over hardware budgets, admin, the whole shooting match. Needless to say it was an absolute mess. Permissions were all over the place, NFS and Oracle servers haphazardly brought up, no fault tolerance, poor performance and reliability, the list goes on. I took the approach that my job was to help these folks get software products out the door as efficiently as possible. With that in mind, I sought to make sense of the environment and let the developers write code and the testers test it. They still had some control over day to day stuff, but I manage their total infrastructure for them.
This involved retiring unneeded systems, consolidating storage and servers, upgrading some machines, implementing some proper change management, auditing, performance monitoring, backups, standard system images etc etc. We are not there yet, but needless to say that there has been a significant improvement in the last six months. The coders seem to be happier and the infrastructure is more stable. That sounds like a win-win situation to me.
I think the key for any admin is to make a partnership with your users, understand their needs and address their pain. I agree that some admins rule with an iron fist and have users cowering in their presence. This is counterproductive. But so is having no admin staff in a development centre. If the chance arose, I would actually like to work with the author of this article and perhaps help ease some of that pain
Re:Arrogant developer crap (Score:2, Insightful)
Please understand your place. You are basically one level of abstraction away from being an end-user. Your knowledge represents a subset of the average developer's knowledge. We can do your job, and you should be greatful that we have other things to worry about than the 8th grade bullshit you do!
Re:In all areas (Score:3, Insightful)
You mean like this? [f-secure.com] The vulnerability has been patched, but that doesn't mean the architecture isn't vulnerable to viruses anymore. Not to say that the administrator in question is the most brilliant guy in the world, but at least he stayed tuned in to his virus warnings.
Hmm (Score:2, Insightful)
Slower development times
Increased communication overhead
Increased dependencies
Slower rates of change
De-skilling of the workforce
Extra manpower needed
More paperwork
They forgot to add:
Servers set up correctly.
Servers patched
Less hacking incidents
Coders unable to browse pr0n and get away with it...
Re:We need more planning and less coding. (Score:5, Insightful)
A 600hp car with the keys locked inside isn't good for much other than looking at. Who cares about the uptime of development boxes? If your developers are so incompetent that they can't keep from permanently mucking up their own machines, then your company has other things to worry about.
Developers don't need root access. Simple.
For what? Give me one good reason why.
tcpdump/snoop/ethereal - sometimes watching the packets on the wire is orders of magnitude faster than any other option for debugging a network app.
OS bugs - they aren't "suppossed" to exist but we all know that doing weird shit to a system can push it off into corner cases that the OS engineers haven't handled so well. Sometimes once in the corner, the only way out is to reboot. Until a patch shows up, letting the developers reboot the system after they dork it up trying to debug their own code is a real time-saver. Especially if they are working 2nd or 3rd shift when finding a sysadmin to reboot the system can be difficult or even painful for all parties (just love gettings those 3am pages to reboot a computer and if the system is classified you have to physically come in to do the reboot).
That's two reasons. There are more.
Re:We need more planning and less coding. (Score:3, Insightful)
Giving all the programmers root access meets the first test, but fails the second one.
---
I used to be a programmer. I moved to system administration because "not creating new problems" takes time for a programmer. The ones who don't care what new problems they create are "more productive".
How Not to Suck As An Admin (Score:2, Insightful)
1)Provide the best development environment you can.
My company is not fond of buying development boxes. I see them as an absolute necessity. I scrounge hardware, software, and what ever else I can to make sure the devs have what they need.
2) Give appropriate access.
Dev's need root access to the development box and DBA access to the development instances. They get what they need as soon as possible, I don't get interrupted to restart Apache 10 times a day. Dev's don't get root to production machines. Processes are not run as root.
3)The BOFH doesn't work here
I'm a service provider to the devs, our end users, and ultimately our customers. It's not a kingdom, it's a big freakin' amusement park. My job is to make sure the rides all work, the popcorn is fresh, and the soda is cold. I will install tools, modules, kernel patches or whatever else you need. All I ask is that you test the heck out of it before you ask me to put it in production.
4)Commumicate and Collaborate
Tell everyone what you've done with performance tuning and why. Explain why you've had to freeze database changes. Tell them why you had to disable port xxx. Ask the devs for help with those nifty widgets to read the database system tables. Send them the SQL performance tuning tips you find so the SQL can be done right the first time. Look for ways to make the devs life easier. It will make your life easier, too.
5) Stay Professional
Projects get delayed. Things break. Tempers flare. Keep your cool at all times. (I have a very hard time with this, BTW) Do not add to the problem. If you screw up, admit it, and fix it. Now. If you find a dev's mistake, handle it in a diplomatic and low key way. Nothing sucks worse than an us vs. them IT department.
Blue
Re:We Need Less Planning and More Coding (Score:5, Insightful)
Funny you should mention that. Every single insecure, unstable, unscalable, unmaintainable project in my datacenter got here precisely because there was no "deployment admin" (or equivalent role) involved in the planning process.
I wouldn't dream of telling you how your code works internally, but if I find out that you have everything running as root in your lab because it's easier that way... well, I'm sorry, but there's no way in hell you're getting that design into my production datacenter. I spend enough time trying to fix poor architectures after the fact as it is, without having a whole new craptastic one-off shoved down my throat by some PHB who blessed the damn thing three months ago and is too busy being smug about his fait accompli to hear me explain how thoroughly fucked up his decision was.
Re:no http server on port 80? (Score:2, Insightful)
- anyone who wants to do harm is capable of running the service (whatever service, not just https) on port 443 on some box
- but your normal user/developer/etc (especially consultants connecting to their own office) who need SSH, imap, etc. access can't do so.
So you're only reducing the functionality of the network while adding no security. You're probably costing the company thousands of dollars a year because consultants who are paid by the hour can't access the materials they need to do their job quickly.
- Erwin
Re:No, we need more GOOD planning (Score:5, Insightful)
I think the same is true of other areas of society, too. Armies are not led by committees, but by a strict hierarchy of responsibility, with one person responsible for the group below them, as you go up the chain. Not complete democracy, but it's the structure you need when you have to get things done. A good commander will listen to the feedback of his troops, but ultimately make the decision and be responsible for it. I find it a bit ironic that Western culture embraces democracy and distribution of control (in theory), but tends to use an autocratic structure when things are critical.
I'm not a coder, but.... (Score:2, Insightful)
Yes, administrators can cause a hit to productivity (good ones can actually help productivity), but there are other gains to be had. And any person responsible for planning software development should of course take that into account.
Yes, there are also a lot of administrators who don't know their ass from their elbow, but that is a function of the person not the position. Administrative overhead should be kept to a minimum, of course, but administration is a key component to developing a stable, scalable, mature company.
This article is little more than a libertarian fantasy that just demonstrates why coders need to have administrators (and supervision). It sounds like coders are looking for someone to blame for the whole offshore fiasco. The offshore fiasco is in part due to coders' over-aggrandized opinion of their ability and place in a company as well as the obscene greed of CxOs and shareholders. There are two sides of the issue.
The bottom line is that if coders want to produce code unconstrained by any external source, then they should do it from home on their own time for open source projects or their own projects, but if they want to have a job at a successful business, then they should grow up, increase the scope of their perspective, and learn to play well with others.
Please note, these observations only apply to coders that believe what was written in this article. I have had the privelege of working with many talented, mature, and highly productive coders. I hope they are the norm rather than exception.
I just had to chime in here. (Score:3, Insightful)
Politically speaking, being a sysadmin is like riding a k-mart skateboard in the middle double lines of the road with traffic buzzing back and forth both ways. You have to deal with everyone in the organization from shipping to the executive board and insulting the wrong person, even unintentionally could cost you your job. It's more than just a tech job, it's a lesson in both diplomacy and technology.
Developers on the other hand just have to deal with their project managers. They don't have to come into work in the IT persons uniform (polo shirt, jeans, pager, cell phone) They don't have to be the "first ones" in the office at 7:00am because the CEO is a workaholic. On the same note they are usually the last ones to leave, because someone absolutely positively needed something done "ASAP" although the trouble ticket was submitted at the end of the day.
I've worked with nice devs that were smart and understood what my job entailed. I've also worked with devs that were just so arrogant and annoying that it made me grow a grey hair or two. Either way they were always treated better than us corporate whipping boy gophers IT helpdesk people. I don't think the author has any place to bitch.
Before and After (Score:5, Insightful)
After a change at the CIO level, we now have multidisciplinary teams - programmers, admins, DBA's - working together to prevent such expensive oversights. The problem with the article is that it romanticizes the past. How many of us have had to live through DOS programs whose programmer assumed their program was the only one running? Today, more than in the past, we cannot afford to have walls built between the various groups in IT. The costs of failure are too great.
Re:Have you noticed (Score:3, Insightful)
Guess you didn't even stop to consider that maybe he was a Cisco admin, and he meant he needed to find someone who knew what Unix machine the ftp server was on?
Nope, because clearly, you've no idea what you're talking about.
Until developers learn... (Score:1, Insightful)
(and so on) - until then I'm not gonna trust them or give them any privileges.
democracy is intentionally bad process (Score:3, Insightful)
From my 7th grade history class -- what we've got here in America is a system of checks and balances. For example, the House, the Senate, the President, and the Supreme Court each get a chance to cancel any bill becoming law. Only if all four of them approve will it stay in the books. Any sane manager would look at this system and say, hey, you've made it impossible to get anything done -- and that, of course, is exactly the point. We give governments power over us is dangerous to give to anyone, and that power should be damn hard to use.
Once the decision is made, though, it should be carried out effeciently, and that bit is done by normal chain of command, not committee. What you're pointing out isn't irony. It's the right tool for the right job.
Re:They should be there to help us (Score:2, Insightful)
- RustyTaco
Re:no http server on port 80? (Score:5, Insightful)
But that's exactly the point. Developers should have their own little playground to play in, where security should not be the primary focus. If you are concerned about security, make the computers in the development lab join a private network that is separate from the company lan.
Then everybody will be happy. You won't get calls from developers who are more than happy to fix their own troubles. Developers won't have to deal with you. So, what's the problem?
Oh, you mean you do development work on production boxes? Well, shame on you. How much is a nice development box these days?
Re:In all areas (Score:3, Insightful)
As someone else pointed out, tou could have a vulnerability in the Flash player.
However, my attitude toward running a Windows environment is that it (Windows) really can't be relied upon to be secure, stable, or virus free, so you pretty much have to use disk-imaging software like Norton Ghost regularly anyway. In other words, my answer to "what if it has a virus?" is:
"So what?" Any Windows-based network architecture which is fatally vulnerable to virus infection is inherently flawed and should be replaced as soon as possible.
Sysadmins are bastards because they must be. (Score:5, Insightful)
Yes, I'm a corporate bastard sysadmin. If I weren't a bastard the company would need four more of me to clean up the constant messes the developers create due to their complete lack of consideration for company resources. I'm not talking about the legitimate development work, either, but rather the pure crap that these guys do with their systems that end up introducing all sorts of malware into the internal LAN.
Just stop your bitching, and remember you're not at home. This isn't your network. It isn't your computer. It's the company's. Try to respect that a bit more, m'kay?
What the real problem is... (Score:2, Insightful)
This guy is venting one-sided only! Reality Check -> Developers don't know jack about SysAdmin nor DBA duties, nor the bigger picture!
I'm a SysAdmin and I've seen it all! I just cleaned 253 spyware packages off a 'developers' laptop. Developers have admin rights but they are always breaking their systems! Last month, we caught a consultant with the MS-SQL Slammer worm on his laptop. The laptop was not a company laptop but the developers personally owned laptop! That's a big time no-no to connect uncontrolled computers to the enterprise network. The consultant was fired on the spot because he brought down numerous production MS-SQL servers. All because he wouldn't play by the rules. Now we have MAC address checks before an IP address is doled out. Took months to collect all the appliances MAC addresses. (yeah they should have been patched but the worm would not have been introduced normally either).
Granted, the developers sometimes have problems, such as a non-production server going down (dev or QA) and it's used for source code control. This means they can't work, but then they report it to the help desk at 6:45pm on a Friday and the help desk cannot open a priority ticket for a non-production system. A priority ticket pages the world and kicks in emergency restoration managers and various conference calls. After this happened a couple of times, it came down to communication!
i.e. we hold daily outage meetings and weekly meetings where those responsible for various systems have to attend if their system was down. Root Cause is determined at these meetings as well as procedure refinement.
So the developers were told they needed to have a seperate source control server which they needed to purchase for their department. Then it was assigned a priority support level which they again had to pay for.
Yeah, the money is funny money, it's all budget dollars. But you can't ask the SysAdmins who pull 60 hour work weeks without extra pay and they are on call all the time to work for nothing. This money goes into keeping them trained and paying for days off after a big outage.
Frankly it goes like this:
Production Server (financial transactions) - QA Server (duplicate of ProdSrv for testing) - Dev Server (Latest wiz-bang code testing). - Source Code Control Server (some ability to run as a backup to the Dev Server)
Change control prevents any code from going into production without proper red tape paper work and lab testing. It's your ass if you rollout a big system riddled with bugs into production. The change control process may be a pain but it will save your butt.
The production servers go down and literally hundreds of users performing financial transactions can no longer do their job. Within an hour of down time, millions of dollars could be lost!
QA or Dev server goes down? After Hours? Nobody cares except for the QA developers. It will be fixed during normal business hours the next working day.
Source Control server goes down? After Hours? Well it was decided that was important because developers work long hours and under tight schedules. So therefore, these boxen are given priority as it directly impacts the development teams productivity. BUT THEY PAY FOR IT! IT'S NOT FREE!
Whole seperate lab setup with isolated network for testing. It's easy and convienant for developers to get into the lab. They just need to schedule time.
Yeah it sucks for everyone. The developers hate the red tape and the sysadmins. The sysadmins hate the developers, the engineers hate the users, the users hate the help desk. Everyone hates the consultants and the entire idea of outsourcing. So we end up shooting ourselves and IBM comes in and half the staff is let go. Part of the reason for the outsourcing is due to enterprise management being unable to get change happening fast enough! (several mergers of compan
Re:We need more planning and less coding. (Score:3, Insightful)
we need more Generalists (Score:3, Insightful)
Specialists can only exist because of the foundation laid down by generalists. Sadly, corporations do not see this obvious truth and value people with only very narrow skillsets because they are able to assign a title to them.
How many individuals can completely design, implement, test, install and manage complete end to end systems spanning desktop GUI, web, database and distributed servers these days? Not many. And the specialists will continue to drive these resourceful people out of organizations due to self-interest.
However, these same companies will soon begin to wonder why projects are not elvolving at the rapid pace they did in the "old days". They will just assume that they need to hire more specialists, and the problem cycle begins again. A top-heavy organization like this cannot be salvaged.
Do not confuse the term "generalist" with the much overhyped term "architect". An architect is a person who is simply all talk and cannot do what they say. A generalist can perform all the functions of an architect and can back up what they say with action. Generalists are the people that actually get the job done.
Re:Declare independance, but be smart about it (Score:3, Insightful)
The big failure there was that it was politically impossible to tell the admins about the development network. Has it been possible, the admins could have set up a firewall and probably intrusian detection to keep any problems localized.
The failure was in not recognizing the need for a developer's network in the first place. More generally, in failing to meet the needs of that department to the point where they met it themselves by stealth.
Re:we need more Generalists (Score:2, Insightful)
1 - Projects need 'Managers' and managers should be 'Generalists' so that they can see the whole picture and coordinate the 'specialists'!
2 - The problem is that you can only be a 'Generalist' and a 'Good Manager' if you make a carrer from developmento to management. But the companies usualy hire managers that don't have real development experience instead of promoting developpers inside the company.
3 - Everyone this days (including me when i was in the same situation) wants to make a carrer in 5 years! Every time you are real productive you move up, and need to learn new skills, then you are never really productive for more that 6 months every 2 years.
4 - It can happen that you move ??up?? until you reach your level of incopetence and can't get promoted any longer
Re:Arrogant developer crap (Score:3, Insightful)
+1 Funny or -1 Flamebait, I can't tell.
No, but seriously, the opposite is true. I've yet to meet a sysadmin who couldn't do some programming, whether it's 150 lines of Korn Shell script or 5000 lines of Python/TK. Inclination is all that really prevents the administrator from being a programmer. A good sysadmin will also understand the system as a whole and how the work it does interacts. In contrast, few developers understand any more than their own little bit of an application, which is but one of many applications the business runs. The developers that do quickly become system architects and stop coding day-to-day, because they're too valuable as "big picture" people to get bogged down in details.
Developers tend to assume, for example, that they have exclusive use of a machine, whereas an administrator knows that the database is busiest at a certain time of day, that the network is saturated on that segment at another time of day, etc etc. A developer thinks he can rely on a version of a library being there, an administrator knows that that library version has a bug in a function that the developer doesn't use, but another application on the machine does. A developer thinks his app is behind the firewall so he doesn't have to worry about buffer overruns, an administrator has been bitten by that mistake in the past. And so on and so on.
Frankly, if you have competent system architects, engineers and administrators, coding is no more difficult than data entry. It's this truth, combined with the prima donna attitude of developers, that's driving jobs offshore.
"Specialization is for Ants" - Robert Anton Wilson (Score:2, Insightful)