Tips on Managing Concurrent Development? 256
from the dealing-w/-many-hands-in-the-cookie-jar dept.
"Take, for example, the extreme case of something like Linux (not only concurrent development, but geographically distributed development), how is this managed? One solution we were contemplating was to try to do an 'air traffic control' type of sequencing and conflict resolution. As early as possible in the development stage, we try to identify what will be finished when, and assign a one-up sequence number to each patch. Developers then know that they will be patching against the baseline that was patched by the patch with the previous sequence number. It is hoped that this prevents a lot of rework of patches. A potential problem with this approach is the need for a responsive central authority to assign sequence numbers. Also, such sequence numbers may have to be rearranged in the face of last minute advances and setbacks in developer progress. Despite careful scheduling and detailed design, it may be impossible to know the exact check-in sequence of patches more than a week or two in advance.
Will such an idea be successful, or is it fatally flawed? Are there better solutions to the problem with less effort? Are we treating symptoms and not the disease (i.e., should we be planning better so that we know patch sequences and dependencies early on)? Management likes to keep staff productively occupied and working up until deadlines, so this usually means a lot of checkins within a short period of time, rather than staged checkins. Can checkins be spread out over time while keeping developers productively occupied?"
Tips on widening pages! (Score:0, Insightful)
12 people a large project ??? (Score:5, Insightful)
Of course some of these problems sound like lack of planning early in the game...
For example changing headers that two developers need... The only headers that two groups need should be interface headers, these should be set early and not need a lot of change, with any change taking both developers changing the code internally...
Another note, I get really worried when people say that process problems only show up at the end crunch time. If it is crunchtime it is time to use all of your processes, because the processes should be designed to produce the best bug free code the quickest... otherwise it shouldn't be in the process...
That is just my 2c worth however
Modular Isolation (Score:5, Insightful)
I know this isn't really easy to do (can't be done retroactively), and doesn't really fit all cases (such as near a release when there is a lot of chaos), but it's the only elegant solution I know of, all the rest are more brute force.
Why are patches applied multiply? (Score:5, Insightful)
If your problem is with people overwriting the changes that previous submitters made, then you've got a very different kind of mgt problem -- one that can be solved by getting people to use the tools they already have. CVS, for example, lets you merge the current branch head with your working copy, incorporating any changes that may have been made since you checked things out.
Submitters should always diff their current code with the head before commiting a check-in, to see if they are breaking previous changes. This kind of practice is more important when schedules are tight, and you shouldn't let people off the hook because they were in a rush or some other lame excuse.
--tsw
This sounds like an architecture problem (Score:3, Insightful)
There will always be some cases where there's a little overlap, but if your architecture makes these overlaps rare, it becomes relatively easy to see them coming when you lay out your schedule and plan around them.
Sounds like no one is in charge (Score:1, Insightful)
And 12 developers isn't large at all. 400, that's large.
The simple fix? Back out any check in that fails a merge or fails to compile, and publicize it!
"Ha ha! You broke the baseline again" is a wonderful motivator for doing your fucking job correctly!
It will only happen once or twice to your good programmers before they learn, and if you piss off those who don't learn and they quit on you, do you really want programmers around who are continually breaking your baseline product?
.2 cents (Score:2, Insightful)
1) Minimize dependencies through refactoring.
2) Try to avoid branching as much as possible.
3) If branch, minimize the lifetime of it.
4) Before merging back a branch into the main, merge the main to your branch, recompile, test and then merge back.
Just some thougths...
Re:Source Control + Automated Build & Test (Score:3, Insightful)
You might try it, but you'd probably find it lacking. Bugzilla is far from a good bug tracking system. Actually, let me clarify that -- it's a great bug tracking system, if you're tracking bugs in Mozilla. It's horrible for anything else. Data is hardcoded in source files, and if you want to configure it for non-mozilla bug tracking, then you have to edit the source directly. Red Hat and GNOME have obviously put the time in to do so, and have got good results, but for a small business like ours, we couldn't justify the manpower needed to get the system up and running, so we were forced to go with an alternate solution (one I hacked together in PHP one evening -- it may not be pretty, but it works, and gives us 90% of what we need).
What you are doing isn't concurrent development! (Score:4, Insightful)
The trick is to decompose the development task into chunks that are in fact parallelizable. In turn, those chunks may have sub-chunks which are not parallelizable; those chunks should be perhaps done by one developer, in the correct order.
No developer should wait around for another's patch, and nobody should develop anything that he or she knows will soon be invalidated by a forthcoming patch so badly that it will have to be substantially reworked. If a unit of work depends on some forthcoming patch so badly, a developer should find something else to work on until that patch arrives. How you know that the patch has arrived is by monitoring your e-mail, or scanning the version control system for changes. The other developers should know that someone is waiting for their patch in order to do the next, dependent part of the change, and broadcast it to the team when they are done.
Division of ownership/responsibility (Score:4, Insightful)
Before I describe how we handled this situation, I want to stress the fact that if someone intelligently devides the labor according to how the changes will affect other parts of the code, the need for developers to sqabble over specific changes in specific files should be eliminated.
Labor should be devided at well defined "interface points" where the additions/changes to the interface can be done quickly, satisfying the needs of other developers requiring those interfaces to build against, and then completion of the underlying code can be done with little interference or effect on others.
In short, devide work along interface boundaries, and stub out interfaces with enough code to allow compiling, while developer(s) continue to actually implement the code behind the scenes. Thus, swapping in the actual code has no effect on any one else code, exept that the stubs are now full implementations and work correctly.
Ok, so what happens if the devision isnt clean and you have two people working on the same file?
NOTE: When I am talking about file granularity, or developers "owning" specific files, you can also substiture "subsystem", etc. Sometimes developers are working in entirely different areas of the source tree for the most part, and it makes sense to assign an entire sub tree to a specific developer. This is the devision by interface, which is the usual case.
What we did was assign specific files to specific developers, who have the most work to add/modify to the file. When other developers require changes to a file "owned" by another, they perform the merge, which is verified by both sides to work correctly, and then it is checked in by the "owner".
This was all accomplished using locks (file checked out, ClearCase) and multiple views. The locking of files was a benifit, as it prevented accidental overwrites of other peoples code. Once you check out a file and lock it, no one else (short of the administrator) can check in a modified version and clobber your changes.
A short scenario:
Alice: owns file/tree "something.cpp"
Bob: owns file/tree "modified.cpp"
Tree is something like this:
RootBranch
|
|-- Devel
|
|-- AliceView ---- BobView
| |
[code] [code]
Alice and Bob are both working in a development branch. Alice has the files she is modifying and "owns" checked out and locked. Same for bob.
Bob realizes he needs to make changes to "something.cpp" to support some changes he is making in "modified.cpp".
He checks out a temporary version, unlocked, of "something.cpp" and makes the required changes.
He then notifies alice of the changes, and using the automated merge features she adds his changes, manually resolving conflicts if necessary.
Bob changes his view to use Alice's version of the file with the rest of the code from his view. He builds and verifies that everything is working correctly. Once this is verified, Alice can check in the changes, and Bob can now use the most recently committed version and continue on his merry way.
What this boils down to is basically enforcing ownership through locks to prevent accidental overwrites of others code, and defining clear lines of ownership so that a change is only accepted and merged when the person responsible for that code has tested it (in addition to the developer desiring his minor modifications be included)
Re:12 people a large project ??? (Score:0, Insightful)
Lack of planning is often a management problem. I know where I work, several times we've had problems where management promises what amounts to essentially working versions of the program midway (or earlier) through the project development. This is equivalent to letting someone start moving into their new house halfway through building it, which is just nuts. Unfortunately, in software, you then end up putting in the most horrible hacks just to get things to work for the ridiculous milestone deadline, planning and modularity and good design go to pot, and the program becomes a painful mess to maintain and extend. Although its improved a lot, partially due to lots of complaining from me, and management slowly sort of starting to realise the problems it creates.
The correct way to use CVS (Score:2, Insightful)
CVS is more than adequate for development on small to mid-scale projects. There are several things that you need to keep in mind, however.
As a side note, problems with header file commits late in the development stage of a project is more likely do to poor planning and design than to the configuration management tools you use. C header files are used to specify interfaces to various modules of a program and should not change late in development, this indicates either a deficient or non-existent design.
Use a transaction oriented vers. sys. and hire a.. (Score:3, Insightful)
1. Create as granular files as possible. If a C file is often modified by more than people, then split that file into 2 files.
2. Use a transaction oriented versioning system (such as IBM's CMVC). CVS is not up to the task of larger organizations with plenty of concurrent development in the same files. You need to be able to roll back an entire transaction, not only a file change! CMVC provides the notion of transaction, either the patch happens, or it does not happen. It also allows you to lock the files when you are ready to commit.
3. Hire a person to take care of the build. It is cheaper to hire somebody than having your developers wasting time doing it. The build manager will be responsible for serialization
and rejection of atomic patches. Somebody can
still brake the build, but your developers
don't have to worry about it (only the breaker, who will have the builder's finger pointing at him and get the shame!)
4. Accept that manual merging is a matter of life. Merging offers also the chance to do code inspection, which is a good thing.
Re:Fly by the seat of your pants. (Score:0, Insightful)
Privation of source code control is one of the hallmarks of a
sw development disaster waiting to happen. Expect to experience excruciating pain when this practice finally catches
up to you. 8^(
People have to trust the system (Score:2, Insightful)
If you have a regular (hourly/daily) build that smoke-tests and reports the results to everyone, people will be more willing to sync to the latest and trust that they won't lose lots of time with problems. Embarass anyone that breaks the build, make sure that everyone understands it gets fixed ASAP when it breaks, checking in broken code is NOT ok; and then people can sync every few hours or every day, and the problem simply goes away.
General Suggestions (Score:4, Insightful)
That's the a few points that I've found to be helpful in my professional work. Your mileage may vary.
Good luck,
--Mid
"Disappearing" fixes (Score:1, Insightful)
It sounds to me as though your developers are NOT updating their local ("sandbox") versions before they merge their changes back in.
Developer A: checks out file, starts work.
Dev B: checks out same file, starts working.
Dev B: finishes, commits changes to CVS.
Dev A: finishes, commits changes to CVS without updating first.
Developer B's committed changes are GONE!
Make it easy on yourself, and do NOT let multiple developers check out the same file.
Software engineering (Score:2, Insightful)
Well designed code avoids these issues (Score:2, Insightful)
Brian
The CVS Way (Score:4, Insightful)
[I am not a CVS guru, I just use it.]
If you have to apply patches multiple times, then you're probably patching branches, and developing in the branches. The "CVS Way" seems to be (corrections welcome) to develop in the default branch, and to tag the tree at drop-points -- when you ship the code. If you need to support an old code-drop, you turn the tag into a branch, and then patch the branch.
If you have too many delivered branches being supported at the same time, perhaps you should upgrade those customers to a newer version of the software. They'll appreciate, and it will simplify your situation.
The develop-in-a-branch-and-ship-the-default is appealing, but troublesome.
Otherwise, it sounds like your developers aren't playing nice... developer A patches the tree, and developer B goes to commit his changes, but gets told that there are conflicts and that he needs to update. Not wanting to deal with the conflicts, he copies his important files to a save spot, updates, copies his "important" files back over the top of the conflicted files, and then commits the whole thing, effectively "rolling back" the patch.
If this is what is going on, you need to educate your developers. With a stick.
Over the years, I've discovered that a significant amount of heartburn I may have with CVS comes not from any deficiency of CVS, but from the fact that I frequently fail to use CVS "properly" -- meaning, of course, "as intended".
Re:SourceForge 3.0 Enterprise Edition (Score:3, Insightful)
I run a small development firm and I wanted to use the enterprise edition. I'll pay a few thousand for something, but $30k for 30 people? I think not.
While I can't say the SourceForge is decent (imho looks like a hodgepodge of thrown together junk), $30k for 30 people is not really a lot. Say salary, benefits, overhead, etc., cost you $1k per week per employee (which is likely very modest). Is something like this going to save each developer a week of time? If there's a tool that good, then it's well worth the money. Sure, it's a big one time cost, but even over the course of a year, it really does pay for itself.Parallel Development (Score:2, Insightful)
Okay, I think 90% of the people responding are missing the heart of the question. The original question was about parallel development, not just working on small changes in the same source tree and then synchronizing in the changes.
The best way to solve this problem is by better design of the software. If your software is well-designed, you the only time you really need to do parallel development is when you're maintaining multiple versions of the software. (i.e. service pack 1 and 2 to windows 95, while you work on windows 98 elsewhere).
The way we solve the problems above is by using a task-based change management solution. We use a commercial product Continuus/CM (now Telelogic/CM) to handle both the parallel release maintenance problem, and the manangement-didn't-enforce-good-design problem. This problem can be exacerbated by having almost random changes in the requirements combined with very tight deadlines. Fortunately, having a task-based CM tool lets you roll with the punches.
In task based change management, groups of file changes are grouped into a task. This task is one unit of work, something like "change product name string from a character string to a unicode string" which may involve touching several files. These file changes are brought in (or excluded) as a whole - the whole being a "task". Integration test approves "tasks", and if a developer wants a task before it has been integration tested, s/he can bring it in manually, and get all the updates.
This allows a group to work in parallel with the main effort, by including groups of tasks themselves that haven't gone through integration test (perhaps because they don't work yet, other developers' tasks are needed, or it's just a large change requiring more people to work on it before it can be tested).
Merging is done when needed, this way. One thing you can do in this program is "show conflicts", to show you what merges need to be done - on your parallel development effort, not the rest of the tree.
The merges never really get confused if you use a decent merge tool (we use tkdiff). The only time you would have problems is if everyone is rewriting the file to be merged from scratch every time... And in that case, the sofware design problems are so bad you really can't do anything about it.
The Continuus/CM software, however, is very costly. I think BitKeepr does some of the same things, but unless your company is one which doesn't mind its company secrets being posted to the web, you will have to pay for that too.
aegis is another free tool you might look into. It doesn't have a GUI, though.
From an ex CM Consultant (Score:2, Insightful)
First. If there's only a dozen of you, and you have a branch, perhaps you should talk. Now if you were like some of my former client base who does follow-the-sun development around the world, this gets a little hairy. By that I mean a company like a TI or BT or Philips is passing code around the world and flexibly allocating resources from multiple sites to a project. Serious commercial SCM systems like Continuus/CM, err.. Telelogic CM/Synergy with DCM and CC with Multisite can be used to coordinate. This is one place that open source has failed to penetrate due to the small size of the potential market and the general antipathy that the average developer has to the CM people. Until, that is, they get religion by losing something very important at a crucial moment ;-) Even then most shops can get by with VSS or PVCS.
OBTW: I'm an ex-developler, system admin, desktop video weenie and computer consultant that backed into SCM. I got religion back around the late 80's but it was pre-emptive. Anyway, concurrent development actually comes in three flavors: concurrent by collision, concurrent by release (patches versus next generation) and finally concurrent by platform (Windows vs Unix, etc). Each is handled differently and under different timeframes (including permanently in the latter).
Your post alludes to the first case of a "popular" file that needs checking out a lot like a header file. In a shop your size, this should be handled by parallel checkout notification (a common feature in commercial high end systems) and hopefully a bit of coordination by phone or shouting over the pod wall. If you are in severe RAD mode, I suggest using a shared work area to enforce sequential checkout. Your SCM tool does build a work area doesn't it? If not, hit yourself on the head and ponder why people like me are now on the dole.
Okay, so you decide to branch anyway since you or the other developer are really going to break things with a rewrite. Hey, wait a minute! Shouldn't a big feature change (or small) be considered as a single unit of work where all of the files changed by the feature get moved through the mill at once. You have just re-invented the wheel known as Change Packages, Task Based CM, ChangeSets and whatever else the marketing wonks are using nowadays. Once again the high end tools have this (or should). Continuus had it back around 1997. Rat got it into CC much more recently, but there it is.
If you have a task based system, it makes life much easier to do merges and to put things in or take things out of the build. Being ex-Continuus, I could pound the pulpit about how build managers can check the configuration and its set of included tasks for consistency, but I won't. Instead I'll just mention that graphic history views and a decent merge tool that takes into account ancestor versions should take care of any merging issues.
One more thing. The sooner you catch the branch and do the merge the better. Don't punt it to CM unless they are ex-developers who know the product as well as or better than you. You may also come to find that an integrated Change Request Management system like ChangeSynergy for Continuus (err Telelogic) and Clearquest for CC become very important in your life.