Tips on Managing Concurrent Development? 256
"Take, for example, the extreme case of something like Linux (not only concurrent development, but geographically distributed development), how is this managed? One solution we were contemplating was to try to do an 'air traffic control' type of sequencing and conflict resolution. As early as possible in the development stage, we try to identify what will be finished when, and assign a one-up sequence number to each patch. Developers then know that they will be patching against the baseline that was patched by the patch with the previous sequence number. It is hoped that this prevents a lot of rework of patches. A potential problem with this approach is the need for a responsive central authority to assign sequence numbers. Also, such sequence numbers may have to be rearranged in the face of last minute advances and setbacks in developer progress. Despite careful scheduling and detailed design, it may be impossible to know the exact check-in sequence of patches more than a week or two in advance.
Will such an idea be successful, or is it fatally flawed? Are there better solutions to the problem with less effort? Are we treating symptoms and not the disease (i.e., should we be planning better so that we know patch sequences and dependencies early on)? Management likes to keep staff productively occupied and working up until deadlines, so this usually means a lot of checkins within a short period of time, rather than staged checkins. Can checkins be spread out over time while keeping developers productively occupied?"
nice acronym (Score:1, Informative)
SourceForge 3.0 Enterprise Edition (Score:4, Informative)
I know this sounds corny, being said on a VA property such as Slashdot, but SourceForge 3.0 [vasoftware.com] is easily the best concurrent development environment i have ever used. It was my love for Sourceforge which made me pursue a job at VA Software in the first place! The fully web-based administration hides all the niggling details of commandline cvs tools and makes managing huge projects a piece of cake.
In short, if you haven't been to VA Software's site [vasoftware.com], you don't know what you're missing.
Better Solution: Use CVS or ClearCase Properly! (Score:5, Informative)
We nearly never have merge problems. It is standard procedure that people keep their tree up-to-date with the cvs tree, and thus conflicts rarely arrise. Even at crunch-time, I probably have one merge conflict every 2 weeks, and with CVS - you are notified of the conflict and it is wrapped with CVS comments.
To put this in perspective - while at Oracle with 1000s of engineers working on the same tree, we used ClearCase and it was awesome. The difference here is that there was much steeper a learning curve, and no normal engineers could actually do complex tasks - i.e. create branches etc. We had a complete groud dedicated to ClearCase.
Conclusion:
Educate your engineers - and politely have the senior engineers tell them when they mess up - enforce a policy that people must update the source every day that they plan on checking-in files.
Also - I don't know what CVS versio you are using, but the latest free WinCVS client will not allow you to check in a file with a conflict! It will force you to update/merge/resolve the conflict before updating the tree. I highly recommend CVS and WinCVS due to the ease of use and cost.
Re:I recommend SourceForge OnSite (c) (Score:1, Informative)
Use Continuous Integration (Score:3, Informative)
Write tests (Score:2, Informative)
Source Control + Automated Build & Test (Score:4, Informative)
For the second part, I highly reccommend that you have automated build and tests that run after changes have been submitted. You can see how this is done en mass on the mozilla.org [mozilla.org] site. Also, developers should have access to the same build and test infrastructure on their machines so they can do the build and test before they check in their code.
Finally, you need a good bug tracking system. You might try Bugzilla [bugzilla.org].
Good luck,
Sam
Methodologies and Tools (Score:2, Informative)
We use a couple of methodologies in my work place. Granted we are a Java shop so some of this stuff doesn't apply across the board, but the concepts work for almost any language/development platform.
First, XP Style testing. Test first, and test often. Write a test case for every class you make, test everything, unit test, regression test, integration test you name it just TEST it.
Second, simplify your development process. There really should not be the need for multiple people to be working in the same file/class/header etc... Assing pieces of the project to different developers and model it out, have them work in there rescpective pieces, if you MUST assign multiple people to the same header, thats okay, but make sure they work together closely to not step on each others toes. This is really a planning issue.
Third, I assume you are following a build process (nightly, weekly etc..), we use Ant [apache.org] to help with this. Granted it doesn't help for the problems of developers stepping on each other during the day. But it forces everybody to check in there code and make sure it works, everyday (we use nightly builds).
Okay with all of this stuff, we rarly EVER have problems. Our code is usually close to bullet proof (the constant testing), each developer really knows the portions of the code they worked on, and can quickly make fixes if needed (the simplification of the development process), and we are constantly aware of our timeline and progress (nightly / weekly builds).
Anyways, thats just how we do it ;)
-ryanRe:Better Solution: Use CVS or ClearCase Properly! (Score:5, Informative)
I'll second this: At many companies I have heard over and over that CVS sucks because of the conflicts. When I inquire further, I find out that people never or rarely use the cvs update command to synchronize. cvs update should be executed almost like a nervous habit, or at least a couple times a day. More if code is being checked in frequently, or if you are working on the same code as someone else. Use the watch commands in cvs to be notified when others edit your files.
It's a shame, really, because programmers seem to be afraid of cvs, preferring a more primitive tool such as rcs or pvcs. cvs lets programmers work in the style that is most natural in an open-source arrangement, and in my opinion can be a far more productive environment than a systems that locks the rest of the team out of critical files.
On CVS and Clearcase (Score:5, Informative)
Now on to CVS. CVS is most everything you want from revision control. It's biggest shortcomings are in branch management and the ease with which changes can be made incorrectly. Its ability to interface with well known and standard protocols like rsh, ssh and gzip (which is a format more than a protocol) make it painful to move to anything that's overly proprietary. Its use of your local diff is wonderful ("cvs diff -u" was a revelation for me).
Clearcase manages branches better and can handle non-realtime latency in updates (e.g. you can have two Clearcase repositories at different sites and you can connect them by mailing tapes around or by dialing up once a day). This can be invaluable when you're working in high-security environments, but is otherwise mostly a moot point.
Clearcase has improved in the last few years. They've added some local-checkout features where you don't have to work off of the virtual filesystem, and that helps.
Overall, I'd say CVS is the better system, but Clearcase will sometimes get jammed down your throat, and there are definitely worse fates than to have to get your project working under it.
Perforce and process recommendation (Score:3, Informative)
- Smaller. You only need one executable on your client. And one more for your server. No kernel patches, no drivers, no installation, just the binaries. Ubercool.
- Multi-platform. Perforce has binaries [perforce.com] for practically every platform in use out there. Find me another Version Control System with BeOS, QNX, AIX, SCO, MacOS9 and MacOSX support.
- Fast, fast fast. Because of the low communication overhead, it works extremely well across slow/high latency links.
- Ease of use. It's really easy to configure and setup.
- Great support. We've had to go to perforce support twice and both times they've been awesome, with quick responses and knowledgeable people.
- Price. The single server, two client setup is FREE! And per seat licensing + support is very very reasonable. I use the free one on my laptop to version files.
- Plug-ins. Perforce publishes their API, and they have perl, python, ruby and tcl utilities [perforce.com] galore.
Now my process recommendation. If you have paid for ClearCase then pay for some more education. ClearCase/Perforce can do lots of what you need automatically. You should practically NEVER have to repeat the same change on a file. You may want to look at some branching/merging techniques which can eliminate the need for colliding checkin's also. Rational has a bunch of whitepapers [rational.com] up on their site, as does Perforce [perforce.com].Most of all, I would advise you to educate yourself on the options/methods of version control. 12 isn't that big. Wait 'til you get to 1200.
managed branching and merging (Score:2, Informative)
My job has been to create a version control system that solves exactly this problem and automates others. Right now, it uses CVS, but is based off of a system that used PVCS and ClearCase.
Our revision system starts with release labels. Each directory gets tagged to a particular release label. Baselines, then are groups of release labels.
When a developer releases, their changes are branched off of the release label the directory started at. Then, the latest release is merged into their changes (if a later release from the one the branch is based off of occured). If everything still checks out, the branch is merged to the main development tree.
We have two scripts that make this process very simple: a commit and a release. The commit will take the current directory and create a users branch. This is for moving between trees - the developer now has a sandbox label they can update to. The release, which calls commit to make the user branch, handles both passes of merging.
I've noticed that CVS is not the greatest system for handling branches. It's ability to detect common ancestors is, well, flaky. I'm eagerly awaiting the release of Subversion, since my company really doesn't have the ooodles of $$ for ClearCase.
Also, note that when a developer releases, we also collect information which can link this release to a bug tracking slip. We also collate the release notes for releases. Ergo, you want to make a patch for your product, it's easy to collect all of the notes of changes and then create a single ChangeLog (which you usually want to edit a bit).
The system actually has been extended to handle multiple streams of development. This allows us to bugfix released versions of the product while making later changes, and smoothing the bugfix integration between releases. Adding this feature is both an enormously complex task and a huge timesaver for a large development team. You'd definitely push the bounds of what CVS can handle at this point.
Just some thoughts. This isn't a simple issue, but can have a solution that is simple to use.
-T
Re:SourceForge 3.0 Enterprise Edition (Score:3, Informative)
Minimum user license is 30 users, and that is roughly $30,000!!!
I run a small development firm and I wanted to use the enterprise edition. I'll pay a few thousand for something, but $30k for 30 people? I think not.
Perforce is the way to go. (Score:2, Informative)
Make developers mere on checkin (Score:3, Informative)
What I have found makes the most difference in reducing merge issues is (1), have developers merge on checkin (more on that in a moment), and (2) develop a branching strategy that reduces the need to merge.
On (1). When using ClearCase, have people set checkouts to be "unreserved". When they check out a file it won't lock it so everyone else can use it, and when they check something in ClearCase will force them to perform a merge if it has been updated since they checked it out. Make everyone learn how to use the henious merge tool. Also (and I'm not sure if this is fixed now or if I never figured out how to configure this right) I have had major issues with EOL markers in files and clearcase merge - if a developer edits a file that changes the EOL character(s), then ClearCase merge will claim the whole file differs.
For CVS, it pretty much works naturally the way a system should - the default is to check out something "unreserved" (or at least that's how I remember it!). You can either update a file while you're making changes to keep up to date an make merges smaller, or just wait until you're done - before you can check it in CVS will make you update the file and merge your changes.
In both systems, an approach of having the developer merge the file means the person who really knows what's going on can resolve any conflicts with the merge. Most merges are automatically handled and so often no work is required. I'm not sure how much this really addresses your issue, but it can't hurt to rely more on the source control system helping you manage merges.
For option (2), think carefully about how you want to structure releases. One approach I've used before is having dfferent "levels" of development - you have a production branch, fix branch, development 1, dev 2, etc. You start out code at some given level (say development 2) and as it is completed and tested it moves up through the ranks until it reaches the production level. That worked pretty well and meant a fixed number of branches.
In more recent projects I've been working with a monthly release structure - each months release requires a new branch. This was not my idea, believe me... it might seem nice in theory but in practice you might have three or more months of development at the same time. This leads to something of a merging cross-nightmare. Some sort of branching structure might well help solve your problem, if only in thinking up how you can more clearly seperate changes to a file or set of files.
The Free Software solution! (Score:2, Informative)
We all know that BitKeeper isn't free software, and I don't really know why it was chosen by Linus to manage Linux development since we have a much better solution in arch.
I think it is great. It follows the *nix spirit, spliting a problem in lots of little parts as addressing each part with one pre-existing tool or some home-made bash scripts. For the diff/patch lovers (like myself), it is great to find another uses for these tools.
I used to use CVS... but it doesn't scale so good. When Linus move to BitKeeper, I give it a try... but it was so disappointing. So I decided to give arch a try.... I am using it 'til now.
You can check it here [regexps.com].
Actually Use CVS (Score:2, Informative)
This is a dumb way to manage a project. It barely works for Linus. And he's a genius.
The right thing to do is to give your 12 developers access to the CVS repository. If they are geographically separated, use a VPN or ssh to connect to the repository. When they finish their work, they first update their local workspace, compile and test, and if it passes, commit their changes back to the repository. Other developers get the changes as soon as they are available.
Twelve developers is the leading edge of a small project. You can have a single team, and everyone can be aware of what everyone else is doing. The best process is the one with the minumum of overhead that suffices for the project at hand. Don't add process for the sake of process.
On the other hand, don't sacrifice the basics.
Version control is not 'Advanced', it's fundamental. You might as well say you're using advanced visual editors, such as vi. SCCS is from '75. RCS is from '82. CVS is from '86. This isn't new stuff.
rational (Score:2, Informative)
UML is good, but as many things it is often misused.
Rational software not one I would use, but it should work considering all developers are on the same page. Tools should be used to get understanding of system being built, not building system like lego toys, slapping a schedule on imlpementation and stilling back and relaxing.
Revisions to diagrams are often painful, because you often have to explain just basic things, and gut feelings to managers who do not care, and can't understand what you really mean. Often enegery is spent on changing diagrams around and talking to pseudo programmers, than implementing quality code.
Any methodology can be great, but only if management believs it in, keeps everyone on same page and makes process a development and understanding tool, rather than programmer control tool - which often happens.
I suggest using bitkepper or CVS in combination with automatic documentation generators(Doxygen?).
Re:Subversion! (Score:3, Informative)
When Subversion is ready, you might check it out. It keeps track of not specific versions for files, but revisions/patches for the entire tree. This way you can tell exactly the state of all the code at, say, revision 2735. No manual tagging needed. This would take up a lot of the work of your sequence numbers, without the severe administrative overhead. You could even try to assign a range of actual revisions for which a specific feature is targetted.
I'm already using Subversion for the early stages of one of my projects. It seems to be very stable currently, and of course I still make backups of the DB in case there still remain bugs that would corrupt it. I figure, I won't need to make any branches or merges in this project until well after the time that subversion can support it (due in April).
I've found in my experience... (Score:3, Informative)
But that doesn't scale... So you've got to modularize the project so that each team works against their own baseline. Any changes that you do to another teams part, must be given to them, for them to check out, and integrate themselves. The important thing is communication, it often happens that they are working on fixing the same problem, but in another and better way, and can give you back their fix instead.
So when you're having problems with your revision control system, what you are really experiencing most of the time, is problems with communication.
Ideally, all developers should be in the same building, they should work at (mostly) the same time (normal office hours), and they should all be friends, and keep a list of each others telephone-numbers, and eat lunch at the same time. Development should be split into projects (having 10-50 people on each project) that are mosty independent. All the people on the project should meet weekly, and discuss their project, important changes, etc... On each project there can be one or more teams, each consisting of four to eight people, who should have offices really close to each other. On each team, development should be split into sub-teams, consisting of 1-3 persons (depending on difficulty and experience), who should share an office, and thus communicate even further. And just as important: this should not be formalized (at least not too much). People should rotate around somewhat, not being stuck with the same people all the time, to get them to know other parts of the project, and other people to communicte with.
The important lesson I am trying to give is that the most beneficial communication, is often the informal. While having a tool help you with managing deltas is surely helpful, it can't solve every problem in the world. You need to work together, but you also need to modularize, and most importantly, you all need to be friends...