Mark Edel Answers Project Leadership Questions

Okay, here (as a refreshing break from all the political stuff) is what amounts to one of the most lucid primers on software development management I've ever seen, contained in NEdit leader Mark Edel's answers to your questions. If you ever expect to lead an Open Source project (or a closed source one, for that matter), you need to read this from start to finish, possibly even print out a copy and read it over and over. Great stuff!

Project coordination
(Score:5, Interesting)
by techmuse

First, thanks for producing Nedit. I've been a big fan of the program for years, and it's always one of the first things that I install. I'm wondering how you go about coordinating the development and testing of code on an open source project. Do you create well defined APIs for modules to interact with, and then ask people to work on modules that use those APIs? Do you put one person in charge of each segment of the project? How do you resolve disagreements over the features and functionality of the project? PS. What happened to the // or /* */ commenting macro in Nedit? I miss it!

Mark:

Everyone working on NEdit is a volunteer, and that makes the development process very different from what you would find in industry. I can't say that I even fully understand it. What is most peculiar and unnerving about it, is that you can't actively recruit someone for a given task. People appear out of nowhere and volunteer to work on things that you'd never dream of asking. Whereas, putting something on the wish list seems to kill off all enthusiasm for it.

Good APIs have indeed played an important role in coordinating and advancing NEdit's development. By far, the most successful of these is the one for syntax highlighting. Almost all of NEdit's language support is user contributed. Between the built-in patterns and the ones available on the web site, NEdit has highlighting patterns for more than sixty languages. There's probably something to be learned from the fact that a similar API for language sensitive indent has been so much less successful at attracting developers. Smart indent macros are harder to write, and require you to know the NEdit macro language, whereas highlight patterns only require understanding of regular expressions. I can only guess that there is a critical threshold where the quantity of work required exceeds the perceived benefit.

The majority of contributions to nedit are unsolicited. The contributor is relying on existing APIs, documentation, and program structure, and either blindly hoping that we'll integrate the change, or aiming for a target audience who will use it as an add-on. For larger projects, where a developer comes to me with an idea and a willingness to implement it, we send email back and forth. It usually takes many exchanges to define the scope and functionality of a change and decide how to fit it into the existing structure of NEdit. For significant projects, it makes a big difference if developer is willing to take some responsibility for ongoing maintenance of the new code.

What all this adds up to, is lots of rejected contributions, a few bad feelings, and even code forks. I'd like to say that we resolve disagreements via the developer discussion list, but in practice, I am acting like a dictator. The worst part about it is knowing that I am making mistakes, and slighting contributors because I'm too busy with other work to process their contributions in a timely manner.

I would prefer a model where this responsibility is shared among a larger group of people, either as a team of core developers, or even better, where we each take responsibility for smaller parts of the overall project. In areas where we've managed to divide up the project like this, we've so far done stunningly well.

(The C/C++ commenting macros are still there in the current version. They are language sensitive, so they show up only in appropriate language mode(s). If you still don't see them in C mode, try temporarily renaming your .nedit file, saving your preferences, and cutting and pasting them from the new .nedit file to your original.)

Using Motif/other 'unpopular' libs
(Score:5, Insightful)
by JanneM

A lot of people are very hesitant to install a whole set of libraries to run only one application -- almost no matter how good the app is -- when there are 'good enough' alternatives for the standard libs they already have.

Do you feel that NEdit has suffered from not using more popular libraries, and does it matter to you?

Mark:

Life with Lesstif and Open Motif is not as bad as some of the comments I saw in moderated-out questions would suggest. Users do not need to install these toolkits to use NEdit. The pre-built binaries come with Motif linked in statically. The "horrible bloat" that this static linking adds is about half a megabyte.

The real problem that Motif has caused for NEdit has been in its relationship with the various Linux distributions. Another question which didn't make it here, was titled "Why so Unstable?". I'm surprised there weren't more of these. Both Debian and Red Hat, refused to have anything to do with Motif, and would not accept statically linked binaries of NEdit. What they did do, however, was worse than just leaving it off. They packaged NEdit with early, unstable, versions of Lesstif, so Linux users who saw NEdit during this period, thought it was a pile of junk. An odd consequence is that NEdit has a larger user base in Europe than in the U.S., because Suse did distribute the binaries for a while.

Now that Lesstif is up to speed and Motif has been forced to open up, I'm hoping NEdit will finally appear in good form on the popular Linux distributions. Unfortunately, Red Hat is now so big, I haven't found anyone there who will pay attention to such a request. The one message I sent them got no reply, and I'm afraid of making a pest of myself.

Converting NEdit to a different toolkit will be a huge amount of work for very little gain. I am no fan of the Motif programming model, and I would not suggest it for new projects. However, NEdit has a huge investment in working around all of Motif's oddities, and I'm sure we'll face a similar battle with Gnome or Qt.

Users. Developers. And How to Find Them.
(Score:5, Interesting)
by omarius

I run a small (one developer: me!) project at sourceforge: a text-based web bulletin board called vaxbb.

Although sourceforge says it's been downloaded 63 times, I've received nary a comment or email of any sort. Granted, it needs a lot of work before vaxbb will be an install-n-go program, but I definitely think it fills a niche (I started writing it because I couldn't find a free bb that I liked the look of).

So, after all that intro, my question is: Does a project have to be super-slick before people will use or contribute to it? How does one find developers w/out a huge user base. . .or get a user base without having a fully-developed program?

Mark:

(I'll answer this question and the next one together)

How do you "advertise"?
(Score:5, Interesting)
by OlympicSponsor

Reading newsgroups and mailing lists, I occasionally come across people who have problems that could be solved with software I have written and made available. I have a hard time coming up with a way to point them towards my software without coming across as some kind of big-headed, know-it-all, "my software cleans your floor and does your taxes" loser. What to do?

Mark:

I'm probably the wrong person to ask these questions, because my answer is just going to depress you, and because I consider myself a complete failure in this area. I really believe that NEdit is the best editor out there, and that everyone should be using it; and yet, they aren't. I've hit people over the head with it, publically made a fool of myself over it, and utterly failed to sell a free product that can measurably make peoples lives better.

Probably I'm just delusional, and the ability to operate in terminal mode is actually the most important characteristic that Unix users want in a text editor.

Let me digress even further:

In the early Macintosh years, my wife and I ran a small software business. Our one product was a utility called SnapJot, of which we managed to sell something like 5,000 copies. The hard lesson we learned from the experience was that the game was all about marketing, and hardly at all about software. Our sales were ALWAYS directly proportional to our marketing budget.

Being engineers, we began with the faulty assumption that the quality of the product meant everything. We wrote the software, packaged it with a simple black and white manual, and submitted it to a bunch of magazines for review. The result was a one tiny entry in a new products listing, and a note from one reviewer claiming unspecified bugs. Sales, near zero.

Making no significant changes to the software, but switching to professionally printed color packaging, we got good reviews from all of the Mac magazines (5 mice from MacUser!). Sales, still near zero.

We began paying for listing in the MacWarehouse catalog. Sales!

More catalog adds, a few small magazine adds Sales, Sales, Sales!

A year or so after we closed up shop, I remember an interesting tidbit of software industry news. After selling on the order of a million copies of the heavily marketed SoftRAM '95, the Federal Trade Commission forced Syncronys Softcorp to admit that it was selling a product which literally did NOTHING.

Having learned this lesson the hard way, I don't feel nearly as put upon by the music/software/film/book publishing industries as the general Slashdot readership seems to. Ask yourself, are you seeking out MP3s of new unpublished artists, or desiring exactly those on whom the accursed music industry has lavished its marketing dollars?

Is the Free Software community different? Yes, particularly if you're the author of something which fits a well defined need. If you write a driver for a yet unsupported piece of hardware, you need only hint at it in an appropriate newsgroup, and users will pile up at your doorstep. If you write something less well defined, though, your voice can be easily drowned out by the ever increasing background noise level of the Internet.

There's definitely luck involved in whether something catches on or not, but having a polished package and pounding the digital pavement will certainly help. I find selling personally painful and degrading, and I'm terrible at it. Good salespeople require a special kind of thick skinned personality that enables them to face rejection, something I think very few of us nerdly types possess. NEdit has certainly suffered for it (the dumb name hasn't helped, either).

I have to ask...
(Score:5, Interesting)
by pb

What data structure do you use to store the text internally, and why? What trade-offs exist in your approach?

Mark:

NEdit stores text in a simple gapped-buffer structure, common to many text editors. The gapped-buffer is a block of memory with a moving hole in it. NEdit maintains the gap's size and continuously relocates it as part of the insertion / deletion process. This organization maps exceptionally well to how people actually use a text editor, and is very effective at minimizing the number and size of memory relocations that need to happen as users add and delete text. For comparison, another common approach to storage in a text editor is to break the text into linked blocks so that any given modification affects only a limited number of blocks. While this better limits worst-case memory movement; it also breaks up and scatters the stored data, destroying its locality; so the virtual memory system can't swap it as efficiently.

While I feel good about the data structure itself, the bigger question is whether the data should be maintained in virtual memory at all. The advantages are simplicity, good performance, and robustness in the face of file system problems. Because NEdit only touches a file on open and save and does not depend on its contents being stable, your work is much less likely to be trashed by mishaps like overwriting, system crashes, loss of network connections, etc.. The disadvantage is that if you have a really big file you want to edit, you may have to increase your swap space, or maybe give up altogether and use a different editor. In practice, text files are absolutely miniscule on the scale of todays systems, and rarely does anyone approach that limit, but I certainly wish the limit wasn't there.

Common sense of FS/OSS development
(Score:5, Interesting)
by dilger

Here's two questions:

1.From which (if any) of the commonly accepted principles of FS/OSS development does your project deviate?

Mark:

The advice, "Release early, release often", contradicted everything I had been taught about software engineering. For most of nedit's life, it has had carefully planned alpha and beta tests, with as much as a year between new versions. When I first read "The Cathedral and the Bazaar", though, something clicked. I had only just started to realize some of the odd nature of what we were doing. I'm still in awe of the power of distributed debugging. The reliability we've achieved with NEdit is beyond anything I could have hoped for had the testing been limited to a small group of people. NEdit simply doesn't crash. I also realized that some of the things I was doing were actually making it harder for people to contribute, and I've been slowly trying to adopt more of the other non traditional practices of the FS/OSS movement.

Since most of NEdit's development happened before the Cathedral and the Bazaar paper, and before the whole movement gained momentum, I haven't yet had enough experience testing those principles to contradict them. We've only recently reached the goal of having the development sources continuously accessible.

One noticeable difference between NEdit and other FS/OSS projects is that the code is neat and thoroughly documented. As I mention in a later question, enforcing this has not been fun, and has probably been counterproductive, as well. It's just something I've found hard to give up on, particularly where I'm going to be the one with the responsibility for maintaining a given section of code.

2.How much time do you spend writing code vs doing more administrative/moderation tasks?

Mark:

I hardly get to spend any time actually coding. Given the limited time I have to spend on NEdit, at all, my first priority is integrating what other people are doing. I'm thankful that other people have stepped in to help with some of the management. I'm really only in charge of the code, now. I'm also thankful to the user community as a whole for participating in the ongoing discussion and answering eachothers questions. I used to get completely bogged down in email, to the point where I couldn't do anything productive at all.

Software Enginnering for OSS?
(Score:5, Interesting)
by Midnight Warrior

Now that you've successfully created a wonderful, stable editor, how many of the following software engineering techniques were used on the project, or would have been helpful?

1.Clear, concise statement of work (since NEdit was developed with U.S. Govt funding)

2.A Requirements Management process (measurable details about end-user goals)

3.Requirements Traceability Matrix (did all the planned stuff get implimented?)

4.Design Documents (GUIs plus interactions with black-box libraries or external packages)

5.Work Breakdown/Size Estimates (what are the pieces of the design that need attacked, what skills are required for each, who is attacking each piece)

6.Project Tracking (some automated way of allowing outsiders to see how it's going)

Colleges are teaching this. Companies who want government contracts are starting to do this kind of stuff. Would these software engineering techniques help OpenSource biggies stay on track and on time even though there is no financial interest to do so? The upcoming [cough] Linux kernel [cough] comes to mind.

Mark:

I'm glad I've never had my productivity forcibly "enhanced" with any of those processes (our software was for research). You skipped the best one, though, SASD (Structured Analysis, Structured Design). First you spend a year creating reams of useless diagrams and unreadable documentation. Then at the "end" of the project (which is actually the beginning) you try to write the code to fit the model you've generated. Unfortunately, without the benefit of bottom-up experimentation you now discover that the libraries you are using don't work the way you expected, and a whole different architecture would have been better. To get a working program, you then write the software conventionally and fudge up all of the silly diagrams to match.

I do believe strongly in documentation, though. Software is communication as much as it is algorithms. Even if you're writing device drivers or embedded systems, most of what you're doing interfacing with other people's work. On large projects, with lots of people or expensive resources, project tracking is what keeps half the company from sitting idle, while one overscheduled programmer contemplates suicide. Of course, I see no reason to force it on individual programmers. It's something managers should do, themselves.

License
(Score:5, Interesting)
by angst_ridden_hipster

First, I want to say that I've been using Nedit for probably seven years, and I've always thought it was the best thing out there. Someone mentioned that it eased the transition from PC/Mac to Unix, and that was certainly true for me. So thanks for your great work!

When I first used Nedit, I was working for a government-related agency, which had restrictions on what kinds of software could be run. Because Mark Edel hosted the source on a xxx machine, I was able to weasel around some of those restrictions. This, of course, was not before GNU but was well before the Open Source movement gained its groundswell of popularity. When the movement did gain popular momentum, it was looked upon with distrust and suspicion within the agency where I worked. There were offhand references to Communism and anarchy whenever people heard my enthusiasm. Likewise, when I wanted to give out source to some utilities I wrote, I was summarily shot down and chastised -- "this was paid for by the Government!" I was told (which was exactly my point; they just had a different conclusion as to what that meant).

So... how did you convince [organization] (or the [government agency]) to allow source distribution in the first place? Did you run into any difficulties when you decided to change the license? Did you have to get permission within the organization? If so, what did you need to do to get approval?

Mark:

Note how I have edited your question; and compare NEdit's documentation, web site, and copyright notices between those earlier releases and today. You might detect that ALL mention of the institution which supported a good part of the original development of NEdit has been removed. This was part of a peculiar (to say the least) compromise that we finally reached to appease a handful of government bureaucrats and lawyers, fanatically paranoid about liability.

Needless to say, it is difficult for me to discuss this subject without violating at least the spirit of this idiotic agreement. I hope someday I can tell the story, since parts of it are really unbelievable.

That we could get away with publicly releasing those earlier versions was not such an extraordinary feat. The institution in question is merely funded by the government, but run by a consortium of universities. Still, we were on shaky ground doing it. Even though sharing software is integral to scientific collaboration, their battle to release software freely still rages. Having been part of it, it's hard for me to think about it objectively. The processes that produced the various versions of the original license were bizarre. When it came to getting the GPL, I mostly just outlasted the opposition. I had little to lose, since I didn't want to put further work into nedit under the old license. Even so, it took a year of being shunned, misdirected, lied to, and even vaguely threatened, to make it happen.

Design and testing
(Score:5, Interesting)
by Grab

One of my pet peeves is lack of design, documentation, commenting and testing. I'd appreciate your views on how much you believe these are necessary. Do you design before you code, or do you just dive in? Do you like to provide loads of support documentation for users and co-workers, or are you of the "the-code-is-my-documentation" school?

More importantly, how do you enforce any such standard on your project, given that you've got other people submitting you code patches which may be technically perfect but visually obfuscated? And has this caused any friction, since managing coders is a "herding cats" scenario, and criticising someone's coding style is often taken very personally?

Mark:

I feel strongly about those issues, too. As I said earlier, I think software is as much about communication as it is about algorithms. The essence of programming, to me, is breaking down a problem into components, until each component is easily understood on its own. I know I have succeeded when I can clearly describe, in words, what each component does, and how it does it.

The coding standards in NEdit are much higher than what I see in other Open Source projects. That causes friction between myself and many contributors. It may well be that what I am doing is wrong. As I mentioned in a previous question, there is an almost mystical power in having a user base willing to test and assist with debugging. By exercising my gatekeeping too early, I may be stifling experimentation and making unnecessary work for myself and for the contributors.

The question of design is a bit more complicated. NEdit began as a working, throwaway, prototype for some data analysis software, and evolved slowly and continuously into its current state. I wouldn't recommend that as a design methodology, but it is a common course for Free Software projects to take. Through all the modifications, it is still very manageable code; nowhere near the hairy edge of maintainability around which much "better-designed" software seems to hover. Incremental projects like these openly defy traditional software engineering wisdom. Why do they work? Small size may have something to do with it, and the magical effect of "the Bazaar". From my own experience, I think the difference might just be commitment. Simply taking care to maintain the modularity and comprehensibility of the code may be all it takes to stop it from plunging over the waterfall.

-- Mark Edel

Re:Emacs (Obligatory retort...)

[lonenut]

s/Emacs/vi/g

But stay away from MSPress Network Essentials

[LoonXTall]

We're being forced to use Network Essentials in class. The problem is, it's inconsistent---after describing the job of each layer of the OSI model, they then say that TCP goes _only_ with the transport layer. Hello? TCP's jobs are ensuring reliable delivery (transport layer) AND establishing a session (session layer)! What do they think SYN/ACK/FIN are for? It's also Pro-MS: they only mention tools in the interoperability section that allow NT to work with Netware---NOTHING of the reverse. In the Survey of NOSes chapter, WinNT and Novell each get 3 pages of decription, Apple gets 1 page, Banyan Vines gets almost a page, and *nix, Win9x, and OS/2 each get a paragraph. Under creating accounts, NT gets 4 pages with illustrations, and *nix just gets 2 paragraphs (which never mention (user|group)(add|mod|del).

</rant>

More objectively, it's just old. Who needs to know everything (terminator resistance) about Thinnet and Thicknet? ArcNet? This from the third edition (2000)... shouldn't the bugs be out by now?

-- LoonXTall

NEdit is the One True GUI Text Editor

[Mike Greaves]

Even given the undeniable potency of vi/emacs/etc. (keystroke driven) text editors, NEdit is still the best text editor out there from many standpoints.

At where I work, in a small, Unix-based, CAD/CAM operation (HP/UX and Linux) it is tremendously empowering. It is approachable for non-Unix people, but nifty and potent enough for old hacks as well. It makes it possible to standardize on a single editor for all normal text editing operations, yet keep everyone reasonably happy.

And any damned fool knows that Motif is *still* the best available, standard GUI toolkit on any platform. Not to start a flamewar, but Qt is just passable, Tk is too slow for many purposes, and GTK+ is a throwback - substantially more rudimentary than Motif.

Hats off to Mark. We appreciate your work!

MISRA??? Where?

[volpe]

Does MISRA have a website where these defensive programming guidelines can be found? I went to www.misra.org, but it was definitely NOT, I repeat, **NOT** what I was looking for.

Free Book Project About Writing Free Books

[Bryce]

The Freebooks Project [myip.org] was started a few weeks ago from an article [slashdot.org] here on Slashdot about Free/Open Source Physics (or other scholastic) Textbooks. The author of that article, Ben Crowell, proposed and helped start this project.

We decided that since our interests had more to do with the idea of writing free books than writing physics books, that we'd write a free book *on* writing free books.

Even though we've only had a few weeks, we've got many of the technical hurdles tackled and are getting down to collecting thoughts and ideas, brainstorming, and filling in the website. And if you're interested in lending your support to this cool (and critically needed) service to the open source and free software communities, check out [myip.org] the site, and join our mailing list [reusability.org].

Re:Damn he's sharp

[joss]

College is a good place to learn because you spend 4 years getting laid and drunk and puking up in flowerbeds, on the basis of which you will be better paid for the rest of your life. Sometimes you have to go and listen to pointless drival and you have less money than you would in a job, but it's more fun anyway.

Be wary of "formal development" - don't think there's anything magical about it, some people just prefer following mindless procedures to doing anything useful. These people manage to give off the impression that they're doing something important and "professional" but rarely produce anything. Developing large projects *is* difficult, it requires experience, intelligence and most of all an insight into the problem that takes a while to obtain, but doesn't come from any "methodology". There's no subsitute for understanding.

Become a good programmer first. Write your own lisp interpreter in assembly before you learn UML. Invent your own design patterns before you read about other people's.

Re:SASD

[joss]

> If the master wants to code in 'vi', who's to
> stop him? If the punk intern wants to do all his > work in Rational Rose, he's wasting time, and
> it's the project manager's responsibility to
> assign him to something small enough that
> he can learn good practice through practice.

That is pure poetry. It's also an accurate example, the master often does want to code in vi while the punk intern pisses about with Rational Rose, generating 1000 pissy ill thought out classes.

project management in a volunteer environment

[filldup]

Reading Mark's comments, I can't help but see similarities between his experience with volunteer development and my experience organizing a volunteer-based social-justice group. We have had almost the same problems, and have unfortunately discovered that a somewhat dictatorial system is actually far more workable and rewarding than a consensus or democratic process. Those running similar groups -- whose members often act like OpSrc programmers -- might be well advised to look into the comments of people like Mark. I know I will.

Re:Damn he's sharp

[Coz]

Methodology, process, toolset... they're all tied together. Where to start, where to start....

In the Days of Yore when I was an undergrad, they worried about teaching us Structured Design, or maybe Functional Decomposition. Painful stuff, but still useful for developing algorithms.

The current technology supports Object-Oriented software development, although much of what's called O-O these days is C compiled with a C++ compiler, where the structs are named "class." The Unified Modeling Language seems to have finally put an end to the Notation Wars of the 80s and 90s, and allows you to document your design in sundry ways. The Three Amigos at Rational (Booch, Rumbaugh, Jacobsen, in the order Rat. bought 'em) wrote it, and the OMG has accepted it.

Now - how do you use it? Rational has their Unified Process, which is a big-momma full-up kill-rain-forests-for-documentation Formal Process, and then there are the other folks like Kent Beck with eXtreme Programming and Peter Coad with Feature-Driven Development and Alistair Cockburn with his Crystal processes....

So - advice for the young - learn UML, learn Java, go to a bookstore and pick a basic OO book and read it, then pick another and read it, then do the stuff that's common between them. Out in the Real World, they'll want to retrain you anyway :-).

Re:Damn he's sharp

[kfg]

A friend of mine once made the observation that the only reason for a blueprint, ever, is to be able to PROVE it was the engineers fault, and not that of the person assigned to implement the design.

He was absolutely correct, and if you can't create something for a rough working drawing on the back of your Denny's placemat, ( which is where you had your 'meeting'), there is something seriously wrong with your design in the first place.

Re:Damn he's sharp

[g_mcbay]

Read many books that are related to software engineering in general...Books like Code Complete, Design Patterns, Xtreme Programming, etc. Of course, if you don't have a great library available, money becomes a factor, but I suggest reading quite a few of these books and trying out their advice.

Be aware though, a lot of these books contain some amount of bullshit (at least in my experience) that is not applicable to most projects...For example, I'm not really recommending you use Xtreme Programming practices, but in my experience some aspects of it are useful.But you at least get different ideas from reading them and hopefully you can take what works for you and leave the rest behind.

Also, its often good to disect what you think doesn't work, so you can potentially argue against those methologies at future jobs if needed (assuming you are arguing for good reason, not because you are too lazy to adhere to what would be a good process..of course).

Whatever you do, don't fall into the trap of believing in the 'flavor of the month' methodologies! As I mentioned in another post to this thread, I've worked for companies where the higherups cared more about developers strictly following certain processes than they did about getting good, working code. And this company was an Internet startup working on their first real project with fairly inexperienced technology leads. They were not basing their methodology choices on experience, but rather the hyped up snake-oil some of these methodologies offer.

Re:Damn he's sharp

[StormyMonday]

In fact, in a decade of experience in the IT field, I've yet to have a company request, much less mandate the use of any academic structured design methodology

Interesting. Lately, it seems like every interview I go on, some manager rags on my lack of experience with "UML Design Methodology". This despite the fact that every real UML document that I've seen emphasises that it's not a design methadology. (It is, in fact, the current incarnation of flowcharting. "If you don't understand it, draw a picture. You still won't understand it, but it looks nifty on a slide.")

Just one more example of the Buzzword of the Week, I guess. Do PHBs have little desk calendars with these buzzwords on them, like geeks have Dilbert cartoons?


--

Congratulations

[Mindwarp]

A very well written article.

I can wholeheartedly agree with Mark's comments about the need for good coding discipline, communication and documentation within a project. In my experience it doesn't matter how brilliant your developers may be; if there is a lack of discipline when it comes to design and documentation then it is extremely hard for that project to succeed. As an example, my current project (a global system for a large investment bank) started out as a group of smart but inexperienced college graduates. Every one of the initial project releases was plagued by 'show stopper' bugs which used to take weeks to track down due to the disparate coding/design styles and lack of documentation. Over the past two years the project has, due to the efforts of the senior developers to instill the required discipline in the team, matured to the point where releases are now extremely close to 100% reliable. The bottom line is that we now have happy users, happy developers, and happy pay-checks come bonus time!

For further reading in a similar vein, I can recommend the "Software Engineering Classics" boxed set (yes, I know it's Microsoft Press. Please don't hold that against them, as these books contain much useful information and insight into Project/Team leadership within Software Engineering.)

The individual books are:

1. Debugging the Development Process Steve Maguire

2. Dynamics of Software Development Jim McCarthy

3. Software Project Survival Guide Steve McConnell

The ISBN number for the box set is 0-7356-0597-1.



--

Re:Coding standards

[CorporateProgrammerD]

OK, this might be redundant, I have /. set to "highest scores first" and haven't read down to the bottom yet, but here goes...

Here is something from my company's coding standard: When maintaining code, the style of the original code should be retained, even if the style conflicts with the style guidelines in this document. I REALLY agree with this. After years of maintenance programming, running across code that had been worked on by 10+ people and looks like it, I do this instinctively, and did before I even started this job. But it's nice to see it in writing, even if it is in a document that most of the programmers here never see. It's just a way to make life easier for the next person.

I just wish the previous coders had been as thoughtful!

Quite Helpful

[CyberKnet]

As a hobbyist programmer I often find myself gaping at the coding styles of other programmers. I started my programming life as a visual basic programmer (cut the flames), but my instructor (well, person who I went to when BIG problems arose) was a C programmer. As such, he refused to help me unless my coding was explicitly set out in a specific way. Everything had to be named right, in the right files, called the right way. And THEN he would look at my work, and say "Should of done this in C CyberKnet... this is quite nice". I never inherrited the sloppy coding that is common to visual basic programmers. Just thought I'd air a public thanks to good C programmers everywhere, and great project administrators. When they do their job well, it makes things SO much easier to maintain.

---

managing oss stuff

[jdwilso2]

yeah, thats great and all... actually, i really appreciate the information, but the problem i've got is getting started... there was one question asked about the same thing, but it was lumped in with advertisement, which (in my mind) is a little different than recruiting developers for a project that has yet to gain any popularity at all...

and on top of that, i'm writing a game, and with games its a whole other story... to my knowledge, there aren't any bleeding edge games that are opensource. i mean, sure, you've got wolfenstein and quake even, but they're all opened up after their money making days are over.

and above and beyond that, it takes a whole lot to develop a good game. the fact that no one wants to put all that work into something that will not only not make money, but will not contribute anything productive to the oss community or the net as a whole kinda takes the bottom out of the whole idea...

and then there is the fact the everyone who has the motivation and the skill to write a bleeding edge game wants to write there own because they can't agree with anyone about how the story should go, or how to do the interface, or how the gameplay should come out, or whether to use an engine or write one from the ground up, or ... well ... you get the idea...

So I just thought that I'd mention how these ideas are all great, but they don't always apply in my experience, because you can't always get to a point where anything becomes useful... which is why there are no great open source games.

JDW

No TODO list?

[Coppit]

Great article. For once, someone with half a brain actually respects Slashdot enough to answer thoughtfully... But this:

People appear out of nowhere and volunteer to work on things that you'd never dream of asking. Whereas, putting something on the wish list seems to kill off all enthusiasm for it.

Wow. This was a shocker to me. I have a few projects of my own, and I've suspected that people might have a "well, he'll implement it, so why bother?" attitude. This seems to confirm it. I wonder if development would go better if the TODO list was renamed a WISHLIST with a note saying that there is no one working on those aspects, even though they would be worthwhile.
------------------------------------- ------------------

SASD

[jabber01]

I work for a company that mandates UML and the RUP. Care to guess what the first thing to go out the window is? RUP. The last work-product to be generated? The UML design docs. It seems that these are mandated just so managers can puff out their chests at meetings, and brag about the modern 'tools' their people use - they're just pissing into the wind, since at milestone #1, they demselves order corners to be cut.

Don't get me wrong though, there is definitelly a place for SASD in the industry. I think that Mark's view is on the extreme left side of the issue. It doesn't work for him - fine. It's not likely to work in OSS at all, since OSS is voluntary and volunteers don't appreciate being ordered around. They simply walk away when told to do something they will not find enjoyable, or in a way they do not know.

The real problem with SASD comes in when management tries to use it as a 'silver bullet'. Quoth Fred Brooks: "There are no silver bullets." A structured, tracable, repeatable process will not compensate for the lack of good people. Good coders can bring order out of chaos, but SASD without inspiration, experience and discipline is like herding cats.

The real benefits of SASD are only realizable when you have a group of proficient developers working together. UML is a very good way to communicate ideas. CRC cards are a great way to flesh out object design. Formal requirement reviews are needed before coding can reliably progress, lest major needs are overlooked or flaws are 'designed in'.

You can give a 'green' carpenter's apprentice a powerful chain-saw, but he's likely to make only bloody sawdust with it. In the hands of a master carpenter, a powerful chainsaw makes a hard job much easier.

So the sophistication of the tools used should reflect the sophistication of the developer. If the master wants to code in 'vi', who's to stop him? If the punk intern wants to do all his work in Rational Rose, he's wasting time, and it's the project manager's responsibility to assign him to something small enough that he can learn good practice through practice.

Once good development practices become intuitive, then the tools that promise to do that thinking for you may be used. But not learning the basics manually, and replacing the brain with a third-party product/process only results in an army of VB-coding monkeys who become completely useless when confronted with a new version of their IDE.

The REAL jabber has the /. user id: 13196

Re:MISRA??? Where?

[krlynch]

MISRA site [misra.org.uk], I think. It doesn't appear that it contains any free information, just ISBN info so that you can buy the specs and recommendations.

Re:Coding standards

[Mark Edel]

Coding style shouldn't break C++ highlighting. Are you possibly using NEdit with an old Lesstif version that might have display problems? If not, report it and we'll have a look. That's definitely not a common problem.

Book recommendations

[dsplat]

Read The Unix Philosophy and The Practice of Programming. The Unix view of how the universe should be isn't the only one. But those two books do a good job of explaining its strengths and how to leverage them, and why.

Read authors who disagree with each other. Decide whether the disagreements are dogma or practical. It is easy to get insulated from the real world by a single worldview. That worldview will serve you well until you meet an exception to it. Reading dissenting opinions is a good way to forearm yourself against that eventuality.

Thanks....

[blogan]

Considering Mark didn't want that information released, I'm glad someone decided to post it for us.

Everyone knows that they could've looked at the original post to find the information if they so desired. It's just impolite to point it out in discussion.

Re:Coding standards

[Speare]

Coding religion flame wars will probably be started by this, but... :)

I like my open curly braces UNDER the IF, thank you very much. It seems like all the rage in Java and Perl circles to put it at the right end of the screen, so you have to scan left and right AND down to see matches.

If I edit someone else's code, though, I put the curlies where THE AUTHOR would want them. I don't pretty-print/indent, and I don't touch more lines than I need to. Ugly or not.

Never crashes?

[faster]

I love NEdit, and use it almost every day.

So I was a little surprised to read above that it never crashes. Try this:

create a keystroke macro, a relatively complex one with selection and copy and paste. Make the last keystroke a down arrow so you can just hit ^k repeatedly to activate the macro on successive lines.

Then whack ^k 10 or 15 times.

When I'm running NEdit on a remote host, this will core NEdit every time. The slower the connection, the faster it'll die.

On the plus side, that's the only way I've ever been able to kill it.

Disclaimer: I break software for a living.

Re:No TODO list?

[flossie]

I wonder if part of the problem could be that anything on a TODO list is perceived as being fairly rigidly sketched out in the lead developer's mind, thus reducing the opportunities for contributors to influence the direction of the project as much as they would wish.
There is also the possibility that anything remaining on a TODO list cannot be deemed to be too important, or worth the effort to implement, by everyone else who hasn't volunteered to do it.
Maybe wishlist would be a better description.
Btw, excellent replies, Mark!

-- flossie
http [2130706433] telnet [017700000001]

Great job, Mark (and everyone else)

[Anonymous Coward]

This is a terrific set of questions and answers, and I'm glad to see NEdit (my personal favorite text editor) get the attention.

Blessed are the project managers. . . I hope

[kfg]

I've been writing free software since the mid 70's, but always in the small, code it myself in a week or two of puttering mode.

The only fairly major project I've ever tackled was a propriatary tool for my own business, and again, handled entirely myself using the same work method as my smaller projects, just extended over a couple of years rather than weeks.

Now that my business is over I may yet take the time to polish it up and release open source.

I am now, after 25 years of coding, ( my first project was writing a blackjack game in APL on an IBM 360 system with plays typed out by the Selectric terminal), I'm looking at undertaking my first truely massive project, and my first truely open development one.

This was a wonderful look at the ideas and experience of the maintainer of a long term major project.

Unfortunately for me it meerly serves to confirm what I have learned myself from experience, observation of others projects, and just plain deduction from first principles.

The first issue that I consider key is that most planning goes up in smoke pretty damn quickly, the project takes on and defines its own course as it goes along and any requirement to show your planning to an overseer meerly results in those documents being fudged after the fact to match what it is you actually ended up doing.

I've seen this happen over and over and over again, not just in software projects but in major hardware engineering projects ranging from new innovative scientific measuring devices to the design of automobiles.

This has always seemed only logical to me. You are filling in blank areas on the map. The very REASON you are doing the project is to figure out how to do it. How on earth can you print a 'roadmap' of terra incognita?

This isn't to say you don't need planning, but you need to do it on the proverbial coctail napkin, and keep a good supply of them on hand because you'll be drawing new maps on a regular basis, maybe multiple times a day.

When going into uncharted territory travel light, pack a big lunch, and don't plan to keep to any itinerary. This is exploration, not a package tour.

The second key point to me is the one I've been dreading but fear is ineveitable. To manage a major project you have to accept up front that your days as a coder are numbered. You will eventually become, and probably sooner rather than later, a dictator/administrator. The cats need to be herded and there's no more useless cat herder than another cat.

Ok, I've managed before, and I know I'm good at. That dosn't mean I like it. I'm a geek. I want to dick with the nuts and bolts of a problem and solve it. I want to know *I* did it, and it was good.

It is interesting to see the administator of a major and highly regarded piece of software talking mainly about how he feels he hasn't done things very well.

Perhaps in the future if I gird my loins and actually take on this project his words will offer me the solace of knowing that at least I'm not alone.

You wrote SnapJot!?

[Anonymous Coward]

You rock!

(Sorry, been meaning to tell the author that for a while now. Didn't think I'd ever get a chance.)

ack! only 6 comments!

[fjordboy]

I noticed that there were only six comments..and only 2 above -1....I wonder...maybe this is because he just ANSWERED ALL OF OUR QUESTIONS! I don't expect much of a discussion now...we have the answers, and the questions, and I am satisfied. Everyone can go home now. Have a cup of tea or something...

btw...thanks for answering all the questions Mark....Nedit rules.

Damn he's sharp

[Greyfox]

His comments on design and testing in particular were dead on. Just recently in an interview I had a company raise an eyebrow at my relative lack of experience with structured design methodologies. In fact, in a decade of experience in the IT field, I've yet to have a company request, much less mandate the use of any academic structured design methodology. I suspect that this is due to the fact that, as Mark suggests, you end up generating reams of documents that you don't adhere to when you finally hack the program up in the end.

At the same time you do need a certain level of design before you start coding, or you'll end up rewriting everything over and over again. Most programmers I know do some ad-hoc design work before they start coding. You really need to understand the problem before you can break it down into managable pieces, and most programemrs I know seem to use the "design" step to do that. Enforcing at least some level of design is a hard task for any project, commercial or Open, but it is a necessary one if the project is to succeed.

Reply to Mark

[Grab]

Cheers for the reply Mark, and good luck with the project. I was curious about the reply on design documentation:-

"Of course, I see no reason to force it on individual programmers. It's something managers should do, themselves."

Yes, the interface must be well-designed, and this can be done at the management stage. But having a readme (or something) to explain obscure elements of the design isn't too surprising an example.

Consider a buffering I/O system. This may have to split its operation over several concurrent tasks, and to define this merely by the overall requirement of "it must do this", and the I/O interface seems to be too high a level. At the very least have a readme or extensive commenting to say what bits are going in what function, and how the functions interact.

Incidentally, for C coders out there, there's an organisation called MISRA which produces standards for safety-related code (specifically for automotive), so they know a bit about getting rid of errors. One of their documents gives a list of rules to follow for defensive programming, which tries to stop silly errors like typos from getting in the way of a working design (as an example, anyone else howled when they spotted an "=" instead of a "=="? well use "if(CONST==variable)" instead of "if(variable==CONST)" and the compiler will always pick up the error). Granted, most of the guidelines are more applicable to embedded development, but I think there's still lessons to learn from that for general coding.

The difference is that crashes of a computer program cause more irritation and dents in pride than anything else, whereas failure of an aircraft engine controller is life-threatening and can't be tolerated. I'd be interested to know how well the safety-related embedded programming and computer science/desktop fields are talking together, whether they're reinventing the wheel, how much overlap there is between methods, and which bits are more relevant to one than the other. I do know that softies and sparkies don't talk as much as they should do!

Grab.

Coding standards

[Coz]

Amen, Brother Mark!

I hate reading half a dozen files and having to figure out half a dozen different sets of personal coding quirks. I've gotten so I can almost identify contributors to some projects by their coding style - which is not, to me, a desirable characteristic.

When I'm leading a development team (on crass commercial software development) I try to get some standards for comments, indenting, and naming conventions in place early on, and point out discrepancies during code reviews. Yeah, there's more typing when you have to name a class for the piece of the system it's in - but it's a heck of a lot better than having the C++ preprocessor "pick" a class with a colliding name (or even worse, having some static data element collide).

Folks may complain about the coding standards you hold them to, but it's obvious from the quality of the product that it's working. Bravo!

Ditto on Microsoft Press books

[jonabbey]

I have to agree about the recommendation for the Microsoft Press books. I have six Microsoft Press software development books on my bookshelf at work, and each of them individually is among the best I have on the actual practice of software development.

Re:Damn he's sharp

[Anonymous Coward]

what's a good way to learn design methodologies? I don't mean the structured type like Mark mentioned in the answers, but real, useful methods for breaking a task into solvable steps?

I'm 17 and will graduate in the spring; I'm interested in computers and programming but I don't have any real experience in formal development outside of my own experimenting. Is college really the best place to learn this sort of thing, or am I better off learning on the job?

(note: your answer will not necessarily determine my fate, but it might help.)

It's Fermilab

[PurinaCatChow]

Just for the record, the organization that supported the development of NEdit is Fermilab [fnal.gov].

I just did an "apt-get install nedit"--it looks like a great editor. Thanks!

Re:SASD

[Mark Edel]

I remember several of the developers of one of our SASD projects expressing almost the opposite opinion; that SASD might be more helpful on a project with LESS experienced developers, where the pacing and structuring of the design process would lead them in a clearer direction than they might otherwise choose for themselves.

Incremental projects

[bolthole]

Yeup, know what you mean about incremental projects.

I have been maintaining a program I first wrote over ten years ago. I DID actually start with some loose design work (I think :-) But every so often, I added a feature, many of which were user requests.

The nice thing was, my original code layout was clean enough so that I could add features without actually having to look at the whole code. Unfortunately, after a few years of Feaping Creaturism, that tended to have a bad result when you look at the big picture :-)

So, I took the time to restructure it. Not so much rewriting code, as grouping the appropriate code in the appropriate files. Making the header files match up (widget.h should ONLY have stuff needed to interface with widget.c)

And once done, it was even better than before. No visible user change, but it made continued work on it SO much easier.

Plus, it helped me track down one or two things on the buglist that I was having difficulty figuring out before :-)

Re:Reply to Mark

[flossie]

The key thing about safety critical software is that it is horrendously expensive to produce. The amount of testing that on-flight software has to undergo is far in excess of that which would be required for a desktop application. The kind of code being executed is also very different. The pilot of an aircraft is not going to ask the processor to instantiate a new window with an odd mix of properties half way through a flight! The developers of safety-critical software also have knowledge of/control over all of the software running on the hardware - no dodgy downloads from Warez'R'Us on an airliner.
Additionally, I don't think that many applications programmers would be over-joyed by the prospect of coding in Spark Ada, and an engine controller certainly won't be developed in Java anytime soon.

-- flossie
http [2130706433] telnet [017700000001]

Re:Damn he's sharp

[g_mcbay]

I've had the misfortune of working at a company where they had a big giant wad of 'methodologies' and 'processes' that would make our development 'easier'. It was a total nightmare. Months wasted in design meetings arguing over abstract ideas and stupid dogma (You must use the facade pattern when implementing this!! says one engineer. No, you must use the factory pattern!!! says another, ad infinitium).

In the end we wound up with an extremely well written design document that had no basis in reality. Coding the project was a total failure because, as Mark suggested, the 'design' rarely matches the APIs you depend on when the design is done completely on-paper with no test code written during the design process.

In any case, this is one of the best interview responses I've read on Slashdot (a lot better than the evasive marketing drivel from Indrema, for example), and I'll check NEdit out just because of that.

OT: your grammar

[Anonymous Coward]

Your use of the word "methodology" is incorrect.

Methodology = the study of methods

What you meant was "structured design methods". Please, help put an end to the common misuse of this word. It is annoying to see pointy-haired boss types using big words to try and sound intelligent when they don't even know what they mean. It leads to them becoming buzzwords which other people such as yourself pick up. This new buzzword-riddled workplace lingo we now have is nothing more than a vocabulary supplement invented by people who are incapable of speaking eloquently without it.

Re:Reply to Mark

[Mark Edel]

That sentance, "Of course, I see no reason to force it on individual programmers. It's something managers should do, themselves.", was about project tracking, not about design documentation. Obviously the people doing the design should be the ones documenting it.