C# Memory Leak Torpedoed Princeton's DARPA Chances

nil0lab writes "In a case of 20/20 hindsight, Princeton DARPA Grand Challenge team member Bryan Cattle reflects on how their code failed to forget obstacles it had passed. It was written in Microsoft's C#, which isn't supposed to let you have memory leaks. 'We kept noticing that the computer would begin to bog down after extended periods of driving. This problem was pernicious because it only showed up after 40 minutes to an hour of driving around and collecting obstacles. The computer performance would just gradually slow down until the car just simply stopped responding, usually with the gas pedal down, and would just drive off into the bush until we pulled the plug. We looked through the code on paper, literally line by line, and just couldn't for the life of us imagine what the problem was.'"

I'll show you mine if you..

I'll show you my perpetual motion machines if you show me your perfect autonomous garbage collector. You go first.

Re:I'll show you mine if you..

The problem was not the garbage collector if you read TFA. The articule is just a shameless plug, some ad spam for some very obscure profile.

This just tells us once again that our wonderful editors on /. don't even try to understand what's behind an article, but they just find some sensationalistic title (the more AntiMS, the better) and done. This results in more comments of the type... "See, M$ id teh SuCkS", or "thanks god for my Linuzzz."..., so they got more profit for their /. ads (oh, the irony often MS ads, BTW).

Yellow press..... yes, I know, /. is not supposed to have any credibility like any other parasite news sites, but anyway....

Re:I'll show you mine if you..

Next you'll be telling me that I'm not a nerd and this stuff doesn't matter!!!

Re:I'll show you mine if you..

Slashdot is a blog not a news source. I wish more people would realize that.

...

News for Nerds. Stuff that matters.

O.K., I'll bite, what part of that line do you not understand? If /. billed itself as a "blog" then I'd understand your point. However, /. is a corporate based, self-billed "News" site. Whether reader submission driven or not, it is a news site. You are wrong. But, thanks for your thoughts.

Slashdot not a news source? Agree!

Slashdot is a blog not a news source. I wish more people would realize that.

Agree. It's time to get this out in the open.

People complain that Slashdot sucks: the headlines are sensationalistic, the editors get commissions based on the number of dupes they post, and articles about 6-month-old events get posted as "news".

So why do I even bother visiting Slashdot? The answer is two things: the community of posters, and Slashcode moderation.

The value of Slashdot is in its community. You and I, dear Slashdotters. Our collective mind will pick through the various articles, point out their flaws, expose sensationalist FUD for what it is (and, surprisingly, will do this equally for anti-Linux and anti-MS FUD), debate various trends, and provide a signficantly international (though heavily USA-centric) perspective.

This value is enhanced by Slashdot's moderating system, so that information and insight can bubble to the top among the mass of inane posts. Metamoderation limits the amount of crack that the moderators can be on.

So, Slashdot editors, take note! *WE* are the reason we are here. *YOU* are not. Many of us don't even bother to read the articles any more, preferring to soak up the collective wisdom of techies from varying age groups and fields. If you piss us off, and the collective community of Slashdot deteriorates, then there's no reason for me (or others) to keep coming back.

Think about it.

Re:I'll show you mine if you..

Fuck. You are so right. This article is just some advertisement for some obscure profiler.

However I take exception to your use of the world "Editor". Slashdot does not have Editors. They have guys who accept submissions.
They don't read The Fucking Articles, They don't check links, The don't edit submissions...

Re:I'll show you mine if you..

Its quite clear in the article that they forgot to unregister their "deleted" objects from events. Since they were still registered, they weren't garbage collected. And rightly so. This was THEIR programming mistake, and has nothing to do with a GC bug in C# or any such thing. Fuck slashdot is pissing me off these days... as soon as they see a story that could be spun as "Microsoft screwed up!", they publish it withouth any fact-checking (or even reading the goddamn article!).

I wonder is MS could sue Slashdot for slander?

Slashdot *does* have editors. When I submitted...

Slashdot has editors. I know this, because the stuff below "nil0lab writes..." is heavily editted from what I actually submitted! In fact, I started my actual submission with something like "in a shameless plug for some code analysis product..."

Re:I'll show you mine if you..

Ditto, I submitted an article about an article that was about a survey done of corporations which concluded that corporations would not quickly adopt Vista. (This was before it was released, which frankly is a 'duh'... do corporations quickly do anything IT related?) But anyway... I spent a lot of time and had someone else read over it, attempting to get the *perfect* Slashdot submissions. There were no grammar errors, no spelling errors, the link was directly over the right keywords, etc.

The Slashdot editor who posted it moved the link so it looked like I was linking to the original study, not the article about the study. It's like they felt compelled to make a change, so they made one even if the change didn't improve the quality of the article.

I will say that the rest of the text remained unchanged, and really the only problem with the submission is that people who thought they were going to a study were actually going to a newspaper article about a study, but the point is Slashdot editors *do* make changes all the time.

in the URL

Slashdot editors are even more pathetic than I thought they were. It's bad enough that they didn't skim through the article, but they apparently didn't even take a look at the URL. Look at this thing:
http://www.codeproject.com/showcase/IfOnlyWedUsedANTSProfiler.asp
"IfOnlyWedUsedANTSProfiler"? That didn't raise any flags?

Of course, I'm trying to assume good faith and not just conclude that the editors knew this was an advertisement, but they sure are making that difficult.

Re:I'll show you mine if you..

It's not the garbage collector's fault. If an object is still in use, it can't be collected and destroyed. Managed memory only prevents the kind of memory leak where the programmer "loses" all references to the memory and thus never frees it. It also prevents the kind of bug where memory which is still in use is freed. Programs usually crash when that happens (either the OS terminates them due to a memory protection violation or they overwrite their own data and crash later on). That is also what would likely have happened in this case if it weren't for managed memory, because obviously the programmers mistakenly thought that these objects were no longer in use, so they would have freed them when they were still handling events.

Perpetual motion machine vendors

A funny thing happened with during my co-op this summer:

I was working at a coal-fired power plant which needed a new pollution control device before 2010. There, I would dig through the literature, and try to find suitable products and operating conditions for this device. Anyway, this involved a lot of meetings, conference calls, and business lunches with the suppliers in question.

Then there was Joe.

Joe was our Alstom sales rep: portly, humorless, slow to speak and slower to understand. He was also a devote Utahnian.

Well, one day, we were killing time while waiting on a conference call, my supervisor left the room, and we started talking about universities. Then he dropped the bomb:

"In my Senior year, I worked on developing perpetual motion machines."

My supervisor then reentered the room, and we got back to work. I felt like I'd just seen a dancing frog.

Re:I'll show you mine if you..

There's also the issue where you need to explicitly remove your event listeners when you no longer need the object. The listener keeps a reference to the object (via the interface) so even if it goes out of scope or what-have-you, YOU may think you don't have any references to the object but it implicitly does, through the listener you handed to the system. So... if you're using event listeners, make sure you explicitly remove them in your object's destructure... or else you'll end up with a memory 'leak'.

Re:I'll show you mine if you..

Mod parent up, and anyone doing any AJAX coding pay attention. This is one of the easiest bugs to create in JavaScript. JS is a very powerful language with a Self-like object model which a lot of people seem to treat as a basic variant. This means that they create closures when they intend to pass function pointers and end up creating large numbers of object references (the closure itself is an object and it retains a lot of references) preventing the garbage collector from doing anything useful. This is particularly common with AJAX where closures are often used for handling asynchronous events (which would be a gorgeous coding style if the syntax were slightly less ugly).

Well, there's your problem!

We looked through the code on paper, literally line by line, and just couldn't for the life of us imagine what the problem was.

This may be the least effective method of debugging in existence.

Re:Well, there's your problem!

No no, that would be something along the lines of printing out the code and then throwing darts at the listing to figure out the incorrect line. I hear it is popular in Redmond, although they reputedly use chairs instead of darts.

Re:Well, there's your problem!

No no, that would be something along the lines of printing out the code and then throwing darts at the listing to figure out the incorrect line. I hear it is popular in Redmond, although they reputedly use chairs instead of darts.

That makes sense: Since chairs are larger than darts, you have a much greater chance to hit the bugs.

Re:Well, there's your problem!

I'm guessing you wanted me to laugh, but in a highly parallel system, going through the code line by line is often the most effective way of finding a problem. When you catch the bug on a running system, you can see the state it's got itself into. But to figure out how it could have got there, you need to read and analyse the code. People are often far too eager to jump into the debugger when critically analysing the code is often a better solution.

It obviously doesn't work in situations like this where the bug is in the runtime and not the application.

Re:Well, there's your problem!

just to be clear, THE BUG WAS NOT IN THE RUNTIME, not by any stretch.

there are very clear constructs in place in the language/runtime to allow any object to unregister itself from event registrations it initiated.

this was VERY MUCH a bug in the end-user software, not the runtime (i've written code almost IDENTICALLY to this and blew lots of time having made this same mistake).

the only thing the runtime could do to protect the idiot developer (myself included) is automagically make all event references WEAK references, but that has plenty of undesirable side-effects too... in clr, you can do this yourself if you're so inclined... (just like in a JVM)

cheers.

Peter

Re:Well, there's your problem!

When writing parallel software, you need to treat the interactions between different threads of execution the same way you treat interactions between different code elements. This is a *DESIGN* issue. That's why debuggers aren't usually useful for finding such problems. Debuggers let you find sequential logic errors but rarely give you any insight into state coupling errors that only occur during real-time execution of the program. This is true whether the error occurs between different threads of the same code or different processes of the overall program. For some reason, people are particullarly unable to see such problems between different threads.

I first ran into this sort of problem in 1983 when working on a CDC mainframe. The only way to find the bug was the line by line analysis method since even compiling the code with debug caused it to run slower and the nature of the problem changed. That's as much detail as I remember.

I expect to see a lot more of these kinds of errors pop up as multi-core CPUs become more prevalent (true parallel execution) and people continue to assume that they can just crank out code without taking the time to understand the design. I'd also expect the prevalence of multi-core processors to create a demand for more parallelism. If you don't take advantage of the additional cores, your program will only be as fast as if it were on a single core system. If the competition can create a program that uses the additional cores, your program will seem slow.

Cheers,
Dave

Re:Well, there's your problem!

It obviously doesn't work in situations like this where the bug is in the runtime and not the application.

RTFA. FTFA:

Though we thought we had cleared all references to old entries in the list, because the objects were still registered as subscribers to an event, they were never getting deleted. We added one line of code to remove the event subscription and, over the next three days, we successfully ran the car for 300 miles through the Mojave desert.

As another poster points out, this is just an advert for a profiler, which helps people who use coding practises that they did not initially fully understand. As much as I wanted it to be a bug in the C# runtime, it's just another PEBKAC issue. The /. article introduction was wonderfully ambiguous on this point, if anything it was inflammatory ("C# memory leak"). Poor article selection if you ask me, but it's been many years since the /. editors genuinely cared about the content on this site rather than the number of hits/adclicks.

Re:Well, there's your problem!

Mod myself funny, please.

Slashvertisement

The linked "article" is just a "sponsored review" for a C# profiler...

Re:Slashvertisement

The sad thing is that Microsoft offers a perfectly servicable profiler for free that can be used on any C# application and is better than most commercial native Win32 profilers...

Re:Slashvertisement

Of course it has nothing to do with differences between "managed code" and native, right?

Re:Slashvertisement

Yeah, it's kind of bizarre. I'm not part of the crowd that actually believes the /. people get anything for posting these articles.

But I do believe that articles written by companies pretending to be written by end-users are not terribly useful and probably shouldn't end up on /.

I mean, the article clearly states at the top "By Red Gate Software.".

So where did the "Bryan Cattle reflects on ..." credit come from? Some random line towards the bottom of what appears a highly edited blurb?

Seriously.

"One of our team members downloaded the 14-day trial of ANTS Profiler"

"To our amazement, it was only minutes before we realized that our list of detected obstacles was never getting garbage collected"

"If Only We Had Used It Earlier..."

ANTS Profiler helped us fix a problem in minutes that would have taken us weeks to track down. If only we'd thought of it before the competition, we would most likely have finished the entire race and had a chance at the top prize money.

All this stuff sounds either very naive or very marketing. You choose.

Re:Slashvertisement

I think this is just poor code debugging skills. The new garbage collected languages (C# and Java) seem to be creating coders that don't know the basics of debugging. If this was C++, the bug would have shown up when an event occurred and the code called a deleted object. It would have been harder to track down. In any case, I would have added code to analyze the state of the stack and it's objects since it only occurred after the program/car had been running for a while. Also, had it not occurred to them to increase the number of obstacles beyond the "controlled" environment? You can easily write test code to to add a lot of objects into the stack without the use of the car.

Our cars used an asynchronous event-based code stack written from scratch by us.

They wrote their code stack from scratch. They should have modified an existing one that was proven to work. Either way, I think this is a case of sloppy programming. I don't think it is worth mention aside from the case what the closest thing to a memory leak is that you can have. Unfortunately, that too should have come to mind from them. They ruled that out because "most of [their] code is written in garbage-collected C#". Just because a language is garbage collected, it doesn't mean you can just ignore the issues. All reference to the object must be removed for it to be deleted. This is definitely a case of sloppy programming. I would be embarrassed if my name were attached to this.

      Some old Visual Basic programmer jokes come to mind when I read this article. People use to make fun of Visual Basic programmers because it was to easy to write programs in VB. They thought it would produce sloppy code with errors, and other similar things. To some extent, it appears that same case could be made for C#, and to a lesser extent Java. In the end this is simply a case of not stress testing their event stack.

Re:Slashvertisement

Agreed. I was a good VB programmer. But my VB experience was an eight-month interval between C++ jobs. I've knocked out minor MS Office applications when needed since then, but that's it. I'm sorry to say that most of the VB programmers I've worked with were very poor engineers. Admittedly I've not worked in VB for wealthy companies whilst I have in C++, so that colours things somewhat. I don't doubt that there are some good VB programmers out there. But in the cases of most of those I worked with, I could very clearly see how VB led them to be poor engineers. The amount of shortcuts and wizards and instances where they would start their program by dragging a form object onto the design panel and dumping form objects onto it was obviously a leading reason for their poor skills. VB *led* them to take this approach. It works for small Excel apps, barely for database front-ends and not well at all for large projects.

As is common with the lower end of Microsoft products, the selling point is that they make it very easy to do what they think you want to do. The ability of VB to knock out an interactive form with near-zero knowledge of programming has encouraged a lot of colleges to sell people the idea that a ten-week course of dragging and dropping text box objects is programming whilst a lot of cheap or ignorant employers have taken the graduates at their word and plunged them in over their heads.

VB is a poor language in many ways and not, imo, suited to a large or sophisticated project. But you can find good VB programmers (was one). It's just that it encourages bad ones.

Oh, the summary is also wrong. C# hasn't started springing leaks. The programmers missed a reference to objects that they were creating and the garbage collection therefore never triggered to unallocate the memory. I don't doubt it's not easy to automate a vehicle to drive any even 9 miles, but this could have been detected with more thorough debugging. At any rate, the article submitter and overseeing editor should be ashamed of twisting this into an anti-Microsoft jab. I'm a Linux programmer. I can tell you that Linux can compete happily without sinking to the level of lies and misinformation.

Don't Slashvertise. Ever.

Come on. Really. What kind of idiot marketer sends in stories like this to Slashdot? We know what happens. First, you get derided mercilessly for trying to sway us with your ridiculously transparent attempt at marketing. Then, the real experts come out and poke holes in everything you've said. Then everyone else chimes in with better (and often free) alternatives. You and your company end up looking like buffoons, and your product ends up looking like utter garbage.

You may think you're pulling one over on the editors, and maybe you are. But you aren't pulling one over on us, and I think after all these years, the editors know this. So, just don't. Unless your product or service is absolutely bulletproof people here are more likely to shoot it full of holes than rush out and buy it.

Re:Slashvertisement

It might be slashvertisement, but it also speaks a painful truth that many developers seem to forget.

The company I worked for, in the efforts to get something out the door, deployed a product to a customer site that had a similar flaw (but, not a .Net app). Every hour or so, it would simply lock up. The solution that was proposed was a script to restart the application at a specific interval rather than track down the actual bug itself and fix it. And, like the Princeton team, they underestimated the time interval when the system was put into a real production environment and more users came online.

In my own work, I wrote NT services that HAD to run 24x7 and were not allowed to crash - especially due to memory leaks. The components we purchased and used, contrary to their marketing ploy, often had memory and resource leaks - we won't even begin to talk about the runtime library that shipped with the compiler.

I used a variety of freely available memory managers and commercial QA tools to track down most of the "leaks" and fixed them. If I didn't have source to the component in question, I replaced them or rewrote them from scratch taking time to make sure it didn't leak. Guess what? It worked and those applications/services run 24x7 (well, until they restart server for some other reason).

Moral of the story - if something is critical - take the time to profile your code and use QA tools to find other potential problems BEFORE you deploy.

RD

Stupid Slashdot headline

This is a stupid, stupid article headline. Of course you can have a memory leak in a managed language! Any Java programmer who's decent understands that.

It's not C#'s fault. The team had references to the obstacle list (event handlers), which prevented garbage collection. The .NET CLR did its job, just like it was supposed to.

Re:Stupid Slashdot headline

There is no explicit delete in C#. They may have added something in their framework, or they're just saying that they called delete on the collection where they thought they stored all objects. The GC isn't psychic. (It could be a good thing to ask the GC to delete an object explicitly and get an exception if it's indeed still referenced elsewhere, but that's not possible in C#, or Java for that matter.)

Re:Stupid Slashdot headline

I think you're getting hung up on the method name. There is no standard "delete" function that marks something as unused (dispose on the other hand sort of gets there). The article itself is unclear but I would assume that they were simply deleting the collision objects from a collection of potential hazards. Whilst that would remove the object from the collection itself it is *not* a delete. As references to the object existed elsewhere the object still exists (look ma, no null pointer exceptions) no delete happens. You cannot specifically say to the GC "We're done with this, delete it", the GC sweeps on a regular basis looking for objects with no references.

Would you really want the GC deciding that just because an object is no longer part of a collection it's safe to unsubscribe it from events and delete it? I know I wouldn't.

Re:Stupid Slashdot headline

You could just use weak references (does C# support that?) for all references that the event sender uses to reference its event receivers. In that case, as soon as a receiver has been removed from all those "obstacle lists" and all other objects that held references to it, the weak reference would be the only reference pointing to the receiver, which would make the receiver eligible for garbage collection.

Re:Stupid Slashdot headline

You might not be able to use weak references since they introduce (at least in Java) a layer of indirection. For example, an addListener method usually takes an interface of some kind of Listener, not a WeakReference (to a listener).

Now, if you have control of the implementation of the object who accepts Listeners you can store them internally in a weak collection, which allows them to be garbage-collected. This would work but may not be what the programmer intends. Actually in a language like Java I'd hazard that usually the programmer wouldn't want that at all: consider an application that listens to UI events. As a programmer I want to be able to stick listeners wherever they are needed and leave them there permanently. If I don't need a pointer to the object, I don't want to keep it around, and thus may not have a reference to the listener EXCEPT in the event-management collection. That's the advantage of GC languages: as soon as the object which creates those events (say, a dialog box) goes away, the objects it refers to have one fewer pointer and may be eligible for GC.

Anyway, lots of code has issues like this: we had a problem at my work where an Apache taglib was caching some compilation in a cache that would grow for ever. It was a simple code fix to solve that problem, but there was no way for us to even SEE the problem until we ran our application under load in a profiler. Fun fun fun.

Re:Stupid Slashdot headline

I do think that this "object lifetime" bug is NOT a memory leak, but that does not change the effects (crash) of the system.
What is interesting is to see that garbage collection changes one class of bugs (forgetting to explicitly deallocate memory) to another one: unintentionally keeping objects around. Princeton's "obstacle object" lifetime policy was stepped upon by a Dotnet library; Java has similar problems in its libraries. For the Princeton car software, an explicit deallocation routine (like in C/C++) would have been easy to implement.

Problem is that both C/C++ style memory leaks and C#/Java hidden reference bugs usually remain hidden until the system crashes or trashes after some time. It makes them hard to find in the course of ordinary testing.