Forgot your password?
typodupeerror
Programming

Modeling How Programmers Read Code 115

Posted by Soulskill
from the read,-comprehend,-complain-about-the-dunce-who-wrote-it dept.
An anonymous reader writes "Following up on an experiment from December, Michael Hansen has recorded video of programmers of varying skill levels as the read and evaluate short programs written in Python. An eye tracker checks 300 times per second to show what they look at as they mentally digest the script. You can see some interesting differences between experts and beginners: 'First, Eric's eye movements are precise and directed from the beginning. He quickly finds the first print statement and jumps back to comprehend the between function. The novice, on the other hand, spends time skimming the whole program first before tackling the first print. This is in line with expectations, of course, but it's cool to see it come out in the data. Another thing that stands out is the pronounced effect of learning in both videos. As Eric pointed out, it appears that he "compiled" the between function in his head, since his second encounter with it doesn't require a lengthy stop back at the definition. The novice received an inline version of the same program, where the functions were not present. Nevertheless, we can see a sharp transition in reading style around 1:30 when the pattern has been recognized.'"

This discussion has been archived. No new comments can be posted.

Modeling How Programmers Read Code

Comments Filter:
  • by phizi0n (1237812) on Saturday July 06, 2013 @11:27PM (#44207093)

    This article is complete garbage. They tested 2 people with different code that produces the same results and then make up a narrative of how novice and expert coders think in different ways. Use the same code to test a much larger pool of programers and then the results might actually be interesting.

    • by tricorn (199664)

      It might help if you actually read the articles linked to. It's an ongoing study, and results are "very preliminary".

      It looks like they have various code samples. The same functionality is coded in different styles. They're studying both how novice vs. experts read code, as well as how coding styles/language features affect comprehension.

      • by Wildclaw (15718)

        It looks like they have various code samples. The same functionality is coded in different styles.

        More like it is coded in good and bad styles. The video at the top is the worst, full of bad style. Just to go through it all:

        * Bad naming of parameters. Both collections and iteration variables feel like the same.
        * No methods and data of program wide priority (x, y) declared at different locations.
        * Using "between" instead of "in interval. There are only really one reason to use exclusive intervals, and that is when the algorithm naturally ends up on it (for example when looping over zero-bound arrays). Ot

        • by Rockoon (1252108)

          with the looping variable properly renamed from x_n to i.

          'x_n' is not an index, proving that you still havent comprehended the code....

          The variable 'x_n' is a member of the list, not an index into the list. While 'x_n' might be a bad name, 'i' is a horrible name.

    • by mwvdlee (775178)

      Yeah, somehow I think their expert Eric compiling a function in his head is somehow affected by the fact that he actually HAD functions in his version of the code to compile. If you were to run any type of code metrics on those fragments, the inline version would be rated as much more complex.

    • Yes, complete [and utter] ...

      I've been programming for 40+ years and how I analyze code [and hence, my "eye track"] varies, depending upon what I'm looking for. If I need to know one thing, I'll tend to zero in quickly. If I'm going to make changes, I'll scan over everything [at least] once to try to glean the overall style so I'll know what I'm up against when/before I start to make the changes.

      Not only that, but the article's last line is about "joining the experiment" if you're in the Bloomington, IN a

    • Also, I suppose the code was written either by a Python novice or a FORTRAN expert :D
      I'm more of a Ruby guy, but I learned Python a few months ago, and one nice thing about it are list comprehensions (the syntax will probably get messed up by /.):

      x_n = [2,8,7,9,-5,0,2]
      y_n = [1,-3,10,0,8,9,1]

      def between(numbers,low,high):
      return [x for x in numbers if x>low and xhigh]

      def common(list1,list2):
      return [x for x in list1 if x in list2]

      x_btwn = between(x_n,2,10)
      y_btwn = between(y_n,-

    • We tested a total of 29 people in the eye-tracker. Here are some more videos: https://www.youtube.com/user/synesthesiam/videos?sort=dd&view=0&tag_id=&shelf_index=0 [youtube.com]
  • Video Speed? (Score:5, Interesting)

    by Psychotria (953670) on Saturday July 06, 2013 @11:37PM (#44207127)

    Is that video real time (adjusted for the 300Hz sample rate)? I ask because I'm not a Python programmer (I do know C, C++, asm) but about 10 seconds into the video I knew what the program would print and yet the video went on for 3 minutes. Something does not add up.

    • I just watched Eric's video. Which is different code (wtf?). Even then I wouldn't read it like he does. They definitely need to refine their experiment's methodology a little bit (ok, a lot) and get a bigger sample group.

    • by Anonymous Coward

      This might be it:
      Expert programmer + Phython programmer = NaN

      • Re:Video Speed? (Score:4, Insightful)

        by Psychotria (953670) on Sunday July 07, 2013 @03:29AM (#44207733)

        Although I don't necessarily agree with your Python joke (pretty funny, though) your comment does provoke me to wonder if Eric really is an "expert" programmer. His eyes go all over the place even when they don't have to. There's only 2 conditions to remember and one "function" (set intersection). His eyes skip all over the place. Further, after the two sets are formed (he's written them in the box and he's already determined what common() does) why is he even looking at the function common() again? Maybe it's to double check, but surely remembering two conditions and a set intersection (3 operations) is well within the grasp of human short-term memory and surely you'd only look at the function name (what it does was verified earlier). Personally after I'd glanced at between() and common() to confirm they did what they suggest they do, I'd never look at them again. But his "expert" eyes keep going back to them while he is forming his output.

        • Adding to my own comment. There's only one condition to remember; I was getting confused with the inline version of the novice. Why he keeps looking back over their code/implementation is bizarre to me... it kind of defies the point of functions in the first place. I'm not an expert but I reckon he's not an expert either.

        • His eyes go all over the place even when they don't have to. There's only 2 conditions to remember and one "function" (set intersection). His eyes skip all over the place.

          My eyes often go all over the place because I'm restless a lot of the time. It doesn't necessarily need to correspond to my mental processes, which is completely lost on the eye tracker. I really don't think that it's possible to make far-reaching conclusions from strictly local stuff like this. The skills in comprehending large programs seem much more important to me (just as it's precisely those large programs that need those skills in the first place), and an eye tracker alone won't help you with researc

        • by Shalcker (989572)
          Maybe what actually gets committed to short-term memory is "there is definition of performed function _here_ (eye position)" - transforming it into memorized conditions and intersection would take longer then to simply reparse it, so brain takes easy way out. Programmer can remember function, but doesn't if he doesn't have to.
          • Yeah I agree, but watch Eric's (the expert) video again. He doesn't just glance at the function definition; he steps through it.

            • In C, for example, do you step through printf() every time you call it? Of course not... you probably don't even have the source code. The only thing that matters is input, output, pre-conditions and post-conditions. But, yeah, maybe expert Python programmers think differently.

        • by Livius (318358)

          Perhaps the novice is expecting the code to follow a straightforward algorithm, and the expert just assumes Python will be full of counter-intuitive operations and hidden side-effects buried in spaghetti code.

        • by guruevi (827432)

          I work with eye trackers on an almost weekly basis. These videos are severely slowed down for us to see the interaction. And yes, us humans have the tendency to read an entire page over and over again (subconsciously) even if entirely unnecessary. The saccades may have been scanning for more information or indeed to confirm something (our memory is incredibly short and prone to error, programmers definitely would know that they can't trust what they remember about a function or input data) or simply thinkin

  • by Anonymous Coward

    I don't even read it, I close my eyes, which are a hindrance, and use my inner eye to feel the code. I become the code. Most times I look like a bowl of spaghetti

    • by Livius (318358)

      "With the blast shield down, I can't even see!"

      "Your eyes can deceive you - don't trust them."

  • Bad link (Score:5, Informative)

    by Anonymous Coward on Sunday July 07, 2013 @12:05AM (#44207225)

    They should link to the follow up post that talks about the experiment with 162 programmers http://synesthesiam.com/posts/what-makes-code-hard-to-understand.html. It also links to the paper that has even more information.

  • Invalid results? (Score:4, Insightful)

    by flimflammer (956759) on Sunday July 07, 2013 @12:11AM (#44207245)

    The code between these two individuals is completely different, even if it produces the same results. How do you discern any meaningful results out of two people reading two different sets of code?

  • by Anonymous Coward

    the style of the code might also make a difference (as well as the specific languages use of form)

    Code of more than average complexity also shifts the reading patterns, as would non trival agorthms that require more study to figure out what it actually does

    I have my own code style that assists in fast scanning and placements of specific language features for when Im working on a project of 100000 lines of actual code (30 years a programmer and I also dont care for having 100s/1000s of tiny files spreading

  • Some call centers / help desks suck with BS metrics and scripts.

    We need less BS metrics as people just game them and people who do a good job have poor metric scores and some with lines and lines of bloated code gets a good score.

    • by PolygamousRanchKid (1290638) on Sunday July 07, 2013 @02:11AM (#44207545)

      Unfortunately, this thing seems to be en vogue in the computer fashion industry. I just attended a conference where this phrase could some up a bunch of the presentations:

      "We are modeling, tapping into the power of social networks, and doing visual analytics!"

      I happen to be reading The Psychology of Computer Programming, Silver Anniversary Edition" right now. An interesting quote:

      The only thing that's changed here in twenty-five years is the fact that the funds dedicated by executive to eliminating programmers from their payrolls have become far more staggering than I imagined back then. And, now, I finally recognize in this executive desire a pattern so strong, so emotional, that it has blinded these executives to two facts:

      1. None of these schemes has succeeded in eliminating programmers . (We have now at least ten times as many as we did then.)

      2. Every one of these schemes has been concocted by programmers themselves, the very people the executives want so passionately to eliminate.

      So, although people say that programmers lack interpersonal skills, they evidently have a skill at persuasion that surpasses that of the late, great P:T: Barum, famous for his theory: "There's a sucker born every minute."

      I guess if I need some money for something from executive, I'll tell them that I need it to model, tap into the power of social networks and do visual analytics. That ought to get me my funds.

  • It seems line Hansen is still a novice too - sampling 30fps video at 300 Hz...
    • by tricorn (199664)

      What makes you think he's sampling 30fps video? There's even a link to the eye-tracking hardware.

      http://www.tobii.com/en/eye-tracking-research/global/products/hardware/tobii-tx300-eye-tracker/ [tobii.com]

      • A human being's reaction time is in the order of 100ms, so sampling at 300 Hz is ridiculous whichever way he does it.
        • Re:300 Hz (Score:5, Informative)

          by ThreeKelvin (2024342) on Sunday July 07, 2013 @06:12AM (#44208157)

          Human eyes "flicker" when they look at something. They will remain stationary for a time, then move quickly to another position. (See Saccade [wikipedia.org].)

          The time for one of the fast movements between positions is in the order of 20 ms when reading, giving us a frequency of about 25 Hz. (It's only half a sine wave, so the period is 40 ms.)

          Using the Nyquist-Shannon sampling theorem [wikipedia.org], we get that we must sample the eye movement with a frequency of at least 50 Hz, otherwise we'll get aliasing. Now, bring in the engineering rules of thumb, which say that it's no good riding on the Nyquist limit, but you'll need to oversample the signal a bit in order to get a useable result (It of course all depends on what you'll be using the signal for. In feedback control you usually oversample by a factor of 8-20, and in signal processing in the neighbourhood of 2-8) and you end up with a samping frequency of 100 - 400 Hz.

          So, in summary, 300 Hz sounds like a perfectly good sampling frequency, perhaps even a bit in the low end, depending on what you'll use the sampled signal for.

        • And _that_ is why so many A/D systems fail miserably: because people have been very, very confused by sampling theory.

          Just because it takes 100 msec to respond does not mean that the eye motion takes anywhere that time, and the motion is not "clocked" or linked to some discrete frequency. It's analog, and to measure its impulse driven movements properly you need to oversample temendously, or use some sort of triggered sensor that can record its triggers very accurately.

        • "A human being's reaction time is in the order of 100ms, so sampling at 300 Hz is ridiculous whichever way he does it"

          It seems perfectly reasonable to me, so I have to ask: What frequency do you recommend?

        • Saccades are on the order of 10s of milliseconds: http://en.wikipedia.org/wiki/Saccade#Timing_and_kinematics [wikipedia.org]
  • Dupe? (Score:3, Interesting)

    by dohzer (867770) on Sunday July 07, 2013 @12:48AM (#44207349) Homepage
  • If you want to know what a code prints, just evaluate it, it will tell you instantly without having to worry whether you made a mistake in evaluating the code in your head.
    Now if you want to understand the code, that's something else, but it's not what's asked here. The code is sufficiently straightforward that the only explanation you could give is the code itself.

  • When reading the lines

    In this last 30 seconds or so of the novice video above, you can see her back-and-forth comparison of the x and y lists. If you look carefully, however, the red dot (her gaze point) is often undershooting the numbers on both lists. Why is this? While it could be a miscalibration of the eye-tracker, the participant may also have been using her parafoveal (the region outside the fovea) to read the numbers. This and the fact that foveation and visual attention are not necessarily always t

  • It's funny how there's huge amounts of people criticizing flash on slashdot every day, and then we have an article with a flash video (I can only assume it's a video, since I don't run flash) attached to it.

    Do these guys even know how their target audience is?

  • There is a follow-up blog post here with more data: http://synesthesiam.com/posts/what-makes-code-hard-to-understand.html [synesthesiam.com] and there are many more videos available here: https://www.youtube.com/user/synesthesiam/videos?sort=dd&view=0&tag_id=&shelf_index=0 [youtube.com]
  • How did they determine a programmer's skill level? Length of time as a programmer? Ability to find and fix bugs? Self-assessment? Some other metric? Maybe they should have asked on Slashdot how to determine a programmer's skill level. That would have yielded the one, definitive, indisputable method of determining a programmer's skill level.

Philogyny recapitulates erogeny; erogeny recapitulates philogyny.

Working...