Amazon.com Widgets Measuring Developer Productivity

Measuring Developer Productivity

By Nick at November 11, 2012 12:19
Filed Under: Delphi, Leadership, Software Development

Introduction

If you go to Google and search for "measuring software developer productivity" you will find a whole lot of nothing. Seriously -- nothing.

Well, okay, not exactly nothing. You'll get a ton of links. But almost all of the links you find will talk about how measuring the productivity of software developers can't be done effectively. Some people will even argue that it shouldn't be attempted at all. Some others will describe techniques to measure developer productivity, that, well, everyone else knows don't really work.

There have been many valiant attempts to measure developer productivity, but all seem to end in less than successful territory. (We all know to laugh at "Lines of Code" as a productivity measure). Virtually any objective measurement you can apply to the software development process can be "gamed" into submission.

There are just too many variables to account for everything that goes into the development process. Measuring any one of them, or even a combination of them, simply cannot begin to capture effectively everything that is involved. You can't pick a surrogate thing to measure, but because of human nature, developers will react in ways that will skew the measurement.

Many studies have been done on this topic, with little to show for it. Every software development manager who has to fill out a developer's evaluation or determine who gets bonuses has struggled with this. Many a leading expert and guru has thought about and researched this thorny topic.

Experts Agree

And not one of the experts has come up with a good idea or method to measure individual productivity. There are plenty of good and effective ideas on measuring team and project productivity, but there is a severe if not complete shortage of ways to measure individual performance and productivity.

For instance, Robert D. Austin wrote an entire book on this subject called Measuring and Managing Performance in Organizations and came to the conclusion that unless you can measure 100% of something -- that is, every aspect of it -- then there is no point in measuring at all. If you attempt to measure less than 100% of the things involved in a given activity, you will get what he refers to as "dysfunction", or what you and I might call gaming the system. And since software development is an incredibly complex and creative endeavor, it's virtually impossible to measure every aspect of it. Often it quite literally involves sitting and staring at a computer screen. How do you measure the productivity of that?

Here's Martin Fowler's conclusion in an article he entitles " CannotMeasureProductivity":

"This is somewhere I think we have to admit to our ignorance."

Neil McAllister of InfoWorld writes in an article entitled " The futility of developer productivity metrics"

"For most other companies, however, it might be best simply to forget about the idea of measuring developer productivity and rely instead on tried and true methods. That's right: A highly effective, productive developer workforce is often the result of high-quality, effective management. Unfortunately, nobody has developed a metric for that yet. Coincidence?"

Joel Spolsky writes:

"It's rather hard to measure programmer productivity; almost any metric you can come up with (lines of debugged code, function points, number of command-line arguments) is trivial to game, and it's very hard to get concrete data on large projects because it's very rare for two programmers to be told to do the same thing."

Steve McConnell, a widely regarded academic, author, and consultant says:

"So while I see the value of measuring individual performance in research settings, I think its difficult to find cases in which the effort is justified on real projects."

Like I said, it's hard to find a leading expert making the case for objective measurements, and easy to find plenty that say you shouldn't even try.

So I don't think that it's a tough leap to agree that there is no objective way to measure productivity in software developers. I know this is controversial, but it seems quite clear to me that software is more art than science, and measuring productivity in art is, well, simply not done. One does not tell Michelangelo that it took him too long to paint the Sistine Chapel because he should have been painting 1.4 square feet per day instead of 1.2. That would be preposterous.

So, What Are We to Do?

Anyone who is charged with managing and leading developers is left in a quandary. We want to be fair, but if everything is subjective, how can we be?

The fact that there is no objective measure of developer productivity doesn't mean you can't measure it. It just means that you have to measure it subjectively . And we do that, don't we. You can look around your shop and just know who the productive developers are. There are probably developers you know that simply crank out good code faster and fix bugs more efficiently than the rest. For ease of discussion, I'll give this developer the name "Jason".

And just how do you know Jason is more productive and more valuable? You just do. You can just tell. You know it when you see it. But if you stop and think about it, Jason has some characteristics that you can figure out and write down.

What I mean is that if you stop and examine Jason and his work, there are things that you can say about him that describe what he is all about. They are subjective, sure, but you can write them down and give an "objective" measure to these subjective things.

A Modest Proposal

So yeah, what I am proposing is a way to measure developer productivity and value via a subjective set of measurements. Think of it as a way to qualify the notion of "you just know that Jason is productive".

Here is the list:

  • Productivity
    • Does the developer get a reasonable amount of work done in a given period of time?
    • Is the developer's velocity on bug fixing sufficient?
  • Engagement
    • Is the developer dedicated to his/her craft?
    • Is the developer committed to delivering software on time?
    • Is the developer dedicated to company success?
  • Attention to Quality
    • To what degree does the developer's code work as designed?
    • Does the developer thoroughly test code and believe it to be correct before checking it in?
    • Do a minimal number of bugs get reported against his/her code?
    • Does the developer write unit tests for all new code?
    • Does the developer follow the Boy Scout Rule and leave a module cleaner than it was before he or she worked on it?
  • Code Base Knowledge and Management
    • To what degree does the developer understand the code base assigned to him/her?
    • Does the developer take responsibility for his/her team's code base, improving it at every opportunity?
  • Adherence to coding guidelines and techniques
    • Does developer's code routinely meet coding standards?
    • Do code reviews reveal a minimum of problems and discrepancies?
    • Does the developer use Dependency Injection to ensure decoupled code?
  • Learning and Skills
    • Is the developer constantly learning and improving his/her skills?
    • Does the developer show a passion for the craft of software development?
  • Personal Responsibility
    • Does the developer first assume that the error lies within his or her code?
    • Does the developer understand that he or she is solely responsible for their code working correctly?
    • Does the developer take pride in their code, ensuring it is clean, tested, easy to read, and easy to maintain?

Again, these seven items are all subjective measurements. All the questions require the evaluator to make subjective judgments. But a good lead/manager is presumably in their position because of their good judgment about such things, and so they should be able to make fair and accurate assessments about Jason's productivity using these "measurements".

Most of the questions ask for a yes or no answer, but all can spark a discussion with management or even the developer being evaluated. You could use these measures on a formal evaluation form. Whatever you do with them, it's probably better than the "just know it when I see it" approach you are probably taking now.

Conclusion

In the end, any evaluation, judgement, or other assessment of the productivity or value of a single developer will be subjective. No one has yet found an objective way to measure individual productivity in our excessively complex profession. But I am saying that it is possible to qualify and make judgments about a specific set of criteria as a small and humble step towards being able to do the thing that our industry has so far found very difficult.

My suggestion is that we accept reality and embrace the subjectivity -- and lack of objectivity -- in measuring developer performance.

Comments (22) -

11/12/2012 6:57:25 AM #

andrew barton

Does the developer make an effort to understand the problem domain which the program under development is targeting?

andrew barton United States |

11/12/2012 7:03:08 AM #

nick

Excellent addition.

nick United States |

11/13/2012 10:20:46 AM #

Russell Weetch

I like this addition very much. I think it is a major problem at times

Russell Weetch United Kingdom |

11/13/2012 1:39:30 PM #

William Meyer

In my one employment by a really large company, this was one aspect of developer skills which was never considered by the managers in their annual reviews. This, despite it being a product which sells into a technically complex field, subject to compliance with shelves full of government requirements. To pretend that developers are a commodity, and easily replaced in such an environment is folly.

William Meyer United States |

11/12/2012 7:53:43 AM #

Doug Johnson

I will be sending this to my manager.

I've been managed badly and not managed at all (which works well because I can just get the work done), but never really managed well...although I think I am in that situation now (it's only 4 weeks...too soon to tell for both of us).

Thanks for pulling this together. Obviously you've been thinking about it...Annual review time?

Doug Johnson United States |

11/12/2012 9:14:50 AM #

nick

Doug --  I hope it is helpful.

It's a really sticky problem that I don't think we've addressed in our industry.

Nick

nick United States |

11/12/2012 1:11:42 PM #

Warren Postma

Technical managers who are competent usually develop a subjective list that they are qualified to use for assessment. I believe Nick is just such a conscientious, competent guy.   The problems occur, I believe when people who are not competent wish to create "silver bullets" to act as a substitute for professional competence when evaluating staff, and this is the enduring appeal of metrics, when used as a way to punish or reward developers.

Some metrics can be valuable if the team as a whole owns the metric, not the individual developer.  If the team as a whole wishes to say "We could be doing a better job at reducing the number of 30,000 line methods in our code", then metrics could be helpful.  But if SLOC or bug-fix metrics are used as a way of ranking developers, then all is lost.

The most poisonous misuse of numbers when evaluating programmers is Microsoft's Stack Ranking system.  Even if you're one of the best at your job in the planet, all that matters at Microsoft is if you're #1 or #6 on your team of 6 people.

Warren

Warren Postma Canada |

11/12/2012 2:30:00 PM #

Bunny

Agreed. Individual performance is measured in the area of consultancy, but also there the 'team level'. Usually things work, this is often forgotten.

Maybe a matter of incentives. Once ago at our university the Software engineering students went through their baptism of fire by developing a 'project' (not big) but merging several modules they had to develop ... the assistant professor locked them in and they have been allowed to go home when the final result worked. The outcome was very positive ... it worked.

Managers cause problems, the Scrum in my opinion is addressing one big issue (don't touch things twice) - avoid spinning. The way it is executed via implicit systemic peer pressure. What I like about the Scrum - finished or not, nothing in between - very beneficial. Scrum applies a relative measure - this can be helpful at the microlevel. Enterprise scrum is an illusion concerning the measure.

Afik Stackranking had been introduced to GE in order to change something, with the result that those with the good ideas left.

Honestly being No. 6 in a team does not say a lot too. The Microsoft Stack Ranking, btw. Google and Bosch use it too and most of Fortune 500 did but now about 20%. You should not have competition between employees. The Germans had to learn this lesson - they introduced competition between departments, afterwards between teams and individuals in the end. What all forget about is creating products and have an eye on the competition.

Have all teams with 20/70/10 and you are average (most profitable for an existing business indeed). The argument when introducing Rank an Yank was 'to give weak performers the chance to improve'. What is a weak performer? HR people evolved from sometimes little strange to professing sociopaths obviously - at least my impression.

Neo liberal thinking - no classic inspired view to labor. Humans don't work this way. Bad for those who want to see numbers. What you see is numbers but they say nothing, especially when brainwork involved. By measuring human's performance, applying rules ... you create competition - nothing but greed - angst (irrational fear for no reason). What you need is cooperation - teams can compensate this the reason why families and clans exist - but they are more powerfull than the hierarchary. This is the simply reason behind most of the new idea in HR - crack the team. MS lost this way ...

Bunny Austria |

11/12/2012 2:31:36 PM #

Bunny

The way Scrum is executed via implicit systemic peer pressure. -> I dislike.

Bunny Austria |

11/12/2012 4:23:39 PM #

plombier paris

Article super int�ressant. Beaucoup de discernement dans tes explications. Je met �a en favoris. Merci pour le partage !

plombier paris France |

11/12/2012 9:01:05 PM #

Wouter

You can scrap that whole list once the average manager  gets the choice between an expensive "western" developer and the first Indian guy who's willing to work for 1/5th of the salary.

Management thinks more or less like this:

The new junior only programmed some trivial stuff in his study but never did any real work? Hardly speaks English? Not motivated and doesn't understand the domain that he's working in? Development takes 3 times as long and results in an unstable and unmaintainable mess of code? After we've trained the guy he'll immediately leave the company to make more money somewhere else?

Who cares? The labor is practically free!

Wouter Netherlands |

11/13/2012 10:45:13 PM #

Glenn Forest

"The Parable of The Two Programmers" was published in the ACM's Software Engineering Notes 27 years ago by one of my professors.  Although very slightly dated, I've found it to be sooo true.  

http://mcfavell.com/classic/parable.html

And the end of the fairly short article, he writes: "...ask anyone who regularly teaches computer science courses in which programming projects are required. A five to one ratio between the shortest and longest program is quite typical, and usually the shorter programs are better."

Glenn Forest United States |

11/14/2012 1:44:31 PM #

Nils

I would recommend that you read:

The Mythical Man-Month: Essays on Software Engineering written
by Frederick P. Brooks Jr.

Nils Germany |

11/15/2012 10:09:49 AM #

plombier paris

Article super interessant. Beaucoup de discernement dans tes explications. Je met ca en favoris. Merci pour le partage !

plombier paris France |

11/15/2012 10:51:42 AM #

plombier paris

Article super interessant. Beaucoup de discernement dans tes explications. Je met ca en favoris. Merci pour le partage !

plombier paris France |

11/16/2012 2:42:52 AM #

David S

Over the years I've observed that programmers can be lumped into two general camps:

1) they spend time carefully crafting and optimizing their code, and then go back and re-optimize and minimize duplicate code;

2) they write some code that solves a part of a problem then take a copy-and-paste approach to duplicating anything they've already written that comes close to what's needed and tweak each copy to make it work.

By most measures, the folks in (2) will rank higher in the area of "productivity" than those in group (1).  Unfortunately, without code reviews, or a manager who ever bothers to scan the code his people are producing, nobody is going to know what's going on.  The copy-and-paste coders just seem to look like maniacs when it comes to producing code.  This approach might seem good for initial product development, but it SUCKS when the product moves into maintenance mode and there's very little new stuff being added to the code.

The people in group (1) are good candidates for the metrics identified above.  Those in group (2) are false prophets -- the metrics will make their work look outstanding relative to group (1) folks, while the long-term VALUE of the REAL work they've done is actually much lower than those in group (1) produce.

David S United States |

11/16/2012 6:34:04 AM #

nick

David --

You might like this article:

thatextramile.be/.../theres-only-one-valid-metric-for-developer-productivity-and-quality

Nick

nick United States |

11/20/2012 2:42:07 AM #

David S

Yeah, it's that "future effort" part that's the fly in the ointment.

David S United States |

11/20/2012 1:57:30 PM #

nick

It's ironic that you can't really measure the only true measure of developer productivity.  Wink

nick United States |

11/21/2012 5:28:20 AM #

Ciprian Khlud

The link shows that productivity has to be matched with quality and bug fixing. John Carmack describes how if they would use the quality control that he was working for NASA projects, he would never finish any of his "mainstream" projects. This brings to me the idea that programmers are productive if the quality can be easily validated or implicit.
If you have TDD for example you are sure that your functionality is to some level thought about. Another way to do this is to use frameworks/languages that minimize the rate of errors by either: reduce the number of lines of code (and hopefully this code is also easy to read) and reduce the number of errors per line. And about the most of the other errors, a static code analyzer by default it would reduce the defect ratio and implicitly would increase the productivity.

Ciprian Khlud Spain |

11/16/2012 8:57:15 AM #

andrew barton

Managers are motivated to favor type (2) programmers.

If they insist on a type (1) programming style, they have to justify the resultant slow-down to their superiors.

The company bosses ultimately have only two metrics: how long and how much.

andrew barton United States |

11/16/2012 7:30:41 AM #

andrew barton

We had a programmer who never fixed his bugs. Instead, he would always add on code that corrected mistakes caused by his previous code.

His productivity was impressive.

Then he left and now, whist we are struggling to maintain his stuff, management is mourning the loss of their superstar.

andrew barton United States |

Comments are closed

My Book

A Pithy Quote for You

"A people that values its privileges above its principles soon loses both."    –  Dwight D. Eisenhower

Amazon Gift Cards

General Disclaimer

The views I express here are entirely my own and not necessarily those of any other rational person or organization.  However, I strongly recommend that you agree with pretty much everything I say because, well, I'm right.  Most of the time. Except when I'm not, in which case, you shouldn't agree with me.

Month List