All good developers write unit tests. The Delphi team is no different. The team has a suite of unit tests for the RTL and a very, very large set of tests for the compiler. The tests for the RTL are, to a large degree, DUnit based. The compiler tests are a bit different, as they date back well before DUnit even existed. They are a set of console applications that write out PASS or FAIL to the standard output depending on whether the test passed or failed. I believe that this suite of tests also tests a lot of the lower level RTL. I know the QA team doesn’t like to test the RTL using a tool that depends heavily on the RTL itself.
So anyway, this suite of tests exists. They get run constantly as part of a continuous integration process at Embarcadero. And the suite gets expanded as time goes by. When I managed the R&D team, I even contributed to the test suite myself.
But given that you can’t ever have enough unit tests, and that like all companies, Embarcadero has limited resources, the test suite doesn’t expand fast enough. (Actually, I suppose no test suite expands fast enough, right?)
So I have a plan: Embarcadero should open source their unit test suite and accept inputs from the community.
Here’s why I think they should do that.
- It will enable the community to expand the suite of tests -- As I noted above, every unit test suite is too small and doesn’t cover enough of the possibilities. Embarcadero can continue to expand their test suite by themselves, or they can harness the power of the community to write tests. The current public code repositories allow for forking and pull requests, and it would be dead simple for me or anyone to fork the project, add tests, and then issue a pull request. This will allow people to write tests for their favorite parts of the RTL, and as will be discussed below, allow them to report bugs by writing tests that should pass but fail. Many hands make for light work.
- It will help build customer confidence in the quality of the Delphi RTL –A public facing, expanding suite of tests with community participation will build confidence in the quality of the product. If people know and see that the RTL and the compiler are bathed in a complete, growing suite of tests, they’ll feel better about the product. If they see how thorough and complete the suite is, they’ll know that the product they use is solid.
- It will help people learn the proper way to use the product – One of the things that unit tests can do is to provide a sort of “documentation” for the things being tested. They can illustrate the proper way to use the RTL code, and show customers the things that can be done that they might not have known about. A nice suite of tests for a new RTL unit or class will help customers see how to use the class and what it is supposed to do.
- It will encourage the team at Embarcadero to write more tests – When Lee Ioccoca took over a failing Chrysler in the late 1970s, one of the first things he did was throw down the gauntlet in front of the company: He would give all buyers a 10 year, 100,000 mile warrantee for the car. That was an impetus to the entire company to improve quality. Publishing the compiler and RTL test suites will have the same effect on the team at Embarcadero. If the tests are public, then the team will feel the pressure (the good kind) to ensure that any new code has tests. The community won’t want to see a new RTL unit without any tests (Will we, guys, right?)
- It will enable people to report bugs in a better and more useful way – The easiest kind of bug to fix is one with a failing unit test. If things got really rolling, customers could write unit tests that reveal bugs, Embarcadero could fix the bugs, and then the test would be a true regression test. That would be awesome. Customers could even use the tests to suggest fixes. Having all the tests would enable them and Embarcadero to be more confident in a suggested fix.
I could think of some objections that they might have. Here are some of those objections, along with reasons why those objections should be ignored.
- “The code isn’t ready for public consumption” – No one cares. Seriously. Anyone in the community that complains about the quality of the tests or lack of coverage can get to work and fix things. No complaining if you aren’t helping. The tests are probably better than most, and even if they are bad, they’ll get fixed.
- “It’s too much work for us” – No it isn’t. “We don’t have the time.” Yes you do. It shouldn’t take more than an hour to gather the tests and post them on BitBucket or GitHub. (Please, not SourceForge. And by the way, whatever you do – for the love of Baby Elvis -- don’t post it in Subversion. Only Git or Mercurial will allow for the community involvement you want and need.). Plus, that small investment will enable an army of developers to submit pull requests with new tests. Those tests can be quickly reviewed and accepted. It will take less time to accept tests than it will to write them. Shoot, you could even enlist trusted community members to help out. I can think of at least one Embarcadero MVP (cough, me, cough) that would be willing to help be a gatekeeper.
- “This is internal, private stuff that we shouldn’t be sharing”. No, this is exactly the kind of stuff you should be sharing. There are no trade secrets here. The tests are useless without the products. There is nothing to be lost by doing this, and much to be gained. This is the kind of thing that a modern, forward-looking company would do. Be a leader. Be brave and bold!
I thought long and hard for reasons why Embarcadero shouldn’t open source their tests, and I couldn’t think of any. Seriously, I can’t think of a downside here.
So how about it, guys?