Tools Team Test – 5 steps to better tools

This post originally appeared on http://altdevblogaday.com if you have comments that would be the best place for them.

Several years ago Joel Spolsky wrote about The Joel Test. A simple yes/no test you can use to measure the quality of a software team.

It looked like this:

  1. Do you use source control?
  2. Can you make a build in one step?
  3. Do you make daily builds?
  4. Do you have a bug database?
  5. Do you fix bugs before writing new code?
  6. Do you have an up-to-date schedule?
  7. Do you have a spec?
  8. Do programmers have quiet working conditions?
  9. Do you use the best tools money can buy?
  10. Do you have testers?
  11. Do new candidates write code during their interview?
  12. Do you do hallway usability testing?

If you haven’t tried it, wait until you’ve finished reading this post and then run through it for your team. Ideally you should be scoring 9+.
Inspired by this test, I started thinking about simple yes/no questions that could be applied specifically to the tools department on a game team. I couldn’t get to 12 but I came up with 5 questions that I think a good tools team will answer yes to.

1. Do you have dedicated tools programmers?

This was probably the first thing I thought of and is hopefully redundant if you are answering questions about your ‘tools team’. However, it is not universally the case that a game team will have programmers who’s only job it is to design, write and maintain the tools that the rest of the team use.

The problem with overloading your other programmers with writing the tools for their systems is that they don’t necessarily have the skill set to do it and usually their heart isn’t in it.
AI programmers want to write AI, not UI.
By taking the tools implementation away from the system’s author you get a second technical mind looking that the interface into that system. The tools programmer can be the one to worry about whether it is better to expose a range of 0-65535 or simply Low/Medium/High.
Graphics programmers think that everyone knows how many photons it takes to generate a good light-map.
The mind of a tools programmer is a wonderful thing, thinking about things like work-flow and iteration speeds. These are not the same issues that plague the resident graphics guru.
Audio programmers shouldn’t be writing code to render orientated bounding boxes.
If you force a programmer who’s interest is in one area to write and support the tools needed to use their systems one of the areas will suffer, and it will likely be the tools.

2. Do you deliver your tools atomically?

Often you will find that support queries to the tools team fall into one of these two scenarios:

Artist: The tools are broken.
Tools Programmer: Have you got latest from perforce/svn/git/cvs/the network drive?
Artist: No.
Get latest
Tools Programmer: Fixed
Artist: I love you.

Or like this:

Artist: The tools are broken.
Tools Programmer: Have you got latest from perforce/svn/git/cvs/the network drive?
Artist: No.
Get latest
Tools Programmer: Ah, still broken… I’ll get back to you.
Artist: I hate you.

To fix this, you must remember one thing: Source Control is not a delivery mechanism. Even if you have labeled/tagged a release there is nothing to ensure that the end-user has synced all the files necessary. ‘Did you forget the dev/bin directory, what about etc/other/important? Oh, you definitely need that one’.

These queries can be a significant chunk of support requests and you can make them go away very simply. Deliver tools updates atomically. By atomically I generally mean via an installer with a version number. This has the added bonus that each release has a unique version number which means bugs can be tracked more easily.

This opens up another question which is: Should you force a tools update on users? On balance I think the answer to this is yes, you should. It is likely that your users will need the latest tools to create data compatible with the latest game build and besides, if Google can do this with an browser shipped to millions of people I’m sure we can manage with one small toolset. It does, however, probably require that you have nailed the next question with a yes.

3. Do you test tools releases?

Your users must trust your tools. If they don’t trust your tools they won’t use them. They will hack at xml files in notepad and bad mouth you behind your back and to your face.

The level of trust from your users can wax and wane and it can be earned and lost. It will start out high, your users are a nice bunch, you’ve told them you have a tool that will make their lives easier. Why shouldn’t they believe you? However, if your tool breaks, and they lose data, and then they have to stay late to re-do it the night before a milestone you will lose that trust. Quickly. It’s a lot harder to earn it back than you might think. Once that trust is gone the users will hack the xml files forever because ‘your tool just crashes all the time’.

If you can somehow make sure that your tool doesn’t crash, and doesn’t lose vital data, then your users will continue to trust the tools and they will continue to use them. The answer is testing your tools before you release them.

Testing software is hard and it is time-consuming. On PlayStation Home, testing the full tools suite would be several days work for several testers. Which is why you need to rely on automated testing as much as you can. If you haven’t heard of them, look up unit testing, functional testing, Test Driven Development, Continuous Integration and Build Machines. Pick and choose which solutions suit your needs but do use this stuff to bring the testing time down, and your trust levels up.

4. Can users submit bugs and feedback?

I’ll keep this one short. You have a bug database for your game, right? Then get one for your tools.

5. Can your tools be used without a manual?

Your tools probably don’t have a manual, and even if they did, the vast majority of users won’t read it before coming to you for help.

Even though you’ve added an awesome new feature the users may not know what it does, and they may not know when they should or shouldn’t be using it.

Some of things I’ve tried in the past on teams are:

  • Seminar-style lectures – where the tools team go over new features and take questions from the team
  • Documentation – i.e. a manual
  • Release Notes – a list of new features with descriptions distributed with each new version

These were all relatively successful, but they can require a lot of work, and generally suffer from the ‘no one reads them anyway’ problem.

Solutions that I think are more effective:

  • Inline help – tool-tips, pop-ups, wizards. These are harder to ignore and give help at the time of need.
  • A No-Mistake workflow – Don’t allow invalid data or actions to occur. If you do, make sure that you…
  • Validate early – So users are told about errors in data before spending 12 hours rendering a scene.

So that’s my simple tools team test, how many does your team score?

  1. Do you have dedicated tools programmers?
  2. Do you deliver your tools atomically?
  3. Do you test tools releases?
  4. Can users submit bugs and feedback?
  5. Can your tools be used without a manual?

I’d love to include more questions, so which ones would you add?

Comments are closed.