Archive for the ‘Production’ Category

Parallel vs. Serial Concept Development

More comments on that Playdough report… My copy of Winning at New Products hasn’t arrived yet, though, so nothing specific about Stage-Gate.
While I agree that in an ideal world developers would be able to take multiple projects part of the way through development process and pick the best of them to survive at each phase, I’m not sure it’s practical. While the first few phases (like concept art and vision-level designs) are dirt cheap compared to the later phases (like cranking out thousands of models and a million lines of code), they aren’t free. Smaller companies (like the one I work for) may not even have the right personnel to develop multiple concepts at the same time.

Filtering projects in parallel

Trying to keep track of this number of potential concepts is going to have a fair amount of overhead. In the case of an MMO, you have probably recently launched a game when you’re going through this process, and most of your team is still on the live team for the previous title. Running through exactly the same winnowing process, only in serial instead of parallel.

Filtering projects one after another

In this scenario, you would go through the same brainstorming session, but pick a much smaller number of ideas to develop into full blown game concepts. You might even concept out one at a time. You still run all the ideas and game concepts through exactly the same filters. You still need the same level of discipline to kill off weak ideas, you just don’t have to develop multiple concepts simultaneously.

In terms of actual dollars spent, the cost of these two methods is probably pretty similar. Assuming that your first concept doesn’t make it all the way through to release, you probably still build the same number of total concepts. You have to apply the tests the same number of times. The biggest difference in cost is the smaller overhead from trying to manage multiple concepts at once.

Of course the obvious disadvantage of serial concept development is that you don’t have multiple concepts going through the filters at the same time. When you can see all the concepts at the same time you can just rank them all from best to worst and, assuming they pass whatever requirements you have in your filter, let just the top few through to the next phase. When you are testing concepts one at a time days or weeks apart, you definitely aren’t going to be able to compare a given to the next one on the list, and you may not remember enough of the previous one to compare against THAT concept. You have to rely on absolute measurements of value, which are likely to be awfully subjective.

Rapid Winnowing

The best approach is probably to bite the bullet and develop many concepts in parallel in the early stages, and then cut back to a single project at the point where you dedicate significant staff to the project. The first phase is probably a single designer with a word processor spending a day on a few pages of design doc. The second phase is probably a week’s worth of expansion on that same design, a programmer spending a couple days taking a SWAG at what tech might be required (defining, not building), and a few days of a concept artist’s time to come up with some initial visuals. Not a big price to pay to be able to pick the cream of the crop.

How did the last game concept selection process you went through go? Were competing game ideas allowed to progress beyond the one-page-description level, or did you pretty much pick the next game idea in the initial brainstorming and run with it?

Stage Gates are Wrong for Games

The Playdough report from Project Horseshoe suggested that using the Stage-Gate Model would allow the game industry to improve its hit rate. According to the report, 20% of the games released actually make a profit. This rate is much higher than the 5% achieved by the cheap-to-produce product out of the music and book publishing industries. This rate is only a quarter of the 80% profitability rate that the movie industry manages.

The stage gate system was developed in the 1980s by Dr. Robert G. Cooper, and is detailed in his book on new product development. It is used in a diverse set of industries by “almost 70% of leading U.S. product developers”, according to the Product Development Institute website. It involves taking a new product idea through a well defined set of stages, each of which is followed by a gate that kills off the weak products. The idea is that if you can kill off an unprofitable product earlier in the development process, you will save your company money.

The stages used in the stage gate model are as follows:

  • Scoping — The rough idea is assessed for it’s technical merits and market prospects. This is an inexpensive check that can be performed on a given idea quickly.
  • Build Business Case — This is where all the planning for the product happens. The product is defined and scheduled. An estimate of return on investment is made based on estimated revues and the product plan.
  • Development — The product is developed, operations plans are written, and the staff required for launch are hired and trained. Everything required to launch the product is developed in this stage.
  • Testing and Validation – All of the products of development are evaluated. The development process is evaluated against the plans. The economic fitness of the product is evaluated.
  • Launch — Full commercial launch of the product.

So the question is, how can this model be applied to game development? Here is a pass at how that might work:

  • Scoping — The game idea is evaluated for feasibility. The basics of setting and gameplay are described.
  • Build Business Case — A budget and schedule for the development of the game are developed. The entire game is designed, top to bottom. Concept art is developed for the assets that the game will require. A technical plan for the game is assembled, including which middle-ware to use, what platforms to launch on, etc.
  • Development — The programmers, artists, and level designers go to work building the game. They build the entire game top to bottom. The game is 100% playable by the end of this stage. The game box is designed, the marketing campaign is planned, and the customer support people are trained. If appropriate operations centers are staffed and trained. The server hardware required for testing is installed and configured.
  • Testing and Validation – The game is put through its paces by the QA department and focus tested like crazy by the marketing department. After making whatever minor adjustments are required as an outcome of the testing, the game moves to the next stage.
  • Launch — The game launches.

Here’s the problem: This won’t work. The Stage-Gate process is the classic waterfall model zoomed out to look at product development from the business perspective. The waterfall model looks something like this:

Waterfall Model

This model works really well for a defined process, which is any process that you can define well enough to make it repeatable. This is highly desirable for manufacturing processes, and is probably applicable to almost all of the Fortune 500 companies using the Stage-Gate model. Unfortunately this is not the situation with software development in general, and may be even less true of games.

For game development to be a defined process, we would need to have the same goals for a game that we had for the game immediately before it. We would have to staff the teams exactly the same way, with at least the same levels of experience and training if not exactly the same people. We would have to want the same game to come out the other end of the process.

That’s not what we want out of game development. What we want are fresh games, and fresh ideas. We almost never start a game with an equivalent staff to the game we developed before, and given how hard it is to find good people, that’s not realistic anyway. What we need is an empirical process; one that lets us take measurements along the way and apply them to our ongoing development effort. An example of an empirical process is the Iterative or Evolutionary model:
Iterative model

This is exactly the model being pushed by the big-A Agile community. They go to extreme measures to drive the cost of this iterative cycle as low as possible. Some of them even go so far as to flip the whole model and perform the tests before they actually do the development.

While I’m not really an adherent of the Agile school of thought it does seem pretty clear to me that the Waterfall method is broken for our industry. How many times have you seen a feature take many times its original estimate because once the work was underway it was clear that it was actually more difficult that it appeared from the outside? How many times have you seen part of a design be thrown out of changed substantially based on how it actually played? These happen all the time and aren’t the fault of the programmer who made the estimate or the designer whose work is being revamped. Each game explores new territory using the experience of its developers as its foundation.

So if Stage-Gate isn’t going to help us, what can? Well that’s a big topic that belongs in a post all its own. What do you think? Are there any business-level models from other industries that actually can help ours? How can we develop through hundreds or thousands of iterative loops and still be able to budget and schedule in any kind of reasonable way? I’m very interested to hear your thoughts on the subject.

Ten Ways to Improve Developer Efficiency

As I see it, improving developer efficiency is the most important thing you can do to increase your game’s chance of being a success. Game development is a very iterative process, where each iteration results in a game that is better than it was in the previous iteration. Every second that you can pull out of the iteration time of each of the members of your team is going to buy you many additional iteration cycles over the course of the project and have a big impact.

Here are ten things you can do to improve productivity for your developers:

  1. Never, ever require anybody to wait for a build to see their changes working in-game – I can scarcely believe it when I hear it, but over and over I hear tales of artists and mission designers waiting for a build to test their new art or missions in the game. Even if your builds happen every day, this is still ridiculous. Iteration times should be measured in seconds or minutes, not hours or days.
  2. Drive as much of your game as you possibly can from data files – Data-driven development has plenty of its own virtues (not the least of which is that it helps a lot with number 1.) The biggest thing it does is remove the need to talk to a programmer when you need to make a change.
  3. Allow everyone in the company to run their own copy of the server cluster – This will vary in difficulty based on your server architecture, but it’s crucial. If there are any data files loaded by the server, (and your game is data-driven so of course there are) many people will need to be able to restart a cluster when they make a change. And as part of making it easy to bring up a new cluster for Random Artist A, you’ve also made the job easier for Random Operations Guy K.
  4. Use text files for your data files – Text files can be read by a human (or at least a human of the “programmer” sub-species.) They can also go into source control tools and show diffs. Depending on the format, they can probably also be merged in source control tools. Most game data (textures, meshes, and animations being huge exceptions) is small enough to begin with that the extra bloat from a text-based file format isn’t that big a deal. If you’re worried about the performance of text files, feel free to have a binary version that you compile from the text files as part of the build process.
  5. Run builds often – I was on the Middle-Earth project at Sierra for 9 months (up until it was cancelled) and I think we ran exactly two builds. On Rails Across America we ran builds once a week. On PotBS we run them (at least) once a day. If builds are running constantly, and build breakers are hassled appropriately, you can never get very far from having a playable game. At least not as long as you also do #6. When your build finishes, it should notify several people automatically and let them either fix it (if it failed) or start the manual part #6.
  6. Run Build Verification Tests daily – BVTs are intended to tell you if a given build is worth the electrons it’s written on. If you run them every day you will have a much greater chance to detect integration problems and build breakers as close to the source as possible and get them fixed just as quickly. There is likely to be some kind of human component to these tests, but much of them can be automated. We automatically test to make sure that every environment can be loaded, and every ship type can be created at the end of every official build. These tests fail in some significant way about once every three weeks and usually the fix is made and a new build is underway by noon the next day. (We run our builds at night, so nobody is around when the tests fail. Fixing the build is )
  7. Write unit tests religiously – I think the central tenet of Test Driven Development (that you should write your unit tests before the code and then run them to watch them fail) is a bunch of hooey. Testing your code with unit tests, however, is much faster in many cases than testing it by bringing up the server cluster and client and testing it in-system. And once you have them in the build you have a perfect way to test that your basic assumptions about the code aren’t being violated when you make future code changes.
  8. Review all checkins – Programmers will get the most benefit from this one, but other developers will also benefit. Thanks to #4 you have all your source data in text files, so a quick diff will show you what the developer changed. If the changes don’t look quite right the developer can fix it before it breaks the build. The reviewer is also granted some knowledge of the checkin so they are better able to fill in for the developer in a pinch. Everyone on the team should act as reviewer for other developers so that leads don’t become a bottleneck. This is also a great place to enforce coding standards and naming conventions.
  9. Keep your startup time as low as possible – Normally your whole-system optimization happens toward the end of the project. If you can spend some time optimizing startup times as you go, you will save many hours of developer’s time over the course of the project. Don’t forget to pay attention to the startup times in your tools and servers too. Chances are that developers will be launching those just as often as they launch the client.
  10. Prevent developers from having to restart – For programmers, this might mean using a scripting language and setting things up so you can reload scripts on the fly. If programmers can shut down pieces of the server cluster, recompile them, and then relaunch them without ever closing the client, they can get some benefit there too. For other developers, this means being able to re-read data and configuration files instead of requiring a restart of the client, server, or tool in order to see your change. Anything you can do to make this reload of data faster is a big help, possibly including doing it automatically by scanning file times.

These are all ways that you can improve developer efficiency (and therefore productivity.) There are only ten on this list, though, and I’m sure there are many others out there. What are some ways you have improved the efficiency of your teams?