On the other hand, I also strongly believe that the industry’s dependence on weekly or even daily downtimes is bad for us and we need to fix it. Zero downtime is a legitimate goal. Linden Labs can do updates to Second Life without taking everything down for hours; why can’t the rest of us? Sounds like Guild Wars is doing something really similar to what I’ve mentally sketched out as a process, which is neat.

I gotta blog about both these things at some point.

The speed of light starts becoming a serious barrier when you have as much content to deploy as a typical MMO; IMVU gets off a little easy since our deployments are almost always in the 5-50MB range, so we don’t run into bandwidth issues when getting stuff onto the servers.

I’ll try and share what I remember about GW’s deployment process: (Disclaimer: I spent most of my time working on the content pipeline and designing game content, so I’m not a domain expert here. Some of this is probably completely wrong.)

Internal ‘development’ builds took between 15 and 60 minutes, depending on whether we were building large content changes or just code changes. After we had a finished dev build, we could test it on local machines and on a small development cluster that our QA and alpha test groups had access to. From development we did a merge over to our ‘staging’ (production rev) branch, and did another build in about the same amount of time. Once that was done, something resembling a one-button deploy moved the content and game code from staging to our production machines, which took under an hour – most of the time being spent actually sending the bits to the datacenter(s), so the time varied based on how much had changed.
Once the new code was on all the servers in the datacenters, we flipped a switch that notified the live servers that any new ‘instances’ should be started using the latest version of the game code. At this point, any users running older builds of the client got told that they needed to update, and attempts to load into a new instance would be rejected until they updated.
To summarize how the live update worked on the server, I believe we atomically loaded the new game content onto the servers, and then loaded the newest version of the game code into the server processes alongside the older version, because each build was a single loadable DLL. That let us keep old instances alive alongside new instances on the same hardware, and isolated us from some of the hairy problems you’d get from multiple client versions running in the same world.
All game content was stored using a versioned filesystem of some sort on both the servers and the clients, so the update process was fairly efficient and incremental, with our fileservers doing the work of figuring out where revision foo of file bar happened to live and sending it off to the client.

As for the client side, here’s what I remember:
All content (with a few exceptions) was streamed down to the client ‘as required’, based on a set of rules about when content was needed:
There was one set of content that was essentially “required” to run the game. If all you had was the game client, it would automatically figure out that it needed the required content and pull it down for you. After this, you got from the loader/updater screen into the login screen. Things like UI textures, fonts, etc were in here, since the client itself used these resources.
The next set of content was essentially ‘universal’ content. This stuff wasn’t necessary to start up the client, but was basically required for loading into any instance where players could be running around – I think it contained things like the basic human skeletons, common textures, etc. This would be streamed on demand at the loading screen for a given instance once you logged in or changed zones.
Finally, there was a set of content needed for the actual location you were in. The game code had a lot of really elaborate scaffolding that made it possible for us to statically determine what assets were needed by a given instance. From this, we built a ‘manifest’ that listed all the assets the game client could *possibly* need in that instance, and the client loaded them all before leaving the loading screen. This was particularly painful in some respects (for example, if you had a monster that spawned 5% of the time, 100% of your players would have to download the assets for it because manifests couldn’t safely be dynamic in such a manner), but it really did work well when executed correctly. The successes and failures in this particular area depended as much on careful design as they did on clever technology.
There was also some magic for things like skill icons and character textures, where we only required the download of low-res subsets of the data, and we would stream in the high-res versions on demand as soon as they were actually loaded. I’m not certain exactly where we used it, but I know that you’d sometimes see the textures for a person’s armor ‘pop’ from low-res to high-res as we streamed them in over the network if you didn’t already have them locally. It was hard to tell exactly when this happened, though, since we also did lots of asynchronous loading from disk that tended to look like network streaming if you were on a good connection. This primarily helped for the new user case, since it meant they could download Gw.exe off our website in less than a minute and have enough content loaded to enter a town within an hour or less.

The manifest system tied in to the build process pretty directly, and I suspect it was responsible for the short build times. If a build only took 15 minutes, that usually meant that none of the content had to be updated – and since everything was built around incremental updates, that meant we only had to deploy binaries to the servers and users’ machines, and all the build server needed to do was crunch through a bunch of source code using burly hardware. In degenerate cases where a bunch of content changed, there was a lot more time involved, but it still scaled nicely. We also had the ability to do incremental builds of source code, but we didn’t use it for deployment in fear of linker issues (we used it on local development boxes, and while it was much faster, the linker issues were real.)

Based on my experience with GW, if I wanted to get POTBS’s build times down (as a contrived example), I’d try and kill the ‘server upgrade’ stage first by making it asynchronous in a manner like what I described above. Improving the speed of pushing out a build is valuable but not particularly easy to do, so I don’t expect there are many wins to be made there. You could cut down on the length of smoke tests, but I suspect less than 3 hours is irrational – I wouldn’t be surprised if at least that much time went into testing GW deployments and I just didn’t see it happen. Packing the build seems like it shouldn’t take an hour, but the elaborate manifest/versioned filesystem setup we had meant that there was no ‘packing’ process, so I don’t quite know what went into yours. I suspect the overhead of having to collaborate with SOE probably factors in here too; I saw very little NCSoft red tape involved in smaller deployments.

I may regret posting this comment. (:

I would love to hear more about how this process worked on Guild Wars. It seems like the ability to download content from within the client and to operate with a subset of the data would be a big help and GW could do both of those things.

Continous deployment has a fundamental impact on how you think about game deployment and patching, compared to the old-fashioned deployment approach where you focus on building gold masters, but it totally pays off.

My experience with semi-continous deployment on Guild Wars definitely left me hungering for more, and the horror stories about weekly builds that I heard from some of my more experienced coworkers made me wonder why more companies hadn’t tried to get their build times down – it’s really great to hear that other people are trying as hard as the ANet crew were.

There were cases where being able to deploy more than once a day meant that we could tackle an emergency in the span of a day or weekend instead of having to crunch away for weeks, by simply putting together a potential solution, rolling it out, and seeing if the problem went away, instead of having to spend days or weeks waiting on builds and exhaustive QA.

On the other hand, there’s still a lot more ground to be gained, so I hope people like you keep working hard on improvements in this area – it’s amazing how much better the quality of life can be for hardworking artists, designers and engineers when the consequences of a failed build or bad deployment are measured in hours of stress instead of days or weeks. In an industry that’s somewhat famous for bad work conditions and endless crunch time, I think that’s a really big deal.

There was much politicking required to let us use our own written patchservers, as opposed to the “approved” ones.

1) We run the patchservers, and our patchservers handle incremental data uploads, so the build to patcher push is much faster for most builds. It also happens at build time automatically, so when we push an incremental build most of the files are already up on the patch servers.
2) Our full builds max out at 30 minutes or so. Data-packing can take a good bit longer, but that’s incremental so doesn’t happen most of the time.
3) Server upgrading uses an identical process to client patching (grabbing a different subset of files off the patchserver). Also, this means end users can pre-patch at the same time the servers are getting ready in theory.

I’ve run a complete deploy process of a prototype project (ie, much less data then PotBS but just as much code, so the patching is faster than it would be for a real game) in about 40 minutes from a completely new build to externally accessible servers.