A fair amount of Turbine’s non-game traffic is HTTP. It turned out to be a big win for both the obvious reasons and some unexpected ones. As Kevin notes, getting the benefit of load balancing technology is great, not just for balancing load. F5′s BigIP iRule technology turned out to be very handy for quickly developing all sorts of useful tools for handling traffic loads that would have been harder to build into the servers.

From a community point of view it makes alot of sense and I imagine would take alot of strain off of the game servers as the game servers dealing with combat calculations, AI and all the “Hard” stuff won’t have to be concerned with a guild leader promoting members. It would even allow you to have multiple game servers that all feed into the same market servers or read out of the same Character Database.

We could shave off about 5 minutes with better data organization: the first step of building the pack files is to copy all the unpacked client files to a temporary location to ensure that no server-only files get to clients. We sacrifice another 5 minutes by copying the current live pack files down, in order to start the incremental packing from a known point and give a minimal patch size. Analysis showed that a purely incremental approach bloated the patch size in certain circumstances, such as when multiple modifications to the same asset altered its size significantly. The actual packing operation takes 10 or 20 minutes depending on the extent of changes. Another 5 minutes is spent copying the full distribution to the central share.

During the remaining hour, the (SOE-supplied) delta program generates patches, reports and sums their sizes, then throws them away. This is one of the organizational handoff inefficiencies Joe spoke of in the previous post. Obviously we could eliminate this hour or one of the patch deployment hours if we had control of the entire deployment pipeline.

We keep the hour because a) by and large it happens during the nightly build and nobody cares whether the build finishes at 11pm or midnight, and b) it is extremely useful to know when the patch size jumps up significantly. We’ve caught quite a few art bugs this way.

With UO, there was no database backend; everything persistent was stored in flat files. We also had a hodgepodge of text and binary server-side data files. When I first started working on UO we’d actually sync CVS to the servers and build it on each and every server machine. Eventually the deployment process became more sane and we would push pre-built binaries. Since the flat-files were generally opaque to everything but the game processes, the update would never touch them. The slowest part about a publish was the servers coming up and reading all of the persistent world data (since everything in the world was persistent, not just characters).

Interestingly, with UO we would push a new client live a week or two prior to a new server, so the client had to support two network protocol versions. Unfortunately the ‘old version’ support was almost never removed.

When the client version changed, clients would of course have to patch, but client patches on UO were often fairly tiny by today’s standards.

—-~~~~—-

EverQuest II is quite different. We use an Oracle DB to store our persistent data (which centralizes around characters) and we keep our server-side data in packed binary files. Our patcher system allows new client and server data to be sync’d to their respective servers while the live game stays up and running. We almost never do DB updates that require large amounts of downtime. While our character data is a mixture of columns and blobs, the game is authoritative on versioning. We could potentially have very old character versions stored in the DB as they’re not upgraded until loaded. In practice this has rarely affected us negatively but has allowed us to perform very fast updates.

When syncing is done, servers are taken down and a switch is flipped which really causes a directory rename on the servers, followed by a startup. The longest downtime portion of our patches is QA checking out the servers before they’re unlocked.

In the case of most hotfixes, a client publish is more or less “optional”. By this I mean that the game will connect and run if you don’t patch the client because the network protocol version hasn’t changed. However, you’ll still get booted off when the servers go down.

While the best user experience might be trying to maintain old-version uptime even after a patch has synced, it sounds like a logistical nightmare considering that players are going to have to disconnect and patch at some point anyways.

I’m excited to see how the client streaming system for Free Realms will play out. Unfortunately, EQII’s client-side data was not designed for streaming and there are many, many interdependencies that any sort of streaming system would have to be aware of and anticipate.

Always fun to chat about current and upcoming MMO tech. Looking forward to LOGIN 2009!

Yep, Dungeon Runners did this, including the incremental crawler. The crawler process was implemented so early in the project, I actually had to revisit it around the beginning of last year: it was still trying to load EVERY CHARACTER into memory before doing any migration (which worked early on, and continued to work in dev, but started choking in production). I implemented a moving window of loaded characters, so a few thousand (or whatever) characters were in memory at any time, with new ones being loaded as completed migrations were saved to the DB.

That kind of system is a LOT harder when you go for a regular relational representation in the database, something I’m still kind of contemplating for my current project. New (nullable or default value) columns aren’t too bad, but you better be sure to change the meaning of an existing column only under extreme duress.

For server binary upgrade, NCsoft Operations had a classic (but workable) system: the root of the server install had ‘old,’ ‘live,’ and ‘new’ subdirectories. Install everything in ‘new’ while the server is up, take down the server and do database upgrades while deleting ‘old,’ renaming ‘live’ to ‘old’ and ‘new’ to ‘live’. It wasn’t as snazzy as versioned DLLs, but it avoided some of the craziness of DLLs too. This essentially removes server software upgrades from the downtime equation.

Something like PotBS’ ServerDirectory is probably better, and also more webbish: connect to a server that’s still at the version your client is at, next time the client changes instance/zone. You could probably go a step further, and mark servers as needing to be upgraded one at a time, so that the Server Directory won’t send new clients to that server and when everyone is gone from it, you can upgrade it and have it reconnect with its new version.

Dungeon Runners also had package files with incremental updates: that was actually implemented after the game was live, when we decided the slow start time of the client was unacceptable. I think the PlayNC Launcher also had some concept of patching existing files, so what we uploaded to the patch servers (and what clients downloaded) was essentially a binary diff too.

I think for Dungeon Runners the real causes for downtime were bugs, database upgrades (in some cases, particularly because of the character data and other webbish initiatives), and Windows/hardware updates. Aside from that, of course, there’s the span of time between “build finished,” “build signed off,” and “build published.” We didn’t use the incremental patches for server packages, sign off generally took days (but there was no automated testing)… we solved a lot of the technical problems without actually making progress anywhere but server downtime.

Also, somehow I missed your previous blog post on the subject… pretty cool that someone formerly at ArenaNet commented on it, their system engendered a combination of “WTF?” and “cool!” around NC Austin.