1. Star Trek Online will be the only significant MMO launch in 2010. It will do well enough to make Atari and Cryptic plenty of money, but will not do nearly as well as World of Warcraft, so many people will consider it a failure. (Those people are dumb.)
2. At least three of the major unreleased MMO projects will be cancelled. I have a guess about which two are most likely, but I’ll keep that to myself.  Qualifying projects include:
   • Guild Wars 2
   • Whatever Carbine is working on
   • That console MMO Turbine hasn’t said much about
   • Whatever Trion is working on
   • The Sci-Fi channel tie-in MMO that Trion has said they’re working on
   • Whatever Zenimax is working on that may or may not be Fallout
   • Whatever 38 Studios is working on
   • The Agency
   • Whatever Gazillion’s Gargantuan studio is working on
   • Star Wars: The Old Republic
   • The second MMO that CCP is working on down in Atlanta where all those White Wolf people are.  Hmm. What could it be?
   • Whatever Red 5 is working on
   • DC Online
   • That other MMO I know NCSoft is working on that is completely under the radar
   • Whatever Slipgate Ironworks is working on
   • The second MMO Blizzard has in the works
   • APB
   • Jumpgate: Evolution
3. Project Natal and the Playstation Motion Controller will both come out.  Natal will do fairly well. Both controllers will allow some new kinds of games, but we won’t see any compelling examples of those games until 2011.
4. Unemployment will peak and then start to fall.
5. The compass+GPS augmented reality products will begin to shift to general location-awareness and away from their Augmented Reality roots. They will de-emphasize magic lens and start to emphasize aggregation of nearby content.
6. No consumer-level see-through displays will come out in 2010. Significant progress toward them will be made, but nothing will be released.
7. Neither Google nor Apple will release any kind of AR-focused hardware
8. The use of “wave it in front of your webcam” type AR in advertising will peak with an AR-enhanced ad in the Superbowl.  The backlash will begin. By the end of the year the advertising world will have moved on.
9. Apple will release its tablet and a new iPhone (faster and more storage) but won’t release anything that is specifically an AR product.
10. Apple will address the pain caused by its app-store approval process, at least in part. I have no idea what their specific solution will be, but they aren’t going to let their developer community grow to hate them.
11. Android will continue to pick up steam. By the end of the year Android will boast 50,000 applications.
12. People will spend the entire year trying to find something really useful to do with Google Wave. They won’t succeed in 2010.
13. Google will make Wave interoperate with email. This will make it useful as an email client if nothing else.

Ok, that’s the last of this kind of post for at least a year.  If only I could get back to posting regular stuff again.

Please note that just because something is on this list does not mean that it’s something I want to happen, only that it’s something I think will happen. Anything that’s missing from this list is probably just something I didn’t think of.

I would love to hear your thoughts on any or all of these.  Please comment below.

General Technology Trends:

1. Moore’s Law will continue to operate for the entire decade. That means a given form-factor of computing device will be approximately 100x the power of the same form-factor today.
2. Mobile computing will dominate. Everyone who owns a laptop or desktop today will have a mobile device that is about 10x the power of their current computer.  We may still call these “phones”, but placing voice calls will only be one tiny part of what they do. This device will replace most users’ desktop and laptop computers.
3. Digital Distribution will be king. Only a tiny fraction of the media that’s currently consumed digitally (TV, movies, music, and software) will be purchased on a hunk of plastic. Both the subscription model (aka Rhapsody or cable television) or the purchase model (aka iTunes or DVDs) will have at least 20% market share, but one of those two models will be gradually taking over. Advertising supported media will be just as big of a deal as it now, but the user will have much more control over how they consume that media (think Hulu rather than broadcast television.) Books are on the same trajectory, but in 2020 the majority of books will still be sold on dead trees.
4. Speach recognition will gain a lot of ground as the primary way we enter text into a computer. Offices are one place where this trend won’t have advanced very far mostly because of the noise involved.

Game Industry Trends:

1. Total revenues from video games of all kinds (including mobile and social games) will exceed revenue from movies and television (independantly, not added together.) Games will finally learn to exploit merchandising and secondary markets as vigorously as movies do.
2. In 2020 no one will be selling a dedicated gaming console. All computing devices in production in ten years will be about consuming other kinds of media just as much as they are about playing games.
3. Desktop PC gaming will be all but dead, with the majority of triple-A games coming out for multi-media consoles or mobile devices.
4. Gaming that involves exercise will be the primary way that the majority of people get their exercise.
5. Location-aware games will be common.

Augmented reality:

1. A growing minority of people in the developed world will wear heads up displays almost all the time. These displays will be capable of information overlays, but will mostly be about contextual information that is not overlaid on the world. These products will be on the verge of hitting the mainstream, but won’t quite be mainstream yet.
2. Development of these displays will be by small companies (perhaps companies that are around now) but those companies will be acquired by massive consumer electronics multinationals before wearable displays hit the mainstream.
3. Recognition of people and text in images (and video) will be nearly perfect, at least in reasonable lighting conditions.
4. Gestural interfaces will be commonplace. Many hard-core computer users will be sad at how clumsy they are compared to keyboard and mouse.

The fate of specific companies:

1. Google will be huge and influential. Their influence will likely peak in the 2010s, but it will difficult to see that from the ground. Google will have had some sort of anti-monopoly action taken against them.
2. Microsoft will fail to transition to the new mobile-centric world and will be in decline. They will still be a very powerful multi-billion-dollar company, but will not own the end-user to nearly the extent they do now.
3. A company that exists today will be the dominant social network.  that could be Facebook, Twitter, or YouTube, but it probably won’t be MySpace.
4. Apple will be huge and influential. They won’t ever be as dominant as Microsoft was in the 90s, but they will be very successful. Steve Jobs will still be running the company.

US Politics:

1. Gay marriage will be legal in most states.
2. Marijuana use will be legal in California and a few other states.
3. We won’t have elected a woman president. (My wife came up with this one, but I agree with her.)
4. The problems of illegal immigration will not be solved.
5. The problems of providing health-care to everyone that needs it will not be solved.
6. Privacy in an age of always-on location-aware devices will be a huge topic of debate.
7. Silicon Valley will remain the world’s premier startup region.
8. The US will still have troops in both Iraq and Afganistan. These will be like the troops we still have in Germany and South Korea, and will not be in combat often, if ever.

International Politics:

1. Carbon emissions will be at approximately their peak in 2020.
2. Oil production will also be peaking around 2020.
3. Most other countries will be ahead of the US in terms of switching to renewable energy.
4. Most of the rest of the world will have consumer-friendly privacy regulations in place. Those countries will scratch their heads at the debate raging in the US.

Things that will not happen:

1. We will not have flying cars, jet-packs, or most of the other things promised by Sci-Fi in the 50s.
2. There will not be peace in the middle east.
3. Africa will still be the poorest continent.
4. Brain-computer interfaces will still not work very well. No one will be uploading themselves into a computer.
5. We won’t have a human equivalent AI.
6. We won’t know how to reliably unfreeze people.
7. World War Three won’t have happened.

1. Correct. Champions and Free Realms both launched. Those were both smaller than the big launches of 2008 (Age of Conan and Warhammer), and I expect they will end up being smaller than next year’s Star Trek Online launch.
2. Wrong. The console version of Champions Online never came out. I hear it’s because of difficulties working out revenue sharing and certification details with Microsoft, not because of any technical challenge. We’ll see if it actually comes out in 2010.
3. Wrong. Has there been a single large MMO announced in 2009?  Carbine, Trion, Red 5, 38 Studios, and Zenimax certainly haven’t announced anything.
4. Wrong. Nope. Nobody bought Turbine. Atari did buy Cryptic, which is almost the same thing.  Maybe I should be more vague in the future.
5. Correct?  Upper Deck University launched, but hasn’t had much traction. Are there other kid’s MMOs that launched in 2009?  Wizard 101 from King’s Isle launched at the end of 2008 and had an excellent year in 2009. What does that mean for my prediction? No idea.
6. Correct, sort of. Both Microsoft and Sony decided to launch gestural control devices instead of whole new consoles. They are trying to get some more mileage out of their investment in the current generation. Project Natal has had a lot of buzz, but no dev kits have shipped as far as I know. The PS3 motion controller has been a little more under the radar.  Both of these devices are expected to ship in 2010.
7. Hard to say. Maybe we’ve hit bottom, maybe we haven’t.  Ask me in two years.
8. Wrong. Oh so astoundingly wrong.  Between Layar and Junaio’s launches, tons of marketing campaigns, and lots of concept and research videos, AR has had a year full of hype. Those see-through displays I was so excited about in January didn’t happen, but on the smartphone/magic lens front there has been quit a bit happening.
9. Hard to say. Did Microsoft start an MMO in 2009?  They didn’t announce anything.  Wonder if they’ll announce it before they cancel it.
10. Wrong. At least I haven’t heard of one.  Lots of indy games made lots of noise in 2009, but none of those were XNA games.
11. Correct?  iLike’s firesale to Myspace is one.  Metaplace.com going away is arguably another, though Metaplace lives on as a company, at least for a while. Geocities counts too, I guess.  The trouble is the “have heard of” qualifier. Most web startups are completely unknown outside of Silicon Valley
12. Wrong. There has been no SOE reorganization fallout visible from the outside. The Agency lost the head of its studio, but the game seems to live on.  DC universe is going fairly well from what I hear.
13. Correct.  Mythic merged a bunch of servers.  Then they laid off a bunch of people. Then EA kicked out Mark Jacobs. Then EA laid off more people.  It has been a tough year over there. I hope everyone lands on their feet.

So for the year, that’s five correct predictions, six incorrect predictions, and two unmeasurable predictions.  Not such a good year to see the future.

The biggest thing I missed is the way mobile is heating up.  The App Store is a couple years old at this point, but it really exploded in 2009.

What about you?  Did 2009 turn out like you expected?

The argument that digital distribution is a disruptive technology is an easy one to make. It disintermediates retailers, distributors, and manufacturers, by letting publishers and developers by through a single layer of middlemen (in the form of Steam, Impulse, or Direct2Drive) or even directly to their customers. Digital distribution also removes the sharp shelf-space restrictions that keep a game’s shelf-life artifically low and allow game sales to enjoy a long tail. Numbers are hard to find, but digital distribution has also been accelerating in the past few years. 

Part of the Innovator’s Dilemma model predicts that established players will be able to enjoy significant revenue growth by retreating from low-margin and low-volume markets as those fall to the invading technology. GameStop’s last two quarters are the best they have ever been. However, one part of GameStop’s business has been in serious decline over the past several years, and that is PC games. That is partly due to an overall decline in PC games in general, but it is also to the fact that PC games can’t be repurchased and sold as used like heavily DRMed console games can. This resale of games accounted for 23% of the company’s revenue last year, and that number is increasing. According to the Wall Street Journal, used game sales account for 42% of GameStop’s gross profits.

I suspect that the other major reason for GameStop’s withdrawal from PC games is that digital downloads have made their strongest inroads on the PC. Steam is selling tons of AAA games, and casual and indie games are sold primarily online at this point. The company is not mounting much of a defense in the market where its competition is strongest. They will likely lose the console market the same way, since I would be shocked if the next generation of consoles (Xbox 720, Playstation 4, and Nintendo Puu) didn’t include large hard drives and the ability to download full sized titles as easily as on a PC.

GameStop is making the choices they are because that is what is demanded of them by their customers and investors.  As a multi-billion dollar public company the tiny digital sales available five years ago would have just been a distraction from meeting their growth goals. That allowed Valve and others to get experience and partnerships that give them a huge head start in the the digital distribution space. Just in the past 18 months Steam has picked up titles from EA, THQ, and Sony Online Entertainment. These major publishers join Activision and Take-Two that joined the service a few years ago. In fact, GameStop was reportedly so upset at how much THQ was favoring Steam that they temporarily refused to offer pre-orders of Dawn of War II. (Or at least that was the rumor.) All the while, the biggest complaint that most of GameStop’s customers have is that it doesn’t give them enough for a used game. By and large their customers are quite happy with the company.

Broadband penetration in the US has increased from 4.4% to 50% in the past 9 years. At this point the extensive selection available for download is primed to combine with the relatively new viability of large downloads and drive GameStop completely out of the PC game business. I believe the same thing is likely to happen for console games within the first two years of the next generation of consoles. Within ten years the dedicated video game store will have gone the way of the travel agent, cable driven earthmover, and walled garden internet service provider. 

Or at least that’s now it seems to me as a retail outsider and recent convert of The Innovator’s Dilemma. What do you think?

For me personally, this was probably one of the most conflicting parts of working on Titan.  Don’t get me wrong — I’d wanted to work on an MMO for as long as long as I’ve been in games, and this was the dream game of a lifetime.  But while there were a few of us that had played MMO’s before WoW, by far and large, as the team grew, most of the people on the team had never played a single MMO before WoW.  This led to a dilemma that the entire team struggled with throughout the lifetime of the project.  And it’s a dilemma I think every team out there that’s designing an MMO today has to struggle with,  and the actual point of this post, which I’m only just now actually getting around to:

 How much do you copy the genre leader?

Dusty’s actual question is a good one, but that isn’t what really caught my eye.  You see, while we were building Pirates of the Burning Sea we had a similar dynamic to our team.  World of Warcraft came out two years after we started, so nobody had played it. Instead we had one designer who figured that the MMO genre started with EverQuest where most of the rest of us pegged that event at some earlier game. This guy refused to acknowledge Ultima Online as a “real” MMO despite its hundreds of thousands of subscribers and massive success. He thought even less of the games that came before it: The Realm, Meridian 59, and the thousands of MUDs.

For my part, I saw Ultima Online as a logical next step from the MUDs I played in college in the early 90s. I was pretty far gone into a couple of TinyMUCKs back then.  (I just checked and I do, in fact, still have my wiz bit on PegasusMuck.) When called on to date the start of the MMO I usually give two answers: UO was the first commercial success. MUDs (starting with MUD1, I guess) were the origin of the design genre. To me the distinction is important because of all the ways that MUDs break when your playerbase is counted in the tens of thousands instead of hundreds. UO was really the first game to deal with that kind of scale in the design, so it was the first “real” MMO.

It shouldn’t surprise me that there are people working on MMOs today that consider World of Warcraft the first real example of this kind of game.  It has thirty or fourty times the number of subscribers that EverQuest had at its peak. That increase changed the dynamics of the game just as much as the previous 30-40x jump made EverQuest and Ultima Online different from the games that preceeded them.  My only fear is that this will drive more companies into direct competition with WoW (and the $40 million plus games that are intended to compete with it) instead of toward building a nice tidy business aimed at a niche of 100,000 to 300,000 players who are craving something different.

What is your answer when you are trying to come up with the first real MMO?

The reasons your users will hate frequent deployments are:

• Non-zero server downtime when a new build is released
• Being forced to log out of their existing play session for a new build
• Waiting for patches to download for every little change

I will describe solutions to each of these problems below.

Eliminating Lengthy Server Downtime

The database architecture in Pirates requires that every character in the database be loaded and processed whenever there is a schema change.  From the programmer’s point of view such changes are easy to make, so they occur frequently. Unfortunately this processing is why the Pirates servers take an hour to upgrade. I don’t know how common this kind of processing is in other games, but I know we weren’t the only ones who had to deal with it.

The need to process character data stems from the fact that Pirates stores persistent character data in a SQL database, but the actual data is in opaque blobs. The blobs themselves contain just the field data for each object and do not include any kind of version information. The whole system is only able to contain one version of each class at one time.

A better way to go would be to build your persistence layer so that it could handle any number of historical versions for player data. If you add a schema version to each blob and register each new schema as it is encountered. When a server needs to read an object it can perform the schema upgrades on the fly and avoid the long slow upgrade of each object.  If necessary you could even have a process that crawls the database upgrading inactive characters in the background so that they will be ready to go the next time they login.

Supporting multiple concurrent versions

Database updates are the longest section of server downtime at patch time, but they aren’t the only one.  For Pirates (and most MMOs) each world instance must be taken offline so that the new version can be installed on the server hardware and then launched. This adds some downtime for the upgrade, but most painfully, it also disconnects all active players. A better method would be to build your server clusters in such a way that they tolerate multiple simultaneous versions of the code.

It turns out that Guild Wars already supports this.  One of the comments from the last post describes a bit about how this works:

Once the new code was on all the servers in the datacenters, we flipped a switch that notified the live servers that any new ‘instances’ should be started using the latest version of the game code. At this point, any users running older builds of the client got told that they needed to update, and attempts to load into a new instance would be rejected until they updated.
To summarize how the live update worked on the server, I believe we atomically loaded the new game content onto the servers, and then loaded the newest version of the game code into the server processes alongside the older version, because each build was a single loadable DLL. That let us keep old instances alive alongside new instances on the same hardware, and isolated us from some of the hairy problems you’d get from multiple client versions running in the same world.

For players in the PvE parts of the game this method apparently worked quite well. However PvP players could only play with each other when everyone was running the same version of the game, so this caused problems with PvP events. It isn’t perfect, but this is a huge step in the right direction.

The Guild Wars approach worked for them, but would be less suitable for Pirates which has eight different types of processes making up its server clusters.  However with some changes to the Pirates Server Directory multiple versions of those exes could easily co-exist on a single server machine. When a client connects to Pirates it queries the server directory process to find an IP address and port number to connect to. That query already takes the version number into account, and could be adapted to simply never return cluster servers from an older version.

Another approach that would eliminate the need for any client downtime for many patches is to make those systems work more like the web. The next time I architect an MMO (at Divide by Zero Games, actually) I intend to move most of what the traditional game client does outside of the typical persistent client-server connection entirely. Systems like mission interaction, guild management, in-game mail, and the like don’t require the same level of responsiveness as moving and attacking. If those systems use non-persistent HTTP connections as their transport the same protocols can be re-used to support web, mobile, or social network front ends  to the same data. For chat a standarized protocol (Jabber maybe?) will let you use off the shelf servers and let chat move in and out of game easily. The more locked down these APIs are the less likely you are to affect them with your small patches.

Eliminating patch time

So you have eliminated all server downtime and even allowed outdated clients to stay online and play after a new version of your game is deployed. If players have to download a small patch and then spend a minute or two applying it to your giant pack files they are still going to be annoyed.  Well it seems that Guild Wars did a great job here too.

Almost all of the patching in Guild Wars is built right into the client and can be delayed until right before it is actually needed. When you download Guild Wars it is less than 1MB to start.  The first time you run the game it downloads everything you need to launch the game and get into character creation. After character creation it downloads enough to load the first town.  When you leave the first town and go out into the wilderness it loads what it needs for that zone before you leave the loading screen, and so on.  All of this data is stored on the user’s machine, so going back into town is fast.

The natural result of this is that if a user has outdated data when they reach the loading screen for a zone, tiny patches for the updated files are downloaded before they finishing zoning. The game never makes the user wait to patch data for sections of the world they aren’t anywhere near. When the client is updated an up to date list of the files in the latest version is downloaded and the client uses that to request updated data as necessary.

The next logical step once you have this partial patching system in place is to enable both servers and clients to load new data on the fly without shutting down. If you are doing it right, many of your new builds are going to be data-only and include no code changes. Small changes of that sort could easily be downloaded in the background and then switched on via a broadcast from the servers.

Fixing further technical problems

There are two remaining technical hurdles that are not directly visible to the players but still need to be solved if the latency between making a change and deploying it is to drop to under an hour. These two are strongly tied to the partial patching process described above: slow data file packing, and slow uploads to the datacenter patch servers.

The packed data files on Pirates are rebuilt from scratch every time a build is run. This is an artifact of my hacking in pack file support over a long weekend that could be solved with incremental pack file updates. Building them from scratch involves opening tens of thousands of files totalling 6GB in size, after compression, and compressing them out into 66 pack files and takes 80 minutes. Usually a much smaller number of files has actually changed, so an incremental build would reduce both the file opens and the data packing work itself.

That same incremental process could be applied to the patch deployment process itself.  Because of how Flying Lab’s process was arranged with SOE each packed data file had to be uploaded again in its entirity if it had any changes. That slowed the transfer to SOE considerably. I am not intimately familiar with the SOE patch servers themselves, but I suspect a similiar inefficiency existed on their end when a build was deployed.  This could also be eliminated with more meta-data of exactly what had changed, and you will need that information for the partial patching above to work anyway.

So is it worth all this trouble?

These changes represent a “never going to happen” amount of work for an existing game. While the work involved in building your game to eliminate these deployment issues is less for a game that is being build with that in mind, it still isn’t free.  Is it worth the expense to allow your game to deploy in an hour instead of seven and a half hours?

I think it is worth it from a purely operational perspective, and here’s why: One of the first few monthly builds released for Pirates included a bug that caused players to lose a ship at random whenever they scuttled a ship.  Their client would ask them to confirm that they wanted to scuttle whatever ship they clicked on, but when it got to the server it would instead delete the first ship in their list. Though it was only a one line code change to fix the bug it took many hours to deploy the changes. In the meantime quite a few players had deleted their favorite ships and all the cargo in them. We knew about the build in the morning of patch day, but couldn’t get a new build out for 8-10 hours, which put us into prime time. It caused a lot of bad blood that could have been avoided if we had been able to deploy the build faster. This is a particularly bad example, but this kind of thing happens all the time with live MMOs. (SWG had a bug go live where the command to launch fireworks would let players launch anything they liked into the air and then delete the object afterward.  That included but was not limited to fireworks, monsters, buildings, and other players.)

There are plenty of other reasons to be able to patch very quickly, and I may go into them in a future post. I think it’s worth ensuring you are able to push new builds within an hour simply to be able to fix your major screw-ups as quickly as possible and save your players grief.

Earlier this week Eric Ries and Timothy Fitz posted on a technique called Continuous Deployment that they use at IMVU.  Timothy describes it like this:

The high level of our process is dead simple: Continuously integrate (commit early and often). On commit automatically run all tests. If the tests pass deploy to the cluster. If the deploy succeeds, repeat.

Those posts seem to be mostly about deploying new code to web servers. I thought it would be interesting to consider what it would take to follow a similar process for an MMO with a big installed client. I will do that in two parts with today’s part being the deployment process itself.  In a future post I will describe the architectural changes you would have to make to not have frequent updates annoy your players enough that they riot and burn down your datacenter (or go play someone else’s game instead.)

As a point of comparison, here’s how patches are typically developed and deployed for Pirates of the Burning Sea:

1. The team makes the changes. That takes three weeks for monthly milestone builds with a lot of the integration happening in the last week. For a hotfix it usually takes a few hours to get to the root cause of the problem and then half an hour to make the change, build, and test on the developer’s machine.
2. A build is run on the affected branch. When that’s a full build it takes 1:20 hours. Incremental builds can be ten minutes if no major headers changed. The form of build that people on the dev team use is copied to a shared server at this point.
3.  Some quick automated tests are run to fail seriously broken builds as early as possible. These take 10 minutes.
4. The build is packed up for deployment. This takes 80 minutes. The build is now ready for manual testing (since the testers use the packed retail files to be as close to a customer install as possible.)
5. Slow automated tests run. These take 30 minutes.
6. (concurrent with 5 if necessary) The testers run a suite of manual tests called BVTs. These take about half an hour. These tests run on each nightly build even if it isn’t going to be deployed to keep people from losing much time to bad builds. Most people wait for the BVTs to finish before upgrading to the previous night’s build.
7. For milestone builds the testers spend at least two weeks testing all the new stuff. Builds are sent to SOE for localization of new content at the same time.
8. A release test pass is run with a larger set of tests. These take about three hours.
9. The build is uploaded to SOE for client patching and to the datacenter for server upgrades. This takes a few minutes for a hotfix or 6-8 hours for a month’s worth of changes.
10. At this point the build is pushed to the test server. Milestone builds sit there for a couple weeks. Hot fixes might stay for a few hours or even go straight to live in an emergency.
11. SOE pushes the patch to their patch servers without activating the patch for download. That takes between one and eight hours depending on the size of the patch.
12. At the pre-determined downtime, the servers are taken offline and SOE is told to activate the patch. This usually happens at 1am PST and doesn’t take any time. Customers can begin downloading the patch at this point.
13. Flying Lab operations people upgrade all the servers in parallel. This takes up to three hours for milestone builds, but more like an hour for hotfixes. They bring the servers up locked so that they can be tested before people come in.
14. The customer service team tests each server to make sure it’s pretty much working and then operations unlocks the servers.  That takes about 20 minutes.
15. If the build was being deployed to the test server, after some days of testing steps 11-14 will happen again for the live servers.

This may seem like a long, slow process but it actually works pretty well.  Pirates has had 11 “monthly” patches since it launched, and they go well.  Not many other MMOs are pushing content with that frequency.  Some of the slow bits (like the upload to SOE and their push to their patchers) are the result of organizational handoffs that would take quite a bit of work to change. Flying Lab also spends a fair amount of time testing manually over and above the automated tests. Those manual tests have kept thousands of bugs from going live and as far as I’m concerned they are an excellent use of resources. I am not trying to bash Flying Lab in any way, just provide a baseline for what an MMO deployment process looks like. I suspect many other games have a similar sequence of events going into each patch, and would love to hear confirmation or rebuttal from people who have experience on other games.

Assuming minimum time for a patch, these are the times we are left with (in hours and minutes):

Without going into detail I am going to assume three architectural changes to make these targets much more likely. First, the way that the game uses packed files of assets will need to be changed to allow trivial incremental changes on every file in the game without any 80 minute pack processes. I will assume that pack time can go away and be a minor part of deployment.  The other assumption is that unlike Pirates, our theoretical continuously deployed MMO will not require every object in the database to be touched when a new build is deployed so the hour spent upgrading databases on the servers is reduced to about five minutes of file copying. The third architectural change I will assume involves moving almost all of the patching process into the client itself and out of the launcher. This eliminates both the transfer of bits to Southern California and the push of pre-built patches out to the patcher clusters around the world. I will go into each of these assumptions in my next post, but let’s just pretend for now that they appear magically and reduce our time accordingly.

The amount of time spent on automated testing in this deployment process is 40 minutes. Fortunately this work is relatively easy to spread out over multiple machines to reduce the total time involved. Assuming a test farm of at least five machines we should be able to accomplish our ten minute goal.

It is possible that we could have multiple testers working in parallel to perform the manual tests more quickly. There are 230 minutes spent on testing in this sequence, so if we expect to do that amount of work with 11.5 manual testers assuming perfect division of labor. There are two things wrong with that statement. The first is that perfect division of labor among 12 people is impossible so the number is probably closer to 20.  The second problem is that having a team of 12 on staff to do nothing but test new builds with tiny incremental changes in them is not worth nearly what it costs you in salaries. A much more likely outcome is that the amount of manually testing in the build process is drastically reduced. In addition to that we can get our playerbase involved in the deployment testing if we swap things around a bit.