Just a quick update on our progress as we prepare for our massive server merges. First the TL;DR version; after a lot of testing and simulated merging even through today, we have decided to postpone the EU merge this week and are moving it to the same week as the previously scheduled US merge (the week of May 18-22), with the EU merge likely occurring at the start of that week, and the US merge occurring during the second half that week.
The fundamental concern; if this merge runs in to any issues, it would be nearly impossible to reverse once it's set in motion, and any post-merge repair job could take a very long time to complete. At this point we don't have enough test data showing us that the merge will succeed in all cases.
Therefore the decision was made today to give the first merge at least 10 more days of QA and testing time before we pull the trigger.
An insane number of merge simulations...
These server merges may be the riskiest operation we have ever undertaken with the game, other than maybe taking over the game in the first place, since it's going to be nearly impossible to undo or reverse once it's been kicked off.Over the last two months we were busy writing the merge code and revising database systems to handle the size and complexity of the merge, while also running simulations of every type of merge we could conceive of.
Our first approach "worked," but yielded merge simulations showing the game might have to be offline several days and nights to complete the merge! Therefore over the last couple of weeks we reversed course and designed a new type of merge that lets us bring the game back online in less than 24 hours.
Old Plan: Merge 5.7 million characters in two rounds.
Since APB:R launched in 2011, a LOT of characters have been created by you and your friends - over 5.7 million characters to be exact. For each one of these characters we store each character's customized appearance, vehicles, symbols, themes, inventories, friends lists, in game mail and a whole host of internal parameters that relate to each character's relation to all other characters in that world, like clan membership, the "author" of any items in your inventory you bought from the auction house etc.
Suffice to say, reliably moving this data is a colossal undertaking in terms of both scripting, hardware configuration and sheer processing time. As you might expect, the most time-consuming pieces to migrate are related to character customization and inventory.
On the technical side, each of our 5.7 million characters "own" a set of around 30 distinct, long XML blocks, and each of these must be converted, validated and potentially modified as they move to the merged database. This is why our last tests using this method showed it would take a colossal 3 days (72+ hours) of continuous downtime to merge the EU servers, and slightly less than that to merge the US servers. This would simply be an unreasonable amount of time to keep everyone out of the game this week.
New Plan: Merging On-Demand active characters.
Over the last couple of weeks our lead engineer in charge of the merge team devised a clever new way of reducing the 72-hour downtime down to a more manageable 8 hour downtime window, to fit in an "extended maintenance" period. Rather than merging all characters at once, the team implemented a new system to merge a lot of the preliminary data first, and then merge each character in real-time as players log into to the new world for the first time. This means that you will only need to wait 2 seconds for each of your characters to "arrive" before hopping right into APB again.
While this plan is certainly faster, it makes the upfront development effort significantly more complex, since we now have to consider the edge cases around the fact that characters will essentially be created on the fly on the new server at different times. In essence it became a live "character migration" in real time. The new issues that arise are things like - what does this mean for clans with members arriving at the new server at different times? What about mail that was sent by a character that doesn't yet exist, but might "arrive" at the new server eventually?
After the last few weeks of development we are finally confident we have a good solution, but now we need to spend significantly more time testing all these edge cases.
How long before it goes live?
The merge is something we have been working toward for a long time, and we never like pushing these dates back. So while we can't merge this week simply because we don't yet have all the test data showing us that we have a green light, the new plan is to complete both the EU and US merges the week of 5/18.
Thanks for all your patience. I assure you that it will be worth the wait!
Stay tuned for another post later this week, where I'll be going over the brand new roadmap (including the Engine Update) and unveiling the six new names for the six going forward servers for PC and Console for both EU and US!
Jobs
