End of Week 7 Update : Two key "to do" items; CB Client Build and Network Redesign

Happy New Year - time for a New APB!

Happy New Year to all APB-ers. I hope you had a great Holiday, and just like us, I hope you are ready for some really exciting things in 2011. Our New Years vow is simple; launch APB in 2011 :)

So based on the transition work and R&D over the last several weeks, we now have enough clarity to at least make an initial prediction of when we will launch Closed Beta. Right now it looks like sometime in late February if everything goes exactly according to plan (and we know how often that happens :) ), but let's at least put it out there, CB will likely start sometime in February. Over the next week the team (lead by Jon-Enée, aka. Neume) will share exactly how you will be able to apply for participating in the CB, and we will then post the CB signup info as part of next week's update (and Neume will then also start sharing the details on the message boards as soon we are ready).

So what are the main obstacles we are currently facing (ie things that might derail the date, and/or make us delay CB)? Well, that's today's update. Be warned - it's a VERY tech-heavy but hopefully it gives you guys a sense of what we are working on behind the scenes.

Two key items that need to be in place before CB;
(1) New Game Client and (2) New Network for GamersFirst.

I will keep info on item (1) - the new game client - very short this week. I know I know, that's the most exciting update I could give, so you have to hold on for another week for more details. The exact config of the client will be THE focus for each update in each of the next several weeks, so at the moment all I will say is that the Closed Beta client will be a LITTLE bit smaller than the original shipping client, it will have a basic "premium" system built in, various balance fixes, and we MIGHT be able to turn on the chaos rule-set (!). The last one still requires some extensive QA, but that should give you guys an idea of what we are hoping to include in the first closed beta.

Therefore this week I will focus the discussion on item (2) that has to be fully sorted out before APB can go live; a brand new network for GamersFirst, and that means a brand new network for APB as well. This will likely be my last post that exclusively deals with items "external" to the game itself, since starting next week our discussion will focus on in-game features (so for non-tech-heads, this might be a good place to stop reading :) ).

Separate and apart from the actual APB game development efforts, our top-to-bottom network redesign is now a prerequisite for the APB launch. The redesign is not a small nor simple task, especially as we are in the midst of launching several other new games, trying to minimize any downtime to all our existing games during the transition, and on top of this, are also poised to launch APB with its massive network demands. The best analogy is that what we are attempting to do is like changing the engine oil while speeding down the highway at 100 miles per hour. Really tricky stuff. And bound to cause some mishaps along the way.

So why go through the trouble?

Historically, dating as far back as 2005, GamersFirst had servers in a lot of locations (Irvine, New York, Amsterdam, Frankfurt, London, Sao Paulo and Vienna) using a broad mix of various managed hosting setups as well as directly owned colo configs, and purchasing bandwidth from various single-homed providers. This patchwork of services caused various issues over the years, though our IT team put up a valiant fight in each instance to keep things running as smoothly as possible in spite of a sometimes overly "complex" architecture. After I joined the company a few years ago, one of the long-term wish-list items was to redesign and consolidate all the network assets, especially since some of the original designs were not the most secure nor the most cost-effective (something that became apparent during multiple attempted attacks against Knight Online by Turkish and Chinese nationals trying to inject game-gold and during various DDoS attacks).

Oddly enough, taking on more of the back-end network ownership should actually simplify our lives, and also reduces our long-term cost of delivering high end game services, and improves our ability to pin-point service problems in various regions. APB just happened to pop-up while we were in the middle of this massive redesign. Not really a problem, but something that certainly makes the initial APB re-launch just a little slower than we had originally hoped for.

Owning our network

In late August 2010 we were approved as an AS (Autonomous System) by ARIN and technically also as an ISP (since one of our business lines is to host games for other companies). We then promptly started building out what will become the new network, and have very slowly started moving some test traffic onto the Los Angeles portion of the new configuration. As of this moment 80% of our stuff still runs on the old network, with about 20% of traffic being pushed out of our new network. By the end of February we expect to reverse that and hope to have 80% of traffic running on the new network, and APB would be one of those key items running on the new and improved system.

The new design consists of a peering edge in Los Angeles with more than 20 Gbps of connected capacity, a new peering location (and hosting location) in Washington D.C with multiple peers, and a new peering and hosting location in Frankfurt, Germany (and in the process we are decommissioning New York, Amsterdam, London and Vienna) and a stand alone node in Sao Paulo. Between each of our core hosting locations we are going to be running multiple redundant OC-48 paths, and in the process of doing so, we will also be shutting down the jumble of internet access providers we have used previously (or in some cases, upgrading how their services are delivered to us; from being a consumer of IP services to being a direct IP peer). OC-48 is telco speak for a private DWDM fiber connection that actually lets us cross connect our core datacenters across private fiber optic strands.

The biggest benefit for the player; presuming we are well-peered enough (see the below discussion) you should be able to reach the games using fewer hops from your current internet broadband provider than you would use today. Also as soon as your traffic reaches any one of our new key nodes (Washington, Los Angeles, Frankfurt or Sao Paulo) if for any reason you need to connect to one of our far-away nodes, then your traffic would be routed through our private DWDM fiber optic backbone between our core locations once those strands are lit up (ie - the traffic would be physically "off" the public internet and now literally be sent by direct fibers between our locations). In most cases this would be a big benefit, since our network literally would move the data at the speed of light between our private nodes without any need for a public router hop.

Right now with the limited state of the current setup you can see our initial peering report (courtesy of Hurricane Electric): http://bgp.he.net/AS36187 or in graphical form: http://bgp.he.net/AS36187#_graph4 . It shows that at the moment our not yet fully deployed network is multi-homed with two direct peers (Telia and PacketExchange / Mzima). We will have at least one additional direct peer in each location by the time we go live, in addition to tapping in to two public peering exchanges (which would give us a whole slew of peers).

So, what does all this mumbo-jumbo mean? Being multi-homed (with at least two, but preferably three or more peers/providers) means that if one of our upstream peers were to explode (or do something nutty with their routing), traffic will still be able reach us across one of the other upstream providers. It also allows us to perform some basic traffic shaping (for example, it might turn out that in a specific country we might have better performance using a certain provider, and we can then prefer a certain peer for some of the traffic).

The HE toolkit is actually a great little collection of webpages you can use to figure out what various companies (those 37,000 or so companies that as of today actually announce origin routes globally that is) actually look like on the BGP-networking end:

Nexon America http://bgp.he.net/AS26236
Activision Publishing http://bgp.he.net/AS40551
Facebook http://bgp.he.net/AS32934
Amazon http://bgp.he.net/AS16509
Netflix http://bgp.he.net/AS2906

One word of warning; to have any kind of fun with this, you must be a network geek, but since I figure that some of you reading this blog might be geeks, this could be an interesting introduction to BGPv4 peering for you. It's worth noting that Facebook (the worlds biggest site in terms of visitors) have 63 direct IPv4 peers, and Netflix (the worlds largest site in terms of traffic volume moved per day) has only 6 direct IPv4 peers (though they currently move most of their traffic through LimeLight according to the report, which clearly is extremely well peered on its own).

You'll quickly realize (using the search function), that a lot of game companies don't ever go through the trouble of building their own global directly owned network, and instead rely on various providers to give them the first internet service hop. That's a perfectly fine strategy, and one that we have used for a long time, but with the launch of APB, the launch of four other MMO's on our site in 2011, as well as the complete network redesign that we were undertaking to improve performance and security, it made perfect sense for us to go down this route at this time and become a global wholesale network provider (more or less). Yes - technically you could actually now buy bandwidth FROM us on three continents, though I don't think we really have a business model for that yet :)

All this exciting network stuff just means it will make the APB launch just a little more exciting as well, given the fun complexities involved in launching the game on a brand new network. Our critical IT team will certainly have a lot of exciting things going on in the next several days.

Sticking to our mission that we are planning to share a LOT of information with the community as we go along, hopefully this gives everyone a sense of how the future APB network will operate. As new parts of it go live (like DC and Germany), I will point those out in future posts, as we get closer to actual production time.

Next week marks the beginning of the detailed game client discussions, so stay tuned!

Cheers,