Pages

Sunday, January 30, 2011

End of Week 11 Update : Game Changes and Closed Beta Features

Hi guys

This week Zak (aka "Qwentle") our designer will give an overview of design changes to expect in the Closed Beta. I have asked the team to be very open and honest about a lot of the things we are working on, so hopefully you guys will enjoy this week's read.

Bjorn / TechMech

---------------------------------------------------------------------------------------

Hey guys, for those that haven’t come across me yet, I’m Zak "Qwentle" Littwin, now the Designer for APB Reloaded (and previously the Development QA Lead at Realtime Worlds).  I’ve been asked by Bjorn to talk about some of the changes we will be testing during Closed Beta. This is going to be a fairly long post, so you may want to get a cup of tea and some biccies. Even so, this is only Part 1 of many posts about the upcoming game changes (and probably these design posts will occur quite frequently as we progress toward the renewed live service).

“Tease” Warning
First; let me be clear – in today’s post we are ONLY talking about the TWEAKS that are present in the Closed Beta (these are items we are actually testing out in this first release round, along with extensive system and server tests), and I will NOT be talking about the much more extensive changes we are planning to implement in the Open Beta and especially for Live Service. So without giving too much away, one of the key changes for a future release (again NOT for Closed Beta nor in this post) is the expansion of game modes (especially the addition of “Chaos” and the creation of the completely new mode called “Turf Wars’) as well as the release of a whole new district (the “Asylum” expansion) which we are working on to turn in to an awesome map, and which will drastically alter how players can engage each other, and potentially will alter (for the better) the whole idea of missions, while leaving in place some of the ideas behind the original game play from the original districts. 

These future changes will truly change the game play of APB and will make the game much more sustainable (imagine defending areas in the Asylum in a team vs team or even Clan vs Clan setup). But to be clear – today’s post is NOT about those changes. So forget this little teaser. Instead today’s post ONLY talks about the things that in fact will be in the Closed Beta.

Closed Beta Changes (ie - things we need our Closed Beta players to test) 

Gameplay Changes

Integrated Cheat Detection
First; yes – we have enabled PunkBuster in the closed beta. Not directly a gameplay change, but we think this will improve the gameplay experience quite a bit. This is a significant code change and will require a lot of testing (see Aphadon’s post from last week). So there will be a lot of technical changes as part of this update of which some have been mentioned in previous posts. Death to Cheaters!

Weapon modifications
Over in the Gameplay department, during Closed Beta the first thing you will notice is the change to weapon implementations. While previously the game was fairly well balanced (apart from a couple of weapons), the combat model wasn’t particularly fun in a lot of situations, with weapons not feeling very meaty (though the recoil addition in 1.4.1 helped somewhat). In Closed Beta to fix this, we’ve upped initial damage on all weapons, adjusted recoil for most weapons, and changed accuracy so the weapon very quickly blooms out to its minimum accuracy, but regains it very quickly once you stop firing.  These are fairly major changes, and will likely affect balance initially, and it is something we need to try out extensively. This change also gives some advantage to single-shot type weapons. We’ll be testing these balance changes extensively during the beta to see what you guys think. In our internal initial playtests the changes make the combat feel a lot more fun (which after all is the whole purpose of the game). 

General gameplay changes (marksmanship, witnessing, camera)
Other gameplay changes include the following items; marksmanship will not disable when you fall short distances, teammates will now show accurately on your personal radar screen, the arrest exploit has been fixed and a few other minor changes across various systems. Witnessing has been completely removed for the time being (the ready system kind of broke it), but we plan to reintroduce it in a near future build in a much improved fashion, as it is intregal to the open world experience. The default camera position has also been moved out slightly so that your head doesn’t obscure the centre of screen when the player runs to the right.

Core Game Progression Change
We’ve done a lot of work with changing the progression system ahead of Closed Beta. Progression was really convoluted before, with rewards at seemingly random points, and it had some strange mechanics that needed a lot of explaining by game prompts. Additionally in-game money pricing structure on some of the game items resulted in massive run-away economy problems. While there are other areas of the game that are higher priority to fix (for example spawning, matchmaking and missions), progression is something that is a lot harder to change once we are in live service, so it made sense for us to look at it now and create the new Progression System to be tested in Closed Beta. 

The first thing we did was remove organisation standing. It was a fairly obtuse mechanic that didn’t really fit into the game very well. Then we adjusted the contacts so they each fall under the category of either Weapons contacts or Vehicle contacts. As the names suggest – vehicle contacts unlock new vehicles, vehicle upgrades, and vehicle components, while weapons contacts now unlock preset weapons and character upgrades. All contacts unlock preset and customisable clothing options. Since the presets are now properly named you will now be able to tell which items and combinations are actually meant to be worn together (an annoying missing item in the original release).   

Open slot weapons and weapon upgrades are unlocked through specific roles that are levelled through using that weapon class (so shotguns and SMGs are all unlocked through the Pointman role, along with upgrades such as Magazine Pull and Reflex Sight), while higher level equipment is unlocked through performing the activity that relates to it (for example Performing Arsons unlocks better Petrol Cans). One thing that helps a lot is the new ability to see how much progress you have made in a certain role or for a certain achievement.  While progress bars couldn’t be added before Closed Beta due to some current technical constraints of how the UI works, we have added a numerical readout (ie. 5/12) so you can see how close you are to actually reaching the next level (making levelling something that now matters a lot more, and has more impact). 

All Symbols and some pieces of clothing are now rewarded through random rewards at the end of a mission.  Players have a very high chance (50%, modified by mission performance) to unlock one at the end of each mission, with different contacts unlocking different symbols.

High level vehicles are now using a slightly different unlock system. You unlock the base vehicles and presets through vehicle contacts at high levels, with kits and open slot versions gained through completing time trials around the districts with those vehicles (this bit’s not in yet, so open slot versions of the high-level cars, along with their kits, won’t initially be available in the first beta. We’ll be working on getting this in ASAP though).
Another major change to progression is leagues. We felt that leagues were not providing their original purpose, and instead were just causing players to exploit game systems, or ‘Zerg’ things in order to compete.  As such we’ve removed them completely in the short term, though we plan to add back Leagues at some point, potentially along with the revised (and more meaningful) Clan mechanics which are used in other games as well. 

Finally the "in-game" or "earned money" cost for most items has been drastically increased. It’s better for players to have a load of stuff they want and have to work to get the money together to buy them, rather than be able to buy everything right away and have a ton of money left over (something that had made the original money system strangely unbalanced). These changes are something that will need a lot of testing in Closed Beta to get to the right values, so we’re expecting a lot of analysis of user behaviour in this area. It also ties in to the roll-out of Premium vs. Free accounts (something we will be dealing with before and during Open Beta in greater detail).

We have also done some work on the Upgrade system. We’ve moved all upgrades into groups, so you can only use one of each type (we’ve also renamed them to fit with each system). I’ve also removed a couple of mods that either didn’t fit or were duplicates (Savage is gone, and Monolith has also been removed as it was functionally identical to Survivor). Going forward we plan on upping the difference that upgrades make, but adding substantial negative modifiers also, meaning players will specialise within roles rather than just increase stats. To facilitate this, we’ve also allowed the removal of upgrades without destruction. Initially this was added as a cash sink, but we found from both internal testing and live that it only served to stifle player choice, as players made a decision and never changed or explored other options along the way.

Vehicle Changes
During Closed Beta most vehicles have not changed dramatically, though a few vehicles have changed significantly. The biggest change is the removal of the “Han Cellente,” ie. the starting vehicle. We thought it looked pretty awful and handled terribly, leading many to concludes that cars sucked in general (when in fact, they had just been given a really badly behaved one to start with). Instead we have inserted another vehicle as the starter car that looks a lot nicer, and that has pretty decent handling. We have modified how this new starter vehicle operates and with a couple of tweaks we’re left with a cute little 2-door coupe that handles well, is stable and responsive, but not too fast (maybe not a very macho car, but that's why you will be upgrading at some point). So this should now be a really good vehicle to learn the vehicle system on.

Other changes include the G20 Vegas, which has been given massively increased acceleration and torque from scratch (just enough to make the front wheels lift off the ground if you slam your foot down on the pedal).  It’s a nice little effect but if it turns out to cause significant problems in Closed Beta testing we can tone it down a little. But heck – why not go over the top in Closed Beta with a few things, it's the best time to try crazy stuff that might work?

The Fresno has had its weight and power increased (so no change in handling, but better ramming potential).  We want it to feel like an old steel chassis vehicle when compared to it's lightweight contemporaries. The Mikro/Vaquero now grip the road a lot better and both them and the Bishada/Jericho have had their health reduced in order to put them just below average hp. 

The largest change to an existing vehicle is the Balkan Kolva (Dump Truck). We have made it rarer in the game, and massively increased the weight and torque, while lowering the turning speed and initial acceleration. Combined with much higher health (it is a fully loaded dump truck filled with tons and tons of sand after all), we’re left with a lumbering vehicle that needs a run up to get speed, but packs a rather insane punch if it manages to hit something. Ideally the really low acceleration from scratch will stop it from being a griefing tool (it can’t really change direction to block someone in a pinch), but we’ll have to wait till beta to see whether it works in a large scale environment. 

The idea behind these vehicle changes, and more changes we will be making in the future, is to give each vehicle a lot of individual personality and specialisation and unique benefits, rather than everyone rushing straight to the Cisco/Bishada/Jericho/T25.

The Premium Quandary
The last thing I was originally going to talk about was premium accounts and leased weapons since that's another big change for the new version of the game. In short, we want to make premiums highly desirable purchases, but not something that crushes Free2Play players. Our goal is to ensure everyone has fun in the game, while simultaneously encouraging premium activities, since that ensures we can stick around and pay for the costs of running and expanding the game over the next several years. We are also in the middle of creating weapons that can be leased that aren't necessarily better than the ones you earn, just different, more specialized or has some other specific benefits. However, seeing how we are running out of blog space (cough...), we are planning to share all the information about premium/free and earned/leased breakdowns as we get closer to launch. And just another teaser - you might be able to spot some of these new weapons in the images below (and it just means you will have to stick around for the future post where we will dig in to the new weapons in great detail).

Conclusion

So that’s what we’re initially putting into closed beta.  All of the above is of course subject to change and a lot of future changes will based on both measuring behaviour as well as reviewing feedback during the beta (since sometimes people request one thing, but actually acts in a completely different way when we measure their actions).

In a future post I will discuss subsequent changes planned for Open Beta and Live (the maps and complete game system changes I hinted at in the top of the post – since THAT will be truly revolutionary to the overall game play for the entire game), but for now I’ll leave you with some screenshots of the updated game highlighting a couple of the weather changes we have added (plus some of the new weapons we're adding).

Qwentle / your Designer

Weather updates in updated game (with some teaser weapons) 








-----------------------------

Let's all thank Qwentle for sharing a lot of Closed Beta change details in this "Part 1 of Many" series outlining design changes the game will be going through over the next several months. Clearly there are four big "teases" in this post alone; the exact leased weapon configurations, what you actually get for being "premium," the new game modes/engagement modes and finally the actual details about the new world/map (other than the fact that it will be called "Asylum" we know very little). Nice. I hope this has whet your appetite a little bit for our future posts, so we will hopefully see you back here next week!

Cheers,

Sunday, January 23, 2011

End of Week 10 Update : Busting some Punks?

Alright - the time has come to start having the rest of the team start chiming in on the development to help shed more light on what's actually happening under the surface. First up is Johann, who originally had worked on the PB integration for the original release, and is now responsible for the overall anti-hack/anti-cheat detection for the game as a whole. My plan is to let key members share their ideas on this blog regularly, and hopefully that will spark some interesting discussions.

Johann - take it away:

----------------------------------------------------------------------------------


Busting some Punks
by Aphadon
 
Hi, I’m Johann aka ‘Aphadon’ - previously a Living City / AI Engineer for Realtime Worlds, and now a ‘Pretty-Much-Anything’ Engineer on APB Reloaded. When TechMech recently asked the team if we’d like to share anything on the blog, I thought I’d take the opportunity to discuss in greater detail our plans for addressing the cheating problem. This is a subject we are quite passionate about, and while it may be virtually impossible to ever win the war against cheaters completely, we certainly intend to give it our best shot.
But before I get into the details of where we stand with that now, let me start off by explaining a bit of the development history. 

We originally integrated PunkBuster into APB in January 2009. After this was completed, it was disabled and essentially left untested until near the end of Closed Beta (~ April 2010), since it was not deemed an essential feature to enable before launch. However when we finally did turn it on, we were caught off guard by a whole bunch of unexpected issues. Many people were getting kicked due to PunkBuster configuration problems, the servers were hitting lag spikes upwards of 10 seconds every few minutes (we used to joke internally and call them “lag mountains” since they were too big to be called spikes), and some people even reported BSOD’s starting on the day we turned it on. So in a nutshell,  it was quite a mess.

We were forced to disable PunkBuster again straight away until we could fix it, and ended up launching the game with it turned off. I offered to help out at this point since the developers in charge of PunkBuster at the time were tied up with other work, and I spent the next two months working closely with Even Balance to track down and fix the majority of issues. We eventually managed to turn it back on without issues for a few weeks before the servers were shut down, and -  contrary to what the aimbot sites would have you believe - during that period PunkBuster was able to successfully detect every single commercially available aimbot for APB. Over 1000 cheating players were silently identified and logged by the system, but unfortunately RTW was already having financial difficulties by then and we never had the opportunity to get our hands on the list of cheaters to ban them (since once you stop paying the people with the info, they are unlikely to give it to you).

Fast forward to today, and we’ve just completed a full upgrade of PunkBuster to the very latest version. I’ve been going over the issues we’ve seen before, and Even Balance is confident that all of them will be fixed in this version, along with new and improved detection methods and many other behind-the-scenes improvements in addition to Punk Buster alone. We also plan to have anti-cheating enabled on all servers from the very start of Closed Beta, allowing us to get as much testing done as possible. That way if any more surprises turn up, we should have more than enough time to fix them.

Ok, the non-technical people may wake up again now.  The next bit of information I’d like to share is our proposed banning policy. This isn’t set in stone yet, and some parts of it may not even be finished during beta, but this is what we would like to work towards: 

If a player is positively identified as running a known aimbot, he/she will be immediately kicked from the district and a notification message will be sent to all other players in that district. The cheater’s account and PC will be immediately banned for a period of time, during which the person will not be able to play using that account on any computer. If an aimbot is detected a second time, the account and computer will immediately be permanently banned, all monies paid will be forfeited and any upgrades or customizations will be revoked. 

This might seem quite harsh, but there really is no excuse for aimbotting and this time around it will be treated with zero tolerance. We believe this policy will cut down on cheating significantly, and ensure a fairer and more fun experience for everyone.

Next week I believe Qwentle - our Designer - should have a somewhat more light-hearted post to share with you, containing some juicy tidbits about what we’ve been up to recently (aside from plotting the doom of cheaters, that is).

Cheers
Johann / Aphadon.

----------------------------------------------------------------------------------

There you have it. Many thanks to Aphadon for sharing, and as he is alluding to, there are actually in this build going to be things other than Punk Buster to help keep the playing field level. What are those? Who knows :) What we all do know - we take game cheating extremely seriously. At some point we want to introduce bullet-proof vests, and for that to work, we clearly need headshots also be something you can actually achieve.

And everyone should realize that downloading hacks (even commercial ones) almost always entails installing a keylogger and malware on your own machine. Why anyone would ever do that is beyond us.

Til next week,

Sunday, January 16, 2011

End of Week 9 Update : We are still here!

As promised last week, as we move closer to Beta these postings will get really really (really) short, until just before the beta launch when we will indeed have more details to share about the actual beta content.

Therefore this week's post is just a quickie "blip" on the radar, and I will answer just a couple of questions from last week's comments before jumping head-first back into the logistics of getting the game out;

First; YES (and it bears repeating), the goal is to let you keep ALL customizations you created using the old game. In some cases that might mean some odd situations, such as having items that may or may not fit the new progression system, and having items that the new F2P model may not have let you design unless you had a premium account (or similar "issues"). That's ok. You will basically be unique thanks to you having played the game "back in the day" and our goal is to make sure your many gazillion of hours of investment do not go to waste.

Second; To set Beta expectations correctly; we are indeed long term working on "Purposeful End-Game PvP." However, this modificaiton is not likely to make it in to the first beta (and in fact is one of the hardest and most complicated modificaitons we can make to the game). So in addition to the many balance fixes, one of the real goals with the game long-term is to turn it in to a platform for many different unique experiences. As such, high level players need something to sink their teeth into after having done all the initial grinding. But; we don't expect that to appear in Closed Beta, but, we are indeed working hard on ways to add higher-level-purpose to the long term experience. As that initiative gets closer to completion we will clearly discuss that in more detail - both here and on the message boards.

So - with this very short update - til next week. Also starting next week, expect a few guest columnists reporting from the various activities going on in and around the game.

Cheers,

Sunday, January 9, 2011

End of Week 8 Update : Steps to Join the Beta

Closed Beta Steps - easy now, tricky later 

So today's post is going to be short, but somewhat information dense. I will start keeping things short since we are now completely in the thick of getting this game to the finish line.

To register for the beta, we have decided to take a pretty "simple" route in order for you to take the first step;

(1) simply submit your email here: http://www.gamersfirst.com/apb/ in the "stay up to date" box. Make sure you give us one that we can actually reach you on - and not the 14 minute email variation :).
(2) Create a GamersFirst account (preferably with the same email, though not required)

That's it at the moment is the actual first required sign-up step (simple - eh).

But then it gets a little trickier. About 2 weeks before the closed beta (which probably means on or about the first week in February), you will receive an email sent to the address you submitted above. That email will have a unique link where you will go to activate your beta request. In order to do so you will be asked to submit a lot of additional data, among other things your DxDiag (if you don't know that is, then you might not be a great closed beta candidate) and some additional information about you, as well as about your system and your own background as a gamer. Once we have verified the data, we will then send you back your actual beta key. If you feel like you don't want to share the additional data being asked of you, you can always back out of the process, and just ask to be notified about the Open Beta when that's ready instead.

So far we have received about 20,000 emails that have registered their interest in the game at the above location, and we do expect that about 10%-20% of players who sign up for the email notification will in fact fulfill all our closed beta registration requirements.

Our goal is to have a closed beta with approximately 6,000 - 8,000 fully registered and confirmed players, which roughly means we'd expect to see 800 players online at peak times during the closed beta, which would be sufficient to test all the various system components before going in to the bigger Open Beta.

So why are we gathering this extra data, and why have a Closed Beta in the first place?
The question of "why have a closed beta" came up in a previous series of blog replies. It might seem obvious to us why we want to run the game with a limited audience as first pass. But happy to clarify.

For one thing, there are several components that have been updated since the game went dormant. Among them are the Vivox voice over IP libraries, the installation and unpacking components, and there will be some new components we need to test in order to see what might be normal vs. "abnormal" gameplay (you know what I am referring to :) ). As part of this process it's critical we gather data on as many diverse machines and systems as possible. Thus we need to get as many normal, common, uncommon, unusual, strange, weird and just plain odd system configurations into the beta as possible. If we don't think we have sufficient "diversity" in systems we would need to seek out more closed beta testers (though at this point that should not be a big problem).

The second main goal will be to test some of the new balance and progression systems. They will not at first be extremely different than what they were during the original run of the game, however, these progression changes are now "starting their journey" toward incremental change for the real long term (and we hope - improvements), so we need to see how people interact with them during the game run.

What do you "Get" for participating in the beta?

In return for giving us a bunch of extra data, and taking the survey when you sign-up for the beta (and then spending hours in the game) we plan to give you something lasting. As a thank you to our beta testers we expect to give away a token that shows you were a beta tester back in the day. Trust us, we will not forget who you were :)

Finally - what about those pre-existing characters and accounts from the old game?
First of all NO PRE-EXISTING ACCOUNTS will be part of the closed beta. During the closed beta you will create throw-away accounts. That's the purpose of closed beta. 

So this topic is one we will come back to later on, before Open Beta, but at the moment we are trying to allow players to reclaim "lost and found" items as part of the Open Beta (maybe in the "San Paro Lost and Found Department?").

Not all details are finalized for this (since it's really not part of the first beta) - but a few things are pretty clear at this point, and I am happy to share them (though it's possible some will have to be modified down the line);
  • YES - we plan on letting players of the original APB game reclaim the original content (ie characters etc.) that existed at the time the game shut down, since that data is considered "game content." This would give you most (if not all) customizations, gear, and character names back that you had during the first run of the game.
  • NO - we are likely to NOT give you back all the progression data (though some, but this particular point will be explained in much more detail before Open Beta since part of the changes to the game itself relate to progression). 
  • YES - you will need to create a NEW user account (a G1 account that is) before you can reclaim your characters, which means you have to basically sign up and agree to the new terms of service that govern the use of all G1 accounts (ie your old RTW account cannot sign you in to the game or anything else)
  • NO - we will NOT have any of your original payment data on our side, and therefore you also will have no RTW points or cash-equivalent credits. We simply have no information about anything you bought via EA, Digital River nor Steam. So we will have no idea what you did or did not purchase back during the original run of the game. No point in asking us. We simply don't know. If you tell us you spent a bizillion dollars we might feel really sorry for you. And the best sad story might very well win you some sympathy and recognition from our development team. But we simply don't have the data (nor the money) from whatever happened back then. Remember, that company simply does not exist any longer. I know I am going to have to repeat this information many times :(
  • and finally it's LIKELY - that you will be able to "patch-up" to the new game from the original installation. This gets a little bit tricky due to how patching is handled - but it at least appears plausible. Therefore, at least keep any old installations of the game around if you have the space on your drive, since it will save you a lot of content download time. However, we will get back to you before launch to verify this last part.
So, we will continue working hard to keep everyone informed, and I hope the above info has given you guys some additional clarity this week. More info to come next week!

Back to work....

Sunday, January 2, 2011

End of Week 7 Update : Two key "to do" items; CB Client Build and Network Redesign

Happy New Year - time for a New APB!

Happy New Year to all APB-ers. I hope you had a great Holiday, and just like us, I hope you are ready for some really exciting things in 2011. Our New Years vow is simple; launch APB in 2011 :)

So based on the transition work and R&D over the last several weeks, we now have enough clarity to at least make an initial prediction of when we will launch Closed Beta. Right now it looks like sometime in late February if everything goes exactly according to plan (and we know how often that happens :) ), but let's at least put it out there, CB will likely start sometime in February. Over the next week the team (lead by Jon-Enée, aka. Neume) will share exactly how you will be able to apply for participating in the CB, and we will then post the CB signup info as part of next week's update (and Neume will then also start sharing the details on the message boards as soon we are ready).

So what are the main obstacles we are currently facing (ie things that might derail the date, and/or make us delay CB)? Well, that's today's update. Be warned - it's a VERY tech-heavy but hopefully it gives you guys a sense of what we are working on behind the scenes.

Two key items that need to be in place before CB;
(1) New Game Client and (2) New Network for GamersFirst.


I will keep info on item (1) - the new game client - very short this week. I know I know, that's the most exciting update I could give, so you have to hold on for another week for more details. The exact config of the client will be THE focus for each update in each of the next several weeks, so at the moment all I will say is that the Closed Beta client will be a LITTLE bit smaller than the original shipping client, it will have a basic "premium" system built in, various balance fixes, and we MIGHT be able to turn on the chaos rule-set (!). The last one still requires some extensive QA, but that should give you guys an idea of what we are hoping to include in the first closed beta.

Therefore this week I will focus the discussion on item (2) that has to be fully sorted out before APB can go live; a brand new network for GamersFirst, and that means a brand new network for APB as well. This will likely be my last post that exclusively deals with items "external" to the game itself, since starting next week our discussion will focus on in-game features (so for non-tech-heads, this might be a good place to stop reading :) ).

Separate and apart from the actual APB game development efforts, our top-to-bottom network redesign is now a prerequisite for the APB launch. The redesign is not a small nor simple task, especially as we are in the midst of launching several other new games, trying to minimize any downtime to all our existing games during the transition, and on top of this, are also poised to launch APB with its massive network demands. The best analogy is that what we are attempting to do is like changing the engine oil while speeding down the highway at 100 miles per hour. Really tricky stuff. And bound to cause some mishaps along the way.

So why go through the trouble?

Historically, dating as far back as 2005, GamersFirst had servers in a lot of locations (Irvine, New York, Amsterdam, Frankfurt, London, Sao Paulo and Vienna) using a broad mix of various managed hosting setups as well as directly owned colo configs, and purchasing bandwidth from various single-homed providers. This patchwork of services caused various issues over the years, though our IT team put up a valiant fight in each instance to keep things running as smoothly as possible in spite of a sometimes overly "complex" architecture. After I joined the company a few years ago, one of the long-term wish-list items was to redesign and consolidate all the network assets, especially since some of the original designs were not the most secure nor the most cost-effective (something that became apparent during multiple attempted attacks against Knight Online by Turkish and Chinese nationals trying to inject game-gold and during various DDoS attacks).

Oddly enough, taking on more of the back-end network ownership should actually simplify our lives, and also reduces our long-term cost of delivering high end game services, and improves our ability to pin-point service problems in various regions. APB just happened to pop-up while we were in the middle of this massive redesign. Not really a problem, but something that certainly makes the initial APB re-launch just a little slower than we had originally hoped for.

Owning our network

In late August 2010 we were approved as an AS (Autonomous System) by ARIN and technically also as an ISP (since one of our business lines is to host games for other companies). We then promptly started building out what will become the new network, and have very slowly started moving some test traffic onto the Los Angeles portion of the new configuration. As of this moment 80% of our stuff still runs on the old network, with about 20% of traffic being pushed out of our new network. By the end of February we expect to reverse that and hope to have 80% of traffic running on the new network, and APB would be one of those key items running on the new and improved system.

The new design consists of a peering edge in Los Angeles with more than 20 Gbps of connected capacity, a new peering location (and hosting location) in Washington D.C with multiple peers, and a new peering and hosting location in Frankfurt, Germany (and in the process we are decommissioning New York, Amsterdam, London and Vienna) and a stand alone node in Sao Paulo. Between each of our core hosting locations we are going to be running multiple redundant OC-48 paths, and in the process of doing so, we will also be shutting down the jumble of internet access providers we have used previously (or in some cases, upgrading how their services are delivered to us; from being a consumer of IP services to being a direct IP peer). OC-48 is telco speak for a private DWDM fiber connection that actually lets us cross connect our core datacenters across private fiber optic strands.

The biggest benefit for the player; presuming we are well-peered enough (see the below discussion) you should be able to reach the games using fewer hops from your current internet broadband provider than you would use today. Also as soon as your traffic reaches any one of our new key nodes (Washington, Los Angeles, Frankfurt or Sao Paulo) if for any reason you need to connect to one of our far-away nodes, then your traffic would be routed through our private DWDM fiber optic backbone between our core locations once those strands are lit up (ie - the traffic would be physically "off" the public internet and now literally be sent by direct fibers between our locations). In most cases this would be a big benefit, since our network literally would move the data at the speed of light between our private nodes without any need for a public router hop.

Right now with the limited state of the current setup you can see our initial peering report (courtesy of Hurricane Electric): http://bgp.he.net/AS36187 or in graphical form: http://bgp.he.net/AS36187#_graph4 . It shows that at the moment our not yet fully deployed network is multi-homed with two direct peers (Telia and PacketExchange / Mzima). We will have at least one additional direct peer in each location by the time we go live, in addition to tapping in to two public peering exchanges (which would give us a whole slew of peers).

So, what does all this mumbo-jumbo mean? Being multi-homed (with at least two, but preferably three or more peers/providers) means that if one of our upstream peers were to explode (or do something nutty with their routing), traffic will still be able reach us across one of the other upstream providers. It also allows us to perform some basic traffic shaping (for example, it might turn out that in a specific country we might have better performance using a certain provider, and we can then prefer a certain peer for some of the traffic).

The HE toolkit is actually a great little collection of webpages you can use to figure out what various companies (those 37,000 or so companies that as of today actually announce origin routes globally that is) actually look like on the BGP-networking end:

Nexon America http://bgp.he.net/AS26236
Activision Publishing http://bgp.he.net/AS40551
Facebook http://bgp.he.net/AS32934
Amazon http://bgp.he.net/AS16509
Netflix http://bgp.he.net/AS2906

One word of warning; to have any kind of fun with this, you must be a network geek, but since I figure that some of you reading this blog might be geeks, this could be an interesting introduction to BGPv4 peering for you. It's worth noting that Facebook (the worlds biggest site in terms of visitors) have 63 direct IPv4 peers, and Netflix (the worlds largest site in terms of traffic volume moved per day) has only 6 direct IPv4 peers (though they currently move most of their traffic through LimeLight according to the report, which clearly is extremely well peered on its own).

You'll quickly realize (using the search function), that a lot of game companies don't ever go through the trouble of building their own global directly owned network, and instead rely on various providers to give them the first internet service hop. That's a perfectly fine strategy, and one that we have used for a long time, but with the launch of APB, the launch of four other MMO's on our site in 2011, as well as the complete network redesign that we were undertaking to improve performance and security, it made perfect sense for us to go down this route at this time and become a global wholesale network provider (more or less). Yes - technically you could actually now buy bandwidth FROM us on three continents, though I don't think we really have a business model for that yet :)

All this exciting network stuff just means it will make the APB launch just a little more exciting as well, given the fun complexities involved in launching the game on a brand new network. Our critical IT team will certainly have a lot of exciting things going on in the next several days.

Sticking to our mission that we are planning to share a LOT of information with the community as we go along, hopefully this gives everyone a sense of how the future APB network will operate. As new parts of it go live (like DC and Germany), I will point those out in future posts, as we get closer to actual production time.

Next week marks the beginning of the detailed game client discussions, so stay tuned!

Cheers,