End of Closed Beta Week No. 1 - a recap

Wow - what a week!

So clearly I have to thank everyone in our various global teams for coming together in an amazing fashion to make this CB run possible. Very exciting. And a whole lot of work.

What's scary now is the staggering amount of work that still remains ahead for the rest of the year. But at least now we have proven that the game can be resurrected, and the next chapter is entirely going to be focused on improving the game progression, improving the servers and improving everything else that remotely touches the game in the long term. 

So first of all - some closed beta observations:

The NTEC is all-powerful.
Mid last week we had to de-power the shotgun that could blow up a car in (almost) one shot. This week we will have to deal with how to best handle the NTEC (and possibly even the STAR). So, the good news - the new NTEC surely is a good gun. The bad news, pretty much everyone uses the NTEC in game. That was not really the intent, and is a focus during this week's balancing exercises, which involves actually measuring the usage of all guns in the game.

This IS a CLOSED BETA.
It occurs to me (and the whole team), that some people testing the game are not really familiar with the whole idea behind participating in a closed beta, and instead think this is like playing a live game. It is not. It IS a closed beta test. I do need to point out that most players are doing just fine as CB testers and are diligently filling out reports of issues and bugs, but there are a few who clearly are NOT doing really that great.

The entire purpose of CB is to validate the basic game systems, work out all game balance issues, solve server lag, identify any performance problems, identify any unexpected crashes and identify serious game play or progression issues. Especially since right now we only have the Los Angeles location operating the game to test all routes to that one location. Let's be clear - there are SEVERAL of these issues in the game that we are working on right now. Therefore the biggest help for the team are well written bug reports of issues that you encountered in the game or details of any unexpected behaviors that you can document thoroughly. The least amount of help is pointing out that there is a lot of lag from the UK to Los Angeles. Yup - we know there is a lot of lag from the UK to Los Angeles. Instead we ask players focus on items that can actually help improve the game.

Server performance issues (and upgrades)
It's clear that the 100-person districts can only actually run properly on Intel X5570 processors and above. However, we initially were a bit baffled since the district threads (there can be 4 district threads running per processor) were behaving erratically last week and were randomly dropping the server-side tick rates dramatically (causing some server side lag), until we realized that the X-series processors were configured to run in a standard energy saving mode. That created a lot of random frame-lag on the server side whenever the Intel processors arbitrarily decided to reduce the clock rate. This has since been resolved, and this coming week we are working to further increase server tick-rate performance and aiming to eliminate server side lag as much as possible. One thing to keep in mind with APB Reloaded is that there can be as many as 100 players engaged in a single thread. For comparison, most other FPS games don't permit more than 16 v 16 or in some cases 32 v 32. In theory we could actually turn on 50 v 50 in APB in all-out war mode (which is essentially what Chaos mode is) and for future hardware upgrades go even higher in a single district and still maintain good FPS style performance.

The LATENCY tag in the /FPS in game switch
In the game you can run a command called /FPS and you will see both client, server and latency information. The latency information however turns out to be a bit out of wack. Normally you'd expect there to be a "ping" command (to tell you how far away the servers are), but latency actually creates a hybrid measurement, AND it has a couple of minor bugs in it to boot. Our developers looked at this over the weekend, and it turns out that it measures client frame rate (so a 30fps frame rate would give you 33.3ms for this portion) plus round trip (average of 60ms) plus server ticks (so at 15-20 ticks/s on the server that'd be 50ms). That means it's unlikely anyone will see anything less than 100-120ms. That would be ok I guess (though rather meaningless), except that on the return to the server for some reason the client sends it back as UDP traffic. This means that the packets basically can (and will) arrive in all kinds of strange arrangements (if at all) since UDP is designed to permit lossy behavior. When a packet then arrives out of order or disappears (as its permitted to do in UDP), then the bug is that "latency" starts counting upward until it maxes out at 1020ms - even though that's not actually what happened at the traffic level, since it might just have re-ordered one UDP packet along the way. So, the team is going to work on fixing "latency" so that it measures something more meaningful as soon as possible. The basic idea of an end-to-end processing measurement is a good one. But the current implementation just needs to be refined a bit.

If you are curious about PING times to the game, you can ping the core routers that were listed in the original application. Pinging www.gamersfirst.com will NOT work, since that site sits behind batches of DDoS protection, which WILL mess up ping times.

Pando Media Booster impacting network traffic?
We use a P2P program to get people the client faster given how big it is (and it's used by several other games, like DDO, LOTRO and others, just as Blizzard uses a P2P protocol for their downloader as well). This is also not too dissimilar to how Skype works (most people probably don't realize that all Skype traffic and calls travels across other people's computers to get to the destination, which is why those Skype calls are indeed free).

Normally we don't have a lot of issues with the Pando component, but there have been some suggestions that it might increase the LATENCY counter in the /FPS command (just as Skype or any Torrents might). We haven't seen this happening or being a big issue, but, we will look in to see if we can throttle it while the game is running to ensure no resources are used for anything except the game.

We use Pando so that end-users get much better download speed especially during giant "patch-storms" when new patches get released. Contrary to what some conspiracy theories suggest, it's actually not a relevant price difference when compared to hosting the patches on CDN servers of our own (which we also do) since Pando is something we pay for from our end, but it simply seems to work better for end users and makes the download and patching process incredibly resilient and nearly unbreakable.

This also leads to some interesting situations such as users in Romania getting their games 4X faster than US players, since most of Romanian players play in internet cafes (and since Pando enables patches to move P2P across a local LANs instead of the internet, that means as soon as a single user in the cafe has downloaded the first file, subsequent file requests mostly don't cross the public wire, and the same of course would also be true in case you used the program inside a corporate network or other LAN of any kind).

If you have any issues with the program you can either open the Control Panel that says Pando Media Booster (32-bit), and under "Advanced Connection Settings" you can configure the options the program uses (like limiting upload speeds). You can also use the instructions on Pando's site to disable it completely: http://www.pando.com/help-faq-mb  - however - keep in mind that chances are you will need it next time there is a giant patch simply because of the patch load that we tend to see when the patches get distributed.

Where do we go from here?
Wow. Lots of stuff coming up. First of all, we all like the basic premise in the Financial and Waterfront districts presuming we address some of the matchmaking and ranking issues (which is on tap next), but as we have mentioned before, we are also working in the long term to vastly expand the options players have to engage each other in. Everything from actual car-races in-game to all-out turf war missions and possibly entire districts and maps focused on those engagement styles.

Will we send out more invites this week?
Probably. So far we have invited 10,000 people, and are seeing a consistent 1,200 peak CCU during this very initial test phase, which is great. We are in the process of analyzing all the bug reports, Closed Beta feedback information, and a few scattered crash and compatibility reports we have seen to date. So as soon as we have rolled out a few more fixes to the game, we will consider starting up the next round of invites.

What's next this week?
This week is focused on performance optimization, game balance, some minor bug fixes, and the overall EU datacenter configuration. We have another 140,000 people to let in at some point, and we are continuing to work toward letting everyone in to CB in prior to the start of OB.

We also have a slew of new feature and items scheduled for Open Beta, which is why the Open Beta patch is going to be a VERY different beast (but of course incorporating all the balance and game play learning we have gained from the Closed Beta).

I plan on updating this blog mid-week. Our overall goal is to continue providing an enormous amount of information to everyone while we move the game toward OB, and eventual "Live" operation.

Cheers,