Tuesday, March 29, 2011

CB4 update (release 68)

Hi guys

So today we launched CB4 (build 68) of the game, in order to perform some tweaks before the next big release. I have outlined the release notes below. The key take-away is that we are making room for the new upgrade system, new progression and new rating system, and in the processes we have temporarily 'suspended' most of the current upgrades, so we can slot in the new system seamlessly.

Also, as a further update on the EU servers, we have some continuing bad news; we are still in waiting mode to get the equipment cleared, and at this pace we don't anticipate it will happen until next week. In order to stop this frustrating process we have also taken some drastic contingency measure and are shipping MORE equipment with a different carrier just to get the stuff (at this point ANY stuff) in to the country. If all goes well it will be there first part of next week.


CB4 ( Release Notes

  • Upgrades: All upgrades except Field Supplier, Happy Landings, Mobile Field Supplier, Blast Radius and Nitro temporarily suspended (set to 0% effectiveness) until we release the next big update with the new upgrade adjustments.
  • Bug Fixed: 'SuperCop' and 'Busted' achievements now complete at correct times.
  • Bug Fixed: Audio display points show correct description text.
  • Bug Fixed: “Just like Momma used to make” mission.
  • Kill requirement removed from 'capture and hold multiple checkpoint' missions.
  • Kill requirement removed from 'capture and hold item' missions.
  • Improved network latency in Social Districts (nearly doubled the frame-rate)
  • Dolton Fresno: Slightly raised rear end.
  • NFAS: Damage Reduced again, slightly reduced spread, increased ammo available.
  • OBIR: Reduced Horizontal recoil.
  • JG: Improved.
  • SHAW: Reduced Damage significantly. Increased accuracy, reduced horizontal spread, increased vertical spread.
  • NTEC: Increased Recoil. Reduced Recoil recovery.
  • OCA: Reduced Accuracy increase when crouching.
  • OPGL : Reduced Kill Radius.
  • Joker SR15 Carbine: Improved Fire Rate.
  • HVR 'Scout': Improved Damage (now a 2 shot kill, 'just'), reduced fire rate to be a fraction slower than the HVR.



Friday, March 25, 2011

The Missing Skill-Ratings, and a Thanks to Hackers

Ok - so time has come to lift the covers off one of the core problems in APB; the lack of meaningful ratings that underpin the "threat" and matchmaking systems.

There are various important core issues that the team is working on (such as spawning in your enemy's face in particular, which the team has been working on some really elegant solutions for), but the lack of real ratings is critical since it causes a ripple-effect of problems. Matchmaking now consist of a very smart and sophisticated matchmaking algorithm, paired with a very "basic" (and maybe even "dumb") rating algorithm.

We are fixing all this as one of the reasonably early Open Beta tasks (not the very first OB release, but in the second round of OB or so), and at this point one of the core feedback items from Closed Beta is - "that's some crappy matchmaking." We agree. Let me explain. 

No Real Skill-Rating
For a game that has some really cool and well engineered systems, it was a bit of a shocker to discover that in spite of relying almost exclusively on computer generated match making (unlike games like Counter-Strike where you make your own matches), no real thought was made to create a decent skill-rating system to feed those match-making decisions.

APB has a slew of different level and progression measurements visible to the player with titles like "threat", "rating" etc. This gets terribly confusing since "rating" as used in the APB UI actually means "progression" (or in MMO terms really "Level," not to be confused with APB's use of the word "Level"), and "Threat" is actually supposed to be a rough representation of "skill-rating" in more traditional game language.

Right now "Threat" - ie what 'should' be rating and what is the critical value used by the matchmaking system - is computed using a mechanic that tries to emulate Premiership League standings (or "soccer" to all the Americans reading this). A win gives you 3 points, a draw 1 point and a loss 0. Then APB takes that score over your last 50 matches, with a focus on the last 20 and figures out if you are above or below your opponents using this crude straight-win system. No regard is paid to how many matches you have played (is this match 52 or match 952?) and no regard for what level of player you have beat/lost to in the past. All wins and losses are treated equally. Even if you were to beat uneven odds (like 1 v 3).

Thus if you have 20 wins in a row, your "threat" (your skill-rating) balloons. Even if all you did was beat 20 newbies. And you will then be tossed into 1 v 3 or even 1 v 4 scenarios as you have a high threat level.  Killing newbies still shoots your "threat" level through the roof. If you had close fought losses against true veterans, those losses are still considered as bad as if you had lost to some really bad players.

Since there is no consideration for the skill of those you beat, the number of people you beat, nor the manner in which you beat them, you are not ever going to hone in on a TRUE skill rating in the current system, and it becomes easy to game the system. For anyone familiar with the XBox True Skill system the above system probably doesn't make a whole lot of sense. We agree.

Chess, Glicko and TrueSkill
The fundamental idea behind a good rating system is to let people slowly hone in on the rating they truly should be over a long period of time, and then try to predict matches that will provide the best experience to the players involved.

I have been involved in many different game companies in the past, and we always ended up going back to Chess ratings as the foundation for our rating algorithms ( ). People have been extending chess ratings for a long time, most recently Microsoft's implementation of the TrueSkill rating system, which is a rather complex system, but fundamentally is an expression of Chess ratings applied in online matches.

There are less computationally intensive rating systems such as Glicko ( ) which strike a balance between the "drifts" that tend to occur in pure Chess ratings and the complexities of the TrueSkill implementations.

So part of what we are trying out, will be a system that uses a LOT of the different characteristics of the above rating systems. It is still going to be a little while with the current wild Threat system (sure - go ahead and exploit it, lose 20 matches in a row, and then go kill newbies if you like since your threat will suddenly seem "low") - but clearly it's a top priority to get this worked out in Open Beta.

This leads to another requrement...

User Segmentation
In Most MMOs (like in Knight Online, another game that we publish) people tend to get grouped into distinctly different combat segments. In KOL the groupings are for levels 0-35, 35-60 and 60-83. The groupings divide players out based on the types of equipment they have and the amount of progression they have made in the game.

In APB this isn't strictly applicable given the hybrid nature of the Shooter/MMO and the greater emphasis on skill rather than progression, but at minimum "protecting" lower rated players from the being "hammered" turns out to be really important.

Therefore the next big thing is going to be creating some basic "skill-level districts." If a player doesn't 'volunteer' to join a higher skill-rated district, then he/she will be kept in the district with the most appropriate skill-ratings for his/her current skill-rating. We'll share more details on how this will be handled, as we get closer to releasing game changes that incorporate this.

Sure a lower level player can VOLUNTARILY join a higher rated district (maybe his/her friends are there), but the default action would be to put you in a district that most closely matches your current skill.

User Control Debate
One of the key items we keep debating internally is the amount of end-user control we should expose to the players in parallel with the Open World systems that will always be the core of the system.

In other words, how "counter-strike-like" we should make the start up of some matches. In Counter-Strike, since you can determine the exact nature of the team and your opponents, you actually are less dependent on a good rating system. So, one potential fix is to permit players the ability to queue-up and start their own fights (of course NOT removing anything from the current city or mission systems, just adding this ability in addition to the the mission systems).

Even though that might seem counter to some of the original APB ideas of launching things under the veneer of the living city, giving people direct game control would potentially let players set up matches that the system would never really promote on its own (who would NOT like 14 v 14 fights if they were available?) Our addition of Chaos rules and Turf Wars certainly adds some elements of session fights, and Chaos and Turf Wars are certainly our next big focus, but in addition to those enhancements, we are also debating if we should add direct match tools to the game. That way you can decide to play Open Missions (the current system), Chaos/Turf Districts or Player Matches.

We are happy to take feedback on this point, though I have a feeling that the true feedback would come in terms of a user trial (ie - we might put some of that in there and see where players go). The timing however, is obviously a little foggy, given the enormous amount of changes we are already working on and the huge backlog of game changes we are working on right now.

Finally - Thanks to Hackers
For the past 3 weeks we have been watching and observing user behavior in Closed Beta. We'd like to extend a "thanks" to the 60 odd players that have been toying around with various hack tools (about 0.4% of the players). Thanks to your hard cheating work, we are now much better equipped to deal with you going forward. How? I guess you will find out.


Quick Friday morning update

First - no - our critical routing equipment is still stuck in customs. Turns out that FedEx is really lousy at moving equipment. Great at moving papers. Lousy at actually getting stuff cleared (unlike true freight-forwarders, which we clearly should have used from the start - a lesson for us). But it's being worked out, and we are hoping to get this through in the next few days.

It's a bit sad since the servers are now all running at full speed in Europe, but the network setup is incomplete until we get the last batch of equipment in place. What we are now operating on won't give us sufficient bandwidth for the live game (we can run and test it from here, but not actually turn on live traffic).

I will provide an update on this Monday.

Also - later today I will make a full blog-post talking about some critical changes that are coming shortly to the game.

And I know that in a few hours our community guys will release something new on our Facebook page so check that out in the meantime.


Sunday, March 20, 2011

End of Closed Beta Week 3 - prepping for Open Beta

This will be a very short update.

Right now we have 25,000 testers in the game, and now also well over 250,000 people that have registered their interest in the game. We continue to see between 600 and 1,600 simultaneous players at different times of the day, which is great and a slightly better than expected number from the small population we have let in so far.

We are still performance testing and improving things during CB3 and working on some alternate configurations to further reduce server-frame-lag, but overall we now have a rather stable setup. We'll analyze this weekend's results on Monday, and then decide where we go from here.

The dev, design and art teams are still working on a few minor tweaks in CB, but are mostly now focused on the Open Beta build that's coming up.  Most importantly the OB build works out several core issues that we have to solve before going into OB, everything from how to recover previous character data, to how to test taking premium payments, to working on several progression issues.

The IT side is working hard to get the EU datacenter configured for the CB-3 build. Unfortunately we found out at the end of last week, that a pair of critical network components needed to finalize the build have gotten stuck in EU customs, so it's a little unclear when those will be cleared and delivered. The servers are all nearly set up, but now we need those additional Edge networking components before going live. We hope to get an update Monday from customs, and hopefully get our items back in the next few days. As soon as we do we will enable the EU location.

And yes, as soon as the EU launch is imminent, I will post here - and then we'll unleash at least 25,000 people on the EU datacenter as well (and anyone already playing CB in the US can then ALSO play on the EU location, just for the heck of it).

And NO, I know this waiting thing is KILLING you, and it's KILLING us too, so no point in repeating how much YOU deserve to be in the game. We agree. You DO deserve to be in the game.

And we WILL enable the game to everyone as soon as everything is ready. AND we still promise that all those who applied for the closed beta WILL get early access, ahead of Open Beta. And when the EU location goes live, that's when we will NEED all you additional testers to try it out.

Looking forward to seeing you in San Paro.

I will continue to post at least every 2-3 days to keep everyone updated, as well as daily on the message boards along with all the developers and the designers.


Tuesday, March 15, 2011

Doubling the community again - 10,000 more invites just sent

This morning we have confirmed the stability of the Los Angeles based servers. We have also confirmed that the server-side frame-rate now is holding steady at much higher levels than in the first part of CB, so we have now been able to bump the district population up by 20% in each district as well.

Based on this, we just sent out the invites to the next 10,000 eligible CB applicants. For those that were included in this wave but who were NOT from US-West locations, keep in mind that you are playing from a long distance and at this stage any lag you see should be network specific.

As I mentioned in the last post, the team in the EU are continuing the work on our datacenter locations, and as soon as that's up and running we will start inviting EU and Russian beta testers whole-sale into the EU location. Right now while most new invites went to North American players, quite a few long-distance players have been also included in this current wave, so just keep in mind the distance to your game server when you participate.

Keep in mind that when you have been enabled you also get automatic access to our Closed Beta forums where you will be able to post about the Closed Beta experience (when your G1 account has been enabled in the Cloaed Beta you will be able to see the Closed Beta Forum links here: )

See you in San Paro!


Sunday, March 13, 2011

End of Closed Beta Week 2 - CB3 around the corner, new invites will go out as soon as CB3 is live !

Closed Beta Build 3 Status
Unless our QA catches anything unexpected - we presume we will release Closed Beta build 3 (technically build 67) at the start of the week. As soon as that happens, we have queued up another 10,000 new invites to the updated beta to be sent out along with the release of CB3. That doubles the population from 10,000 now, to 20,000 as soon as we go live.

If everything goes as planned (and most of the performance lag spikes are confirmed as fixed in this build) then tentatively this MIGHT be the last major Closed Beta build except a few minor balancing fixes and possibly some progression updates, while the team now focuses on Open Beta and all the new components that are a part of the Open Beta (including the entire purchase sequence, which of course has a lot of interesting issues that we need to deal with and work out), as well as a lot of new functionality.

The European datacenter status
We have engineers continuing to work this week (just as they were last week) in Continental Europe to finalize all the server configs in the EU datacenter. As soon as that's up and running and all systems are "go" - we will then go live with the EU focused tests. The original plan called for March 15 for start of the EU test, but given the amount of work left after last week's heroic efforts by the team (generally until 3am every night), I am ready to bet its going to be a few days later than that. So a more conservative estimate brings that to March 22-23 or so start for the EU CB release and performance test. But if it CAN be done before that date - then we would of course proceed as quickly as possible and invite everyone as rapidly as we can.

In the meantime - below are the CB3 (build 67) patch notes. With these latest balance and performance updates, we are getting very close to prime time (ie the time when we START the very long process of testing out all the new game modes, districts, progressions modes etc that will come to define the game for the long haul)!

Til the next status update
/TechMech (Closed Beta 3) - Release Notes
  • Fixed server performance spikes in in death camera code (should remove most remaining server-lag spikes).
  • Removed client and server frame times from latency calculation to better estimate network latency and response (can be seen using /fps and /fpsdetail).
  • Added new HUD markers and text to distinguish between tainted and fresh targets (i.e. graffiti points that you need to guard vs. graffiti points that you have sprayed).
  • Altered HUD markers colours, and switched HUD markers to be more noticeable. Markers are now outlines when player has line of sight and solid when not.
  • Moved Checkpoints in Wilson LeBoyce spawn area to new Construction site out back.
  • Fixed Cement crucible having an incorrect texture in newly-added construction area behind Wilson Leboyce.
  • Fixed an incorrect reward package.


  • Cost of Muscle Car Increased.
  • Reduced Balkan Kolva Population in both districts down to 1% (Still need to remove a couple of spawners in future builds)
  • LTL Ammo cost reduced.
  • Explosive Ammo Cost Reduced.
  • Most weapons now come with more ammo on initial purchase.
  • STAR: Bringing the accuracy back up again, but not to the extent it was originally.
  • Colby PMG: Increased Base Accuracy, Fire rate slightly increased.

Thursday, March 10, 2011

Key performance improvement fix coming - or - don't die please :)

After various performance improvements over the last few days (now things are smooth at 80 players per district after all the CPU, speed, OS and core tuning that has taken place), there was still a persistent frame lag spike that seemed to crop up with some regular pattern, causing some rubber-banding on the client.

So Johann/Aphadon, using his great detective skills was finally able to figure out the issue behind some of the strange performance spikes (often seen as rubber-banding on the client when it happens).

Turns out there appears to be a bug in the new death camera. So whenever someone dies in the district and the camera swings toward the assailant, there is a chance the server will suffer a frame-blip/skip/jump during the "look-at-killer" process.

Or, one could say the server is sad for the loss, and will take a brief 100ms moment of silence :). Not a great thing since it drops everyone's performance district wide when it happens. The good news is that we presume it will be possible to fix this spike issue in the next day or two.

OR - we could just ask everyone kindly to not shoot their fellow players in the next 48 hours :). No death = no lag :)

That of course would potentially stop the game from actually being a game, so it could be a bit of a problem.

Once this issue has been nailed down by our fantastic dev team, then we are REASONABLY certain that we have taken care of all core performance issues, and then can start looking at expanding all the district to larger populations (next step: 100 people per district), AND adding more people to the Beta Test!

The team is also of course incredibly hard at work for the Open Beta features and changes, and those are critical to re-introduce logical progression and progress into the game again.

I will post another update tomorrow, and if there appears to be a reasonably quick fix (and rubber banding can be reduced) - then we will patch and observe the build in the wild to confirm this.

We also do see some performance spikes with Punkbuster as it monitors players, but at least those spikes are now within reason, and seems compatible with the overall game performance.

Have fun in San Paro!


Tuesday, March 8, 2011

Fixes, fixes and fixes

Performance Tuning - Part Deux

In the last 36 hours we have been working on performance tuning, new patch changes and a whole series of changes to the game that will improve server side performance over the next 24 hours. We just made one critical change to how cheat detection is handled, and that was rolled out in the last hour (yes, we are watching what people are doing, just not banning anyone, and that's on purpose...), and next up we are going to perform configuration changes tomorrow afternoon, which we believe will further improve district performance and get us to the stable release we are looking for.

Once we have the new tuned hardware in place tomorrow, we also plan to release the next set of patches. Below are the patch notes for the next patch. I will spend some time AFTER the next update to put together a more detailed status and roadmap for the next few weeks, especially since once we have tested all the core functions, we then plan to roll out all the NEW functions (yay!), starting in Closed Beta, and then moving in to Open Beta.

Again - thanks to everyone who is testing the game out daily. We REALLY appreciate the feedback, and will continue to make quick patches to address some of the basic items.

Personally I cannot wait until we have the NEW GAME MODES in the game, and after Closed Beta has been fully validated, THEN we are going to focus on that as the next really big project.


 1.4.3 (66) External Release Notes

  • Fixed the mission ‘Just Like Momma Used to Make’, where only one item would drop but two would be required.
  • Corrected Enforcer Bomb Defusal role (starting at 50 rather than 500).
  • Fixed minor typo with Display Points (was using 'Spray' rather than 'Activate', which is not valid in social).
  • Added a building site in the 'killbox' warehouse area behind Wilson Leboyce's spawn zone to add cover and interest to the area.
  • Fixed two district server crashes.
  • Improvements to the FPS display:
    • Moved the /fps and /fpsdetail displays down slightly so they don't render over the HUD.
    • Memory usage is now displayed in MB instead of bytes so it’s easier to read.
    • We now always render the memory usage in green on 64-bit systems.
  • Rewritten the LATENCY calculation code to use less bandwidth and give results closer to the real round-trip time of client-server network traffic.
  • Fixed Macchina Calabria 127 criminal variant showing enforcer front police lights as an option in the Garage.
  • Small fix to rewards to unlock LTL weapons at Ty rather than Grissom. Also tweaked it slightly so the primary comes from Ty and the secondary comes from Chung Hee.
  • Fix for Colby PMG-28 open slot versions not having any open slots.
  • NTEC: Slightly lower damage, worse base accuracy, worse minimum accuracy, worse movement accuracy, Reduced effective range to 50m
  • STAR: Reduced effective range to 50m. Slightly worse minimum accuracy. Slightly better running accuracy (to make up for the min accuracy loss, roughly the same overall)
  • Obeya: Worse movement accuracy.
  • NFAS: Slightly improved damage per pellet.
  • ALIG 762: Worse Damage against players. Increased base accuracy, slightly increased rate of fire (overall a positive effect)
  • OBIR: Slightly increased damage. Significantly reduced accuracy loss per shot, Worse movement accuracy (overall a positive effect)
  • Joker SR-15:  Significantly improved base accuracy.

Sunday, March 6, 2011

End of Closed Beta Week No. 1 - a recap

Wow - what a week!

So clearly I have to thank everyone in our various global teams for coming together in an amazing fashion to make this CB run possible. Very exciting. And a whole lot of work.

What's scary now is the staggering amount of work that still remains ahead for the rest of the year. But at least now we have proven that the game can be resurrected, and the next chapter is entirely going to be focused on improving the game progression, improving the servers and improving everything else that remotely touches the game in the long term. 

So first of all - some closed beta observations:

The NTEC is all-powerful.
Mid last week we had to de-power the shotgun that could blow up a car in (almost) one shot. This week we will have to deal with how to best handle the NTEC (and possibly even the STAR). So, the good news - the new NTEC surely is a good gun. The bad news, pretty much everyone uses the NTEC in game. That was not really the intent, and is a focus during this week's balancing exercises, which involves actually measuring the usage of all guns in the game.

It occurs to me (and the whole team), that some people testing the game are not really familiar with the whole idea behind participating in a closed beta, and instead think this is like playing a live game. It is not. It IS a closed beta test. I do need to point out that most players are doing just fine as CB testers and are diligently filling out reports of issues and bugs, but there are a few who clearly are NOT doing really that great.

The entire purpose of CB is to validate the basic game systems, work out all game balance issues, solve server lag, identify any performance problems, identify any unexpected crashes and identify serious game play or progression issues. Especially since right now we only have the Los Angeles location operating the game to test all routes to that one location. Let's be clear - there are SEVERAL of these issues in the game that we are working on right now. Therefore the biggest help for the team are well written bug reports of issues that you encountered in the game or details of any unexpected behaviors that you can document thoroughly. The least amount of help is pointing out that there is a lot of lag from the UK to Los Angeles. Yup - we know there is a lot of lag from the UK to Los Angeles. Instead we ask players focus on items that can actually help improve the game.

Server performance issues (and upgrades)
It's clear that the 100-person districts can only actually run properly on Intel X5570 processors and above. However, we initially were a bit baffled since the district threads (there can be 4 district threads running per processor) were behaving erratically last week and were randomly dropping the server-side tick rates dramatically (causing some server side lag), until we realized that the X-series processors were configured to run in a standard energy saving mode. That created a lot of random frame-lag on the server side whenever the Intel processors arbitrarily decided to reduce the clock rate. This has since been resolved, and this coming week we are working to further increase server tick-rate performance and aiming to eliminate server side lag as much as possible. One thing to keep in mind with APB Reloaded is that there can be as many as 100 players engaged in a single thread. For comparison, most other FPS games don't permit more than 16 v 16 or in some cases 32 v 32. In theory we could actually turn on 50 v 50 in APB in all-out war mode (which is essentially what Chaos mode is) and for future hardware upgrades go even higher in a single district and still maintain good FPS style performance.

The LATENCY tag in the /FPS in game switch
In the game you can run a command called /FPS and you will see both client, server and latency information. The latency information however turns out to be a bit out of wack. Normally you'd expect there to be a "ping" command (to tell you how far away the servers are), but latency actually creates a hybrid measurement, AND it has a couple of minor bugs in it to boot. Our developers looked at this over the weekend, and it turns out that it measures client frame rate (so a 30fps frame rate would give you 33.3ms for this portion) plus round trip (average of 60ms) plus server ticks (so at 15-20 ticks/s on the server that'd be 50ms). That means it's unlikely anyone will see anything less than 100-120ms. That would be ok I guess (though rather meaningless), except that on the return to the server for some reason the client sends it back as UDP traffic. This means that the packets basically can (and will) arrive in all kinds of strange arrangements (if at all) since UDP is designed to permit lossy behavior. When a packet then arrives out of order or disappears (as its permitted to do in UDP), then the bug is that "latency" starts counting upward until it maxes out at 1020ms - even though that's not actually what happened at the traffic level, since it might just have re-ordered one UDP packet along the way. So, the team is going to work on fixing "latency" so that it measures something more meaningful as soon as possible. The basic idea of an end-to-end processing measurement is a good one. But the current implementation just needs to be refined a bit.

If you are curious about PING times to the game, you can ping the core routers that were listed in the original application. Pinging will NOT work, since that site sits behind batches of DDoS protection, which WILL mess up ping times.

Pando Media Booster impacting network traffic?
We use a P2P program to get people the client faster given how big it is (and it's used by several other games, like DDO, LOTRO and others, just as Blizzard uses a P2P protocol for their downloader as well). This is also not too dissimilar to how Skype works (most people probably don't realize that all Skype traffic and calls travels across other people's computers to get to the destination, which is why those Skype calls are indeed free).

Normally we don't have a lot of issues with the Pando component, but there have been some suggestions that it might increase the LATENCY counter in the /FPS command (just as Skype or any Torrents might). We haven't seen this happening or being a big issue, but, we will look in to see if we can throttle it while the game is running to ensure no resources are used for anything except the game.

We use Pando so that end-users get much better download speed especially during giant "patch-storms" when new patches get released. Contrary to what some conspiracy theories suggest, it's actually not a relevant price difference when compared to hosting the patches on CDN servers of our own (which we also do) since Pando is something we pay for from our end, but it simply seems to work better for end users and makes the download and patching process incredibly resilient and nearly unbreakable.

This also leads to some interesting situations such as users in Romania getting their games 4X faster than US players, since most of Romanian players play in internet cafes (and since Pando enables patches to move P2P across a local LANs instead of the internet, that means as soon as a single user in the cafe has downloaded the first file, subsequent file requests mostly don't cross the public wire, and the same of course would also be true in case you used the program inside a corporate network or other LAN of any kind).

If you have any issues with the program you can either open the Control Panel that says Pando Media Booster (32-bit), and under "Advanced Connection Settings" you can configure the options the program uses (like limiting upload speeds). You can also use the instructions on Pando's site to disable it completely:  - however - keep in mind that chances are you will need it next time there is a giant patch simply because of the patch load that we tend to see when the patches get distributed.

Where do we go from here?
Wow. Lots of stuff coming up. First of all, we all like the basic premise in the Financial and Waterfront districts presuming we address some of the matchmaking and ranking issues (which is on tap next), but as we have mentioned before, we are also working in the long term to vastly expand the options players have to engage each other in. Everything from actual car-races in-game to all-out turf war missions and possibly entire districts and maps focused on those engagement styles.

Will we send out more invites this week?
Probably. So far we have invited 10,000 people, and are seeing a consistent 1,200 peak CCU during this very initial test phase, which is great. We are in the process of analyzing all the bug reports, Closed Beta feedback information, and a few scattered crash and compatibility reports we have seen to date. So as soon as we have rolled out a few more fixes to the game, we will consider starting up the next round of invites.

What's next this week?
This week is focused on performance optimization, game balance, some minor bug fixes, and the overall EU datacenter configuration. We have another 140,000 people to let in at some point, and we are continuing to work toward letting everyone in to CB in prior to the start of OB.

We also have a slew of new feature and items scheduled for Open Beta, which is why the Open Beta patch is going to be a VERY different beast (but of course incorporating all the balance and game play learning we have gained from the Closed Beta).

I plan on updating this blog mid-week. Our overall goal is to continue providing an enormous amount of information to everyone while we move the game toward OB, and eventual "Live" operation.


Friday, March 4, 2011

Double wave now going out. Another 5,000 people admitted to San Paro.

So two hours ago we enabled another 5,000 accounts to access the game and the closed beta and the notifications are on their way to the users. This time it was done on a first-completed-application basis.

We will let this test now run through the weekend to see how the game looks by Monday. Also we had scattered reports of slow download speeds from the CDN providers, so we have now also enabled the game to be downloaded from our normal GamersFirst LIVE installer that you can get off the main site. That installer uses a combination of CDN and P2P and will be significantly faster to complete the install (and it also lets you pause, resume and cancel the download).

So if you received a Beta Application Key at some point and you are now just waiting around for "your wave", feel free to use this tool to install your copy as well so you are ready to go when you account access is enabled. However - if you did NOT get a key we strongly recommend that you do NOT YET install the game, since Open Beta's installer will be structurally quite different (and would trigger a large update if you go from Closed to Open Beta).

See you in San Paro!

Thursday, March 3, 2011

CCU climbing, systems holding steady

This afternoon we permitted another 2500 people into the game. The CCU is now climbing steadily and approaching 1000 simultaneous users.

We performed more server configs throughout the day, which required a few restarts, but now we think we have nailed down a configuration which will in fact sustain the CCU without any noticeable server side lag (other than distance lag for those who play from a long distance to the game or on crappy connections from home).

Therefore - if this holds up overnight - tomorrow we plan to be bold, and are thinking we should DOUBLE the invites!

Right now we have 5,000 enabled accounts. Depending on the overnight performance indicators, we plan to enable ANOTHER 5,000 accounts by midday PST tomorrow. At this point we are just going by order of application and meetings certain minimum requirements, so wherever you are, you will have a chance of getting in.

Next week the goal is to work on the European data center (though it MAY not yet be able to handle the traffic for another 10 days or so, but we will update mid next week on the status).

Presuming the EU build-out goes according to plan, we would start the process of shifting people over to EU that "belong" in EU (since we are now permitting people to play in the US even from a long distance) once that datacenter is live sometime around March 15.

Next update will be in the morning!


First wave - next group!

In the next few hours we will send out invites to the next 2,500 people. In the last 24 hours we have been working on various server side performance items (turning off "power-saving mode" on some machines for example), various minor bug fixes and updates, and some new items we were able to slip in to the patch, as well as addressed a server side bug that would crash out the server when a friend left your list.

As soon as the patch is live, we will send out the next invite group. This time we picked the first 2,000 people who completed the application (and who are not yet in the game), and who HAD played the game before (first come first serve among previous players). Because that group was mostly US based (given the time of day the application was sent out), we added on 500 random non-US people across the planet to make it more even of a test.

We will update on the forums and the blog as soon as the "congratulations your G1 Account has been enabled" message goes out.

Here are the patch notes for the patch that is being released:

Patch Notes

·         Updated Financial to remove vehicle blocking volumes from the ‘Kunst Museum’ stairs.
·         Lowered contact level requirements for T1 contacts (The second set you unlock).
·         Removed floating sign in Waterfront Block 36.
·         Updated location names to be more specific in all districts.
·         Fixed typo in marketplace. Name being listed as Namef.
·         Fixing rating requirements so most new players get most tooltips.
·         Added new loading screen tooltips.


·         Greatly reduced the amount of time it takes to make a vehicle dirty (so a drive through the park will cake your vehicle in dirt), and also fairly reducing the time it takes for a car to get dusty (5 mins rather than 15)
·         Updated the customisations on the starter vehicles. Note that this will only affect new characters. Existing characters will still have the old starter vehicle customisation.


·         Marksmanship mode now defaults to hold rather than toggle.
·         Fixed PMG-28 doing too much hard damage (it was annihilating vehicles).
·         Weapon prices adjusted, a couple of higher tier weapons were wrong.
·         Increased recoil reduction for marksmanship mode on 50 Cal Sniper.

Wednesday, March 2, 2011

CB progress update

So after a night of closely monitoring the game (from the US and UK sides) things are looking quite good.

We have a few issues to deal with right away (and we will hopefully address these today):
The PMG-28 gun accidentally blows up any vehicle you point it at pretty much right away (ooops, I think Zak went on a wild design spreed and forgot to "normalize" the gun when he was done having wild fun :) ), Closed Beta Forum access has a bug (so CB testers cannot post there) and we have seen a world-server crash that happens every few hours that needs to be sorted out. Interesting though, the Action Districts all have stayed stable, and the issue is in the user management server.

So the plan is to patch the game shortly (this gives us a chance to actually test the patching process - so that will also be interesting), and then move forward as fast as we can.

We will evaluate this afternoon if we can send off the next 2,500 invites, or possibly hold those until tomorrow morning. In this next wave (once we release it) we will have a random collection of global players to test high ping times. One idea is to pull that in order that people actually filled in the application. We will also prioritize some people from the original run of the game for this next wave.

Beta status update - midnight

So - the very first "mini-wave" of 2500 invites went out six hours ago. Mostly we first targeted US and/or West Coast (but not ONLY West coast) players that met certain minimum requirements.

We instantly saw a spike on the CDN side, with one of our CDN providers presuming we were being DDoS-ed as people performed the update/patching process. But that part of the systems generally seem to have held up well.

As we cycle through various systems, we did see an issue in one of the "world servers," and we think it may have been a memory allocation issue, and it forced us to shut it down to reconfigure and restart it to see if it is quickly resolved, or if it will require major surgery in the morning (or if it's an previously un-identified bug)

Now - if overnight things seem reasonably stable - then we will increase the number of players DAILY. There has been speculation that we would wait a long time. That's not the case. We'd like to ramp up as quickly as possible.

In fact, we'd like to send the new wave of "account enabled" messages Wednesday again (of course unless something is wrong or buggy under load with any current core systems, in which case we will hold off).

The current plan is to slowly increase the CCU count (simultaneously connected players) from a few hundred (what it is now) to just over 2,000 simultaneous players this week (which means close to 20,000 actual invitees). If THAT holds up fine, then the next really big wave is to go to 100,000 players and 10,000 simultaneous - though that leap might take a bit longer.

In the meantime we are also processing data and monitoring information as quickly as we possibly can.

And - as a fun fact - we had a team in India (10,000 miles away from LA) just test the game, and it turns out it is SORT OF playable with 180ms-200ms ping times. Certainly not fantastic, but clearly playable. So - if we want to go really wild we COULD enable the game globally to reach LA, but at the moment that is not the plan (at least not yet).

If anything drastic happens overnight I will post here, otherwise the plan is to slowly turn up the heat until things croak (right now we are at a "1 of 10" and we're trying to get to "11 of 10" :) )