Player Explosion during Beta

(update May 30, 1:30pm PST; as a temporary measure, in order to reach the in game store to buy premium or character slots without first logging in to the game, you can access the store at the following location: http://www.gamersfirst.com/marketplace/apb/ingame_store.php . We will permit regular we access as long as the queue-ing remains in the game. Note:item purchases should be performed from inside the game.)
--------

Ok - so first off - I want to cautiously remind everyone that we launched APB Reloaded "Open Beta" only six (6!) days ago.

Open Beta means primarily that there will be no more character wipes, and while the game is getting very close to final production quality, and it's certainly already a blast to play APB Reloaded in it's current form, the game still needs some key changes, fixes and upgrades before it is fully production grade. There are code improvements to be released for matchmaking, more purchasable items to be completed for the item shop, and key upcoming infrastructure changes to be rolled out. In short - the game is still in Beta, and every week it will continue improving.

This is also why Premium accounts (ie upgraded accounts that accelerate game progression, unlock customizations, increase drop rates and gives players item purchase discounts) are only priced at $9.99 to start, and $7.99 for renewals. This rock bottom "beta pricing" includes a good 33%-50% discount compared to other MMOs.

Now don't get me wrong - while still a little rough around some edges (such as a few pesky sound bugs for some specific sound cards, some occasional district DC issues, and the fact that the large map size dislikes 32-bit Windows, but LOVES 64-bit Windows, as well as more cheat related changes still to be made, and the fact that we are working to add more things for people to do, game modes to use and districts to explore after they get really high level) - APB Reloaded even in Beta is STILL a really great game, with a ton of wild fun to be had in the city of San Paro.


Premium holders getting a leg up, because they are doing their part to help rebuild San Paro
Getting a premium account supports the development team, and also ensures updates are rolled out even faster since more support means we can invest more in speeding up the development effort. And - best of all - if you don't want to upgrade, you still don't have to. You can still play the game for free. But I do want to take a moment to THANK all the thousands who have already jumped on board and gotten premium accounts! Your support is invaluable. And therefore, we are of course going to let you "cut in line" when things get into a capacity squeeze.

Having a huge volume of free players means that free players will occasionally have to queue for slots at peak times, since it's important we provide priority access to paying players. Even so, our explicit long-term strategy is actually to make sure that everyone (include all free2play players AND all paid players) get basic game access without major game queues, especially when the game goes "Live" after the current Beta phase is over.

But during the massive growth phase we are in right now, or during large upcoming game changes, priority certainly always goes to Premium members, and others will be placed in queues only when network conditions require it, to make sure the game experience is not destroyed for everyone else trying to play the game.


Records set during the 6 days since start of Open Beta, and a blog post "Stat Scoop"
The purpose of the Open Beta is to test the limits of all the equipment and systems. On that score, the Reloaded Open Beta has been a smashing success so far :) !

The good news is that today we again set new records for beta players in the game, and the bad news is that during the beta phase there are some limits on network capacity, with increased capacity originally scheduled to be ramped up over time (which of course - given the response for the game, will now be sped up - a lot).

To give you some inside stats (we can call this the exclusive blog post "stat scoop") even though I will not expect to normally share many (if any) realtime stats. But given the week we have had I will certainly share some of that; at peak today, after only six (6!) days in Open Beta, we had just under 24,000 peak concurrent players connecting to the game globally (aka "Peak CCU") with a somewhat larger number of players logging in to the game in Europe than in the US, with more than four times that number of unique players logging in to the game during the last 24 hours, each player with an average playtime well over 3 hours per game session (average!). This load also meant the system automatically fired up over 200 district server instances around the globe to handle the load from the gamers.

To put that in some perspective with some of the great games out there (some F2P, some subscription); Eve Online announced back in 2007 that they had reached 35,000 peak CCU, which was 4 years after the game was released. World of Tanks announced in April this year that their PCCU reached 25,000 after a year of Beta testing. Nexon announced that Combat Arms beta had reached 12,000 PCCU in Europe in their open beta running for 10 months from December 2008 to September 2009.

So for us to get to 24,000 PCCU in six days (and WELL over 12,000 PCCU in Europe) is certainly a very good start, especially when you consider its still only Beta, and it has not yet been a week :) (and we have now also capped our servers to boot).

That also means the game is consuming A LOT of servers and network capacity to handle the growth, since unlike games that are super-low-server-impact-per-player (FarmVille, Kingdoms of Camelot etc), APB Reloaded consumes an insane amount of processing resources to keep track of 100 players, 250 cars and over 1000 NPCs real-time per district, and performing a large-scale physics simulated district at 30 frames per second server side. To continue the comparison with Eve, the sever side frame rate of APB is an order of magnitude higher than the server side framerate of Eve. Granted, it's not a fair comparison (Eve is a viscous liquid equation system with click to move mechanics that update rather infrequently, and APB is a frame-rate dependent shooter, so both are great games - just of course doing it very differently, even though CCP's upcoming games might change that). But you guys get the point; there are a LOT of sensitive computations going on in this game at a rather massive scale in our datacenters, which also makes the game somewhat "fragile" to large network issues.

Game Signups and Social Media Growth
The game has also settled into a steady growth pace by adding an average of nearly 1,000 new signed-up game accounts PER HOUR for the past several days, and our Facebook page is getting an average of 6,000 new likes PER DAY (which is great, since we now are using Facebook to post all key game status alerts, which gets info out much faster about game status than just using email, forums and other old skool formats). The page has grown from 48,000 on May 17 to 120,000 on May 29. In fact our facebook page has now surpassed the pages of other great games out there, or is heading toward the same like counts as other games that we all admire. Clearly that increases the pressure many-fold on our teams to live up to the qualities of the games that we are now going to be compared with.

Growth Strategy and Issues
The strategy on the GamersFirst side has been to roll out the game with a large but reasonable amount of investment to avoid starting the project from a big financial hole during the redevelopment phase (since this game certainly had a VERY bad history of not doing too well previously), in particular because this game has massive server side hardware requirements, far beyond most normal MMOs, with several hundred servers to execute the district logic for this volume of players.

But interestingly enough, we actually have plenty of server capacity to spare on hand. In fact right now we have sufficient server capacity on standby that we could power up that will give us 40,000 Peak CCU capacity if not more for the game.

Instead the issues we have seen for the last 6 days in Europe have all been firewall related. Even though the best firewall sales people on the planet will tell you that THEIR firewalls (Ciscos, Junipers, you name them) can handle XX Gbps of traffic, as soon as we throw millions of packets per second of encrypted UDP at any firewall, chances are that the "specs" go out the window. This resulted in the theoretical 7Gbps firewall pair we had in the EU croaking at 737Mb/sec (or less than 15% of the "theoretical" limit) and at much fewer pps than ever "rated."

So, alternatively, we COULD run the servers without any firewalls (that would actually solve the packet loss and firewall congestion issue instantly (!) ). But that would be religiously a pretty bad move. And certainly other game companies have shown that would be a rather bad idea. But, that's often the way this type of issue gets solved (throw the servers out on the net without firewalls). The network performance would surely go up. Be so would the risks to the game and the players.

Therefore, what we are doing instead is a measured increase over time, and also rolling out various changes to our core systems that will permit us to slowly increase all the server caps.

Also, we are cognizant that players can come and go, so we are going to continue performing somewhat deliberate growth during the beta phase and plan to pull out all the stops when all final changes, updates, fixes and features are fully in place.

See you on the streets of San Paro!

Cheers,
Bjorn / TechMech

-----------------

(updated: Sun May 29 11:59pm to add):

Sidenote to clear up minor mis-information on the interwebs?

One of our forum members pointed out a video segment by some random reviewer that got a lot of things wrong about APB, and to top it off referred to Gamersfirst as an Asian company "doing mostly things in the Asian markets." Let me quickly clear that up; Gamersfirst was created in California originally under the name K2 Network, is based in Irvine, and its largest investor is Intel Capital. While we have global operations all over the planet, we only publish games in the regions of North America, South America, Europe and the Middle East. So while that maybe "touches" Asia a little bit, we are actually an entirely a western market focused company and currently do not publish any titles in East Asia. Hope that clears it up (if anyone were to ask :) ). Also - all this information is easily google-able too (for example google: k2 network investment), but I think that would be too much trouble for some people.

Temporary player limits and game queues placed on European APB worlds during network maintenance

A quick update, we are preparing to perform network maintenance in Europe this evening EU time to deal with network packet loss, and while we prepare for those changes we have enabled Queue-ing on the European servers in order to improve network performance for those that are already in the game and playing it. It will take about one hour right now until the player levels have dropped to the point where the network performance should return to normal.

After the network maintenance is finished late tonight, at that point we will remove the limits again. For the future if we ever have to use queuing and limits again, then we will let Premium players bypass them (that is not the case at the moment unfortunately, but will be in place as of tomorrow).

So apologies for all EU players placed in a queue at the moment while we rapidly work to expand network capacity to handle the massive influx of new players.

Cheers,

Patch 88 success, EU network packet loss spikes last night, Premium Re-Ups for Only $7.99 and New items on sale shortly

Today we continue the "blog of full disclosure"

Patch 88 release successful

Last night PST / morning EU time we released patch 88 which performed a lot of minor tweaks to the game. The patch launch went well, and we saw quite an uptick of players after the patch. In fact - we ended up setting a record in EU today with tens of thousands of players logging in to Open Beta simultaneously from Europe alone. The game has been growing at a rate of 50% per day globally for the past 5 days, though we expect that pace of growth to taper off rather soon.

Packet Loss on Europe 1 & 2 networking equipment last night

That giant Open Beta turnout wave caused some unexpected network behavior during EU peak times yesterday. The servers were unaffected (since they scale horizontally and new districts will just start firing up), but we did run in to an unexpected problem with our main network edge devices that caused sudden severe packet loss conditions in Europe, even though the devices were in theory running at only 10% of capacity. 

The result was that between 16:00 and 22:00 CEST on the networking gear connected to Europe-1 and 2 (ie Patriot and Obeya) we saw several sporadic but big spikes of up to 40% packet loss during those hours. 

Inside the game those packet loss episodes manifested themselves as severe rubber-banding, even though ping time and server load would actually remain very low (however, you would be able to see the packet loss conditions if you happened to use the test ping IP we provided on the Forums).

We are working closely with the manufacturer of the networking gear overnight to see if they can resolve the issue before tomorrow's peak time. They are potentially able to provide a fix and if it can be confirmed, I will post the information here as an update to this post.

US-East and US-West did not experience the same issue, partly because those locations have of course been split between two different physical datacenters and in effect have double the amount of networking gear infront of the game servers at each location.

Again, we hope this issue will be resolved very swiftly, but keep in mind this type of testing is what Open Beta is all about - to test the crap out of all the equipment at full throttle. So, while we apologize for the un-fun experience in Europe during that time window, we hope people continue logging in and playing since that will help us troubleshoot the problem in much more detail tomorrow night.

New Items on Sale Tomorrow

This week we are going to start posting a lot of cool new items in the store, and we are going to start introducing some permanent items as well on a rolling schedule. We will announce the new items on Facebook tomorrow so stay tuned (or simply log in to the Armas Marketplace when we launch the items).

Renewal of Premiums during Beta - only $7.99 USD (£4.99 GBP / €5.99 EUR) !

So in the next two days our first batch of Premium Players are going to get reminder emails that their first 30 days are almost up. As we have said before - during the Beta period the cost of premium is only $9.99 for all of the cool premium benefits that let you vastly accelerate your progression and item drops, and gives you complete artistic freedom when you design your characters plus a whole range of other benefits including a 20% item purchase discount. 

Thanks to this discount when Premium Players re-up (adds 30 additional premium days) before the end of his/her premium period, those premium extensions get an automatic 20% off (!) off the Beta $9.99 price. 

The result is that premium renewals are only $7.99 in the US, £4.99 in the UK and €5.99 in Europe. And - any benefits that you accrue, including standing, achievements, customizations, permanent items or permanent weapons you buy during Open Beta carry over to the live production launch of the game as well.

And finally - a big thanks for Open Beta testing our game!

And a final word to everyone trying the game out; a really big THANKS for Open Beta testing our game (and liking us on Facebook!). 

While we will try to keep the game as near production quality as possible at all times during OB, there will of course be some hiccups along the way, and that's expected at this stage of Beta. But we are working around the clock (litterally) to ensure the best possible "near production" experience along the way. In the next days and weeks we will continue performing network updates, traffic improvements, game tweaks etc., while also working on the major upcoming changes that we expect to launch before the big formal release of the game!

Happy Testing!

More Vivox servers and patch 88 tonight Maintenance at 12am PST / 3am EST / 08:00 GMT / 09:00 CEST

Two Wild Days of Open Beta Fun!

Open Beta so far has gone extremely well in most areas, and now we are heading in to the "cleanup phase" for several parts of the game, including post beta optimizations, tweaks and preparation for new features and items.

In a very short time we saw a huge number of players in game to the point where we accidentally overloaded our voice over IP systems earlier today (which is part of tonight's maintenance).

We have also seen our Facebook group rapidly reach 80,000 likes (now beating out other great games such as Star Trek Online, Champions Online, Runes of Magic, Dungeons and Dragons Online and World of Tanks), while setting our sights next on Eve Online (104,000 likes)!

Of course Facebook likes are vanity items and not great indicators of good games (after all Charlie Sheen has 1.3 million likes, not sure he is a "good person"), however for us Facebook is clearly another important channel we use as a communication tool with our rapidly growing community (and you guys of course know how much we DO value our ongoing communication with players!).

So a big Thank You for all the FB likes (!), and please do continue telling your friends about our Facebook Page, and if they have not yet heard of APB then certainly tell them everything you know about APB and San Paro as well!


Maintenance tonight

Tonight at midnight PST (aka early am in Europe) we are performing a quick maintenance to continue rolling out tweaks (patch 1.5.1.88), that will further improve the game.

First - we will be adding more Vivox (Voice over IP) servers tonight on our end, since the incredibly high load and demand for the game had us running short on voice channels today. After the additional systems are added, we should see fewer problems with people unable to launch team based voice chat due to unavailable voice connections.

Second - the new threat levels and matchmaking systems have started working out much better game in the last 24 hours. Still a few more tweaks to go, but now we can see clear behavior trends in the data, and what appears to be some really great matches.

Specifically - Patch 88 has the following changes:

Version 1.5.1.88

Threat Level and Matchmaking Updates

• Fixed an exploit where a leader could gain large amounts of Threat by kicking their team just before winning the mission.
• Fixed a bug where a kicked player could still have their Threat updated as if they had won.
• Fixed the Max Threat Player Notification referring to a ‘Threat Level 30’ player instead of ‘Gold Level 10’.
• Adjusted Matchmaking formula to better fit the new Threat system.
• Added a new ‘Threat Training’ system:
     o    New characters will see their Threat level as ‘In Training’, with a unique Threat icon for up to the first ten opposed missions.
     o    This gives the Threat system some time to home in on a characters skill level before displaying it on the UI and to other players.

Going Forward

Over the next few weeks we will continue tweaking and optimizing the game, and among other things we will be adding more network providers to give people more optimized network routes to reach our servers, and also laying the foundation for the next big wave of core game additions that will give people many more choices in the types of gameplay and engagements they can pick in APB. But we will share more about that after the post Beta cleanup.

And - before I go - we DO hate cheaters (when they are in fact able to bypass things for short periods of time). As I responded in one of the earlier Blog Responses; after this first wave of OB-launch-craziness subsides this week, we are going to start a much more aggressive campaign of public naming-and-shaming of cheaters when we catch them. 

The exact community policing methods are still under discussion, but we CLEARLY want people to realize that they are playing with fire when they are drawn to the "dark side of the force." In fact, we will extend the system to friends of hackers, where if your friend gets caught cheating, you will instantly be a suspect too. The motto is "Friends don't let friends cheat in APB" - and we will work on new methods of Community Policing to deter cheaters in conjunction with the "normal" technical policing we perform all the time. 

As hackers attempt new cheats that work for limited periods of time (and idiots pay the hackers for the cheat - which basically lets hackers install a keylogger on each machine running the cheat), we will eventually detect it (we always do), and at that point we will invalidate everything you have achieved in the game, and eventually we will also suspect anything any of your friends have also done or achieved in game.

Nueume will shortly share more information about the upcoming City Council program and other exciting community events in the near future, and as all these initiatives get underway, we look forward to growing a great San Paro community that can thrive and prosper as a self-governing city in cyber space...

Threat Level update, and matchmaking update at 2:30pm PST / 5:30pm EST / 22:30 GMT / 23:30 CEST

In Saturday's post we noted that the visual threat levels were off given the current curve. We also discovered another critical issue with low-level matchmaking that will be fixed with this current patch as well (fewer wild 1 v 4 matches).

One of the temporary side effects of the change though is that new players will appear to be Silver 2 during their first match before moving around properly. In fact, what SHOULD happen is that for the first few matches the player has "NO" rating, until we have some idea of what direction he/she is moving.

We probably are likely to perform yet another update later this week as we continue tweaking the system.

Today we are patching the game at 2:30pm PST / 5:30pm EST / 22:30 GMT and 23:30 CEST, and this would take about 1 hour.

----- (update) -----
The servers went live again 4:15pm PST after a slight security patch delay. Enjoy the game!

Visual adjustment to Threat Levels coming on Monday

The skill rating system is actually working very well at the moment, HOWEVER, we have realized that the end-user display of the data (the Threat Level Symbol - like "Bronze 1") are out of alignment, in particular since they visually show a very large number of players as Threat 1 for much too long while the system is learning the player's skill. In fact we will start changing the TL symbol much earlier in the process.

It's important to know that a player's REAL skill rating are going to be unaffected by this upcoming change, and this just alters the translation from the real system wide skill rating to the Threat Level display since the current translation skews too low to be practical (in other words, right now there is a VAST difference  between people at the very start of TL-1/Bronze-1 and at the very end of TL-1/Bronze-1. In fact the difference between start of TL-1 and end of TL-1 is greater than the distance between TL-3 and TL-11, which clearly is not an appropriate alignment to communicate how "dangerous" or "skilled" a particular player is).

In the higher threat levels the alignments are a bit more appropriate right now, and therefore will only be slightly adjusted by the upcoming change. And yes - we already have a few players (well under 0.5% at the moment) who have reached TL-30 (aka Gold 10). And they seem like the correct players in that category.

When the new threat level visual chart gets implemented Monday, everyone should see drastic jumps in their visual threat levels at that time (though - again - the underlying skill rating stays the same).

The other tweak we are working on a little longer term is modifying the matchmaking system. Because the new skill rating system produces numbers of a whole different magnitude (and accuracy), with the increased player volume they also don't translate very well into the original match making system. It works, it's just not terribly optimized for the new data, and seems to produce too many 4v3 and 2v3 and 2v4 matches at the moment for lower level players, leading to large backup call sequences. Interestingly, the skill rating gets accurately computed in the end no matter who was in the match (ie - if the match was uneven against you, but you still won - then you will clearly gain a lot from the match, in fact, losing a bad match under some conditions can make you move up in skill rating).

Therefore the other upcoming priority is to tweak how the matches are set up, especially now that we have far more information about a player's REAL skill level than we ever had before. Which is a very nice basis to start this change from. Based on this information we will be able to tweak the matchmaking to create better matches in a district at both low and higher levels.

And, there has been a lot of discussion about how we compute Skill Ratings. Without digging in to extreme detail, the basic paradigm of the new system is that at the end of a match the system takes all the ratings of the various team members, then determines a particular person's "participation" quotient in the match (if you don't participate much, you don't get much of the effect), and then the system updates players ratings in accordance whether they were EXPECTED to have wont the match or not. If one group of players were expected to have won, but still lost, then the penalty is much greater than the award they get if they were expected to have won and did win. If they were expected to lose, but still won, then they are going to get a much higher boost to their skill rating.

We will make a future dedicated post to skill ratings, and also start posting some of the data from the real live games so people can get a sense of how these matches play out and how players are shifting in skill ratings over time.

Finally - today for the past several hours we have had well over 250 registrations PER MINUTE for the game, and we expect all these new players to arrive in full force on Monday (except of course for the super-double-secret-early-invites we are doing before Monday morning - but those are of course a big secret). But Monday is also the day we will be formally reaching out to new players, so this coming week certainly should be very interesting.

Cheers!

Open Beta Set for May 23 6:00am PST / 9:00am EST / 14:00 UK / 15:00 CEST

The last patch fixed the annoying disconnect issue we were dealing with, so now version 1.5.1.85 will in fact be the formal Open Beta. Since it is STILL a Beta game, keep in mind we will have various known issues, but at this point we are confident the game will hold up technically while we work on the series of game items we need to improve on and polish (and part of Beta is of course to get those kinks worked out).

We very much look forward to welcoming the Open Beta crowd to the game on Monday.

After we go Open Beta, we will turn our focus to further game and feature polish, as well as inserting some of the new features and functions during Open Beta, all of this leading up to the formal launch, which is likely to take place sometime in the next few months.

The official release is now set for 6:00am PST / 9:00am EST / 14:00 UK / 15:00 CEST on Monday May 23.

Today (Friday) we are having some major site maintenance starting shortly, so everything should be set over the weekend for the Monday launch!

(edited to add: even though I will not promise anything - there is a chance we will have some super-double-secret number of accounts enabled to enter the Closed Beta during both Saturday and Sunday leading up to the beta launch, which "might" permit access to the game for people who had previously registered an interest in the game, so downloading the game early and trying it out probably will not hurt anyone. If you get in - good for you :) ).

Open Beta Release On Hold until next weekRelease 1.5.1.85 maintenance today 4pm PST

Here is Thursday's update - and the aim here is clearly to be crazy-transparent with what we are doing, so for those not that interested in the technical stuff, the short version is; Open Beta is on hold until next week. For Closed Beta users we are going to roll out patch 1.5.1.85 at 4pm PST / 7pm EST / 24:00 GMT / 01:00 CEST today

For the longer version continue reading below;

The Closed Beta issue we have run in to causes the disconnection of all players on a server. Unfortunately it happens to be of those vexing problems that only shows up at full production load, and an issue that leaves almost no traces of its origin in our logs. During QA and testing loads the issue doesn't materialize (throw 50 people on a server and everything seems just fine), even with large static loads and testing the game from multiple locations.

Specifically the issue manifests itself where districts with 60-100 players suddenly lose IPC (inter-process communication) functionality which then leads to the district dropping all players and the district crashing. This happens anywhere from 1-6 hours from the district starting up. Normally this would be easily grabbed by our crash processor and it's usually a pretty straight forward to identify the location of any GPF, however with this particular issue, the logs don't show anything that helps narrow down the scope other than it's part of the changes for Open Beta.

Rolling back several versions is certainly not the preferred option as it puts us firmly back into Closed Beta, but could be done (it would take us back to somewhere in the 1.5.0.7x range), so instead we are making one last push to get through the current issue with the production load, and we continue to ask our Closed Beta testers to help hammer the servers over the weekend so we can reevaluate where we are on Monday.

Today we are also pushing ahead with release of 1.5.1.85 which further adds information to what is happening, fixes an unrelated client-side GPF that happened when a song changed tracks (a general Audio GPF), and potentially MIGHT solve the district disconnect issue (but again, won't be visible until it's on full production load for a few hours).

Once we enter Open Beta, the patching schedule becomes very different (and slower). We will also at that point recruit our first "Test Monkey Crew" for APB - basically volunteers who will get early access to builds in return for running live tests on a public test world, specifically designed to catch these types of hard-to-QA issues. The TMC gets various in-game benefits for being early testers. But again - that system goes in to place after Open Beta goes live and Closed Beta has ended.

I will continue updating this blog daily until the the game is in formal Open Beta. Also while this is being worked on, be mindful that very few of our tech staff will be active on the boards or blogs, since this is pretty much an all-consuming issue for everyone.

Open Beta on temporary hold while we evaluate a player disconnection issue

Hi guys

We have come across a new player disconnection issue that cropped up in the last 48 hours that has made us put a temporary hold on the launch of Open Beta until further notice. We are releasing a new Closed Beta patch today (patch 84) to further investigate the issue and will evaluate as soon as that patch is live this afternoon PST.

The issue manifests itself in disconnecting some groups of players after one to two hours of gameplay, which clearly is not acceptable in Open Beta.

In the meantime we are looking to our devoted community of Closed Beta testers to continue playing the game so we can continue recording data related to the issue and track down the root cause of this particular service-affecting problem.

If we determine after the next patch that the issue has been resolved, we will then proceed to release the game to the public.

As soon as we have more data we will update the community on Facebook, Blogs and through Twitter.

We are clearly disappointed that this afternoon's public launch has been put temporarily on hold, but in the end we strongly believe that first addressing this issue is the most critical and important step we can take as a development team.

As we have done throughout the entire re-launch we will continue being extremely communicative with the community, and will share any status updates as soon as they become available.

On a positive note, we are also in the middle of preparing a thank you “goodie-pack” of game items to all our Closed Beta players for their efforts over the past two months, and we will release that pack the week after Open Beta is released to the general public.

Sincerely,
Bjorn / TechMech

New patch (83) today

Just a quick note; there will be another maintenance today to release a fix to some minor issues in patch 82 that came up in the live environment. In particular this patch will address an issue where people under some conditions receive "negative" ratings which triggers unpredictable matches such as repeated 1 v 4 matches.

One of the side effects of setting ALL players (tens of thousands of players) to 0 simultaneously has been that the matchmaking then has tried to adjust people in some cases to unexpected values.

There is also a district disconnection issue experienced by groups of players at the moment, which will also be addressed shortly.

Cheers,

US East being set up and release of 1.5.0.82

US East is being launched today - and patch "82" just went live on the servers. Enjoy!

We will post more info as we steam toward the OB, and alert the gamers of any issues if they arise!

Cheers
/TechMech

Game Update 1.5.1 (82)

Here are the patch notes to the latest version 1.5.1 (82) update hot off the press. 

Matchmaking Update Part 1

• APB Reloaded now uses a Bayesian enhanced version of the Glicko rating system to determine the relative skill of players, based on the outcomes of opposed missions. The believed skill of the player is known as that player’s Skill Rating, which is then translated into the player’s Threat Level (shown on the UI), and also used by the matchmaking system to construct matches where the skill of each side is as evenly matched as possible.
• The total number of Threat Levels per faction has been increased to 30 and spread across 3 tiers – Bronze, Silver and Gold – with each tier consisting of 10 ranks. For example, a ‘Silver Rank 5’ player is believed to be better than a ‘Bronze Rank 10’ player.
  
• All players will start from Bronze Rank 1, as the new system has to start from scratch in order correctly calculate the Threat level.
• Note that we expect players to see large fluctuations to their Threat early on, as the system narrows in on your actual Threat level over time. So don’t be surprised to jump between tiers as well as ranks for the first few matches.

Witnessing and Open World

• Fixed an issue when players went out of bounds on a witness mission. The mission now ends immediately and the stash is lost.
• Fixed Enforcers on opposed missions sometimes seeing the ‘witness’ icon on criminals.
• Fixed Enforcers in vehicles not seeing the stash value on criminals.
• Fixed the heat multiplier not updating while in a vehicle.
• Fixed an exploit where players were abusing the ‘AFK’ state.

General

• Updated DXT settings to match the following rules:
  
  • 64-bit OS users will no longer have any character or vehicle textures compressed at any time.
  • 32-bit OS users will not have their OWN character and vehicle textures compressed at any time, but all opponents’ character and vehicle textures will be compressed at all times due to 32-bit memory restrictions.
  • These are not to be confused normal ‘Level of Detail’ (LOD) shifts as well as standard texture streaming that occurs when customisations are being built – these have not changed.
• Fixed a separate issue that could cause your own character symbols to have lower detail based on distance and relative screen space (affected small characters more).
• Added 1280x720 as an officially supported resolution. Also resized the in-game browser to 1024x720 for all resolutions.
• Updated inventory and kill icons to match the weapon’s silhouette where this wasn’t the case.
• Updated Contact icons to match some updates to the Contact models.
• Fixed several issues where the expiry time tooltip could display wrong values about the time remaining on leased items. Note this was an interface issue and did not affect the actual expiration of any leased items.
• The ‘auto-AFK’ timer now kicks in at 2 minutes, instead of 5.
• Fixed a parking symbol that was floating above the ground (diagonally opposite LeBoyce spawn point).
• Fixed an out of bounds error near the Church in Financial that some used as an exploit.
• Added Pallet stack to roof under Freeway in Financial to simplify an irksome route onto the freeway and to make it possible at all for those with Kevlar implants.
• Added new double ramp so that the troublesome one-way jump near the ruined warehouse is now possible in both directions, no matter what the vehicle.
• Added new “use at own risk” ramp behind ‘Planetaget’ building on the turnpike. Best jumped in a Cisco. Best avoided in a starter car.
• Fixed wall decal bug where decal was bleeding across floor outside Stu Phoenix’s spawn zone.

Modifiers and Weapons

• Modifier Changes:
  
  • Hunting Sight and Reflex Sight: Now Increase or Decrease Movement Accuracy and Marksmanship accuracy.
  • Slightly lowered fire rate increase on Cooling Jacket.
• Decreased effective range on both SMGs and reduced their minimum damage at 100m to 0 (so the ramp off is more drastic). Also lowered their marksmanship modifiers slightly.
• Upping ammo counts of Obeya CR762 and OBIR to match other weapons (5 clips rather than 4/3).
• Replaced the base STAR with a new model to differentiate it from the leased STAR ‘LCR’.
• Slightly increased maximum reticule bloom when firing STAR full auto.

Vehicles

• Adjusting handling for Dolton Broadwing and Sungnyumun Morai.
  
  • The Broadwing has slightly improved handling, better acceleration, more health and weight and heavily increased ramming damage. It's a good car for intercepting and taking others off the road.
  • The Morai has been tweaked considerably. It’s now much quicker to respond, with a fairly loose rear end allowing for some advanced manoeuvres. Acceleration is up heavily, top speed is up slightly. Health and weight are reduced very slightly (though weight is not really noticeable since you have a lot more punch with the increased speed). This is more of a 4-door sports vehicle now, able to keep up with the roadster (though not quite as nimble and the larger size means it can't squeeze through the same gaps).

Known Issues

• Localisation/Translation
  
  • Much of the new and edited in game text has still to be translated and entered into the game. Playing APB in languages other than English may result in English text being shown anyway in some places.

Pre-Open Beta currently known issues being addressed; key status update before OB launch!

Ok - so time for a long Pre-Open Beta post that outlines some of the current issues that are in the game, and where they are on the "fix-it-spectrum," since we have people yelling "LAG" and other things and I wanted to clear up what main issues we are currently working on from our side, and how some of those might be the same items that you are seeing from your end (in case you are in the unfortunate groups that are having current problems this week).

Also - while there certainly still are some issues (this is still closed beta after all), all the issues OTHER than "Item 1" are "local" to particular users, locations, ISPs or hardware configurations, and thousands of other players may not even be aware there are any. But some of the current issues are SIGNIFICANT - and we are in the process of addressing them. And as such; if you are in any of those unlucky groups affected (which we are really sorry for), the more specialized the issue is to your situation, the longer (unfortunately) it may take to sort them out (and in some cases require you to make some changes on your end), and the more general the issue is, the faster it will get fixed, especially if it's in one of the top items listed here.

But with that - let me roll out the list of top known issues that are now being worked on:


Pre-Open-Beta Issue 1: Multi-second lagspikes every 20 minutes
This is actually a PB server-side network spike. Today we have rolled out a fix on two of the districts in the US. If that fix holds up and doesn't cause any other issues, then we will roll out the fix on all servers over the next two days (this particular patch does not require any major maintenance). Therefore this goal is to have the issue solved by Friday this week. All you will notice is that things are not freeze-lagging any more.

Pre-Open-Beta Issue 2: Lag/Latency to servers that SHOULD be close to you, still making the game APPEAR laggy, which will be fixed with coming Routing update.
In the last two weeks we have run in to a condition where some end-users from certain ISPs report INSANELY long trace-routes to our datacenters, thus destroying their personal game experience. In some areas and regions it seems to affect ALL ISPs, and in some regions (like the US) it mostly affected ISPs using Comcast's backbone.

The issue has been isolated to an issue where one of our ISPs has a "visually optimized" routing system that "hides" their internal route hops, with the unfortunate side-effect that some network traffic now actually travels VERY LONG physical distances since it "appears" that the route presented by this ISP is "short" near the target servers (even when it's insanely far away).

The routing protocols incorrectly pick the long physical route with very few hops instead of a TRUE close route with MORE hops. The exact technical BGP4/OSPF reasons why this happens is too long to go in to here, but, needless to say - it causes BIG issues in the game when you live in Northern California, and your network traffic goes to Los Angeles, then goes to Virginia, and then back to Los Angeles and only THEN  reaches our server (!) (as was the case with Comcast customers from Northern California this week).

Last night (May 10) we made a change on the US side and now anecdotally we have seen players who had complained about 200ms traceroute end-times in the US, are now again seeing the correct 10ms ping and traceroute end times (as they should have all along). We will post more instructions on the forums to repeat this test from the European side (where we have to perform something very similar to suppress the rogue routes).

The Test targets (to try your route distance you can either tracert or ping these locations:)
US West (Los Angeles): 173.195.32.254
US East (Washington DC): 173.195.39.254
Europe (Frankfurt): 173.195.41.254

The only thing to note is that US East will show a 'break' in the traceroute until the game actually goes live there next week (there will be a * * * response as part of that trace route at the moment, which is expected)

For everyone's reference, from our offices in California we get the following ping times: 10ms to West, 69ms to East, 167ms to Europe.

That means if you are in Germany for example you should under ideal conditions see the reverse of that (10ms to EU, 90ms to East and 170ms to West). If you are getting something completely different than that, then you are having a route problem (or more accurately, your ISP is having a route issue) OR your ISP might just be located really "far off" the main trunks of the Internet and giving you a lot of hops just to reach the "super-highway." So test it out and refer to this forum thread for more information: http://forums.gamersfirst.com/index.php?showtopic=159972

If you are in Europe and seeing 100ms+ or 150ms+ ping times to the Europe IP included above, then there is potentially a serious route problem on your path. If so post your information in the forum link above.

Pre-Open-Beta Issue 3: 32-bit Win 7 / Vista users with 4GB of Ram CAN make their experience much better by using 3GB switches 
This issue is that in spite of what's installed in your machine, any 32-bit version of Windows actually can only use 2GB of User Mode RAM. Through some new uses of DXT compression we have been able to cram down the requirements to about 1.7GB to 1.8GB of RAM, but depending on the setup of your 32-bit machine - this might still cut it far too close to the limit to be stable (and leads to crashes and instability in some setups). The answer (other than installing 64-bit Windows which is the best solution) is to utilize the 3GB system switches provided in the new Windows versions (which on Win 7 and Vista are actually NOT called 3GB switches that at all). Autodesk has a very similar recommendation on their side for intense applications like AutoCad and 3D Studio Max:  http://usa.autodesk.com/adsk/servlet/ps/dl/item?siteID=123112&id=9583842&linkID=9240697

So while this is certainly a hard-core tweak (it requires the bcdedit /set IncreaseUserVa 3072 command to be run from the command prompt as an Administrator - and if you have no idea what I am talking about, you probably should not try it), those that have used it now report that they have gone from crashing 5 times an hour down to not crashing for several hours. The reason is that when the game uses 1.8GB of RAM, you have VERY little left to run anything else (Virus-scans, or even just more components of the operating system), so by enabling the memory to use all the way up to 3GB of memery (with the last 1GB reserved for O/S and Kernel) you can perform much better in the game. HOWEVER (and this is key), once you do this, your O/S or Kernel COULD become unstable, if you have far too many drivers or other components loaded (or just crapware on the machine). But on a reasonably clean machine, this is a pretty harmless tweak (but again - do NOT attempt this unless you fully understand what you are getting in to).

Pre-Open-Beta Issue 4: Green graphics splotches / lower frame rate with compression / White Purple Glow / Invisible Man problem opponents

The team is also working on several DXT compression related issues. DXT was added specifically to help performance of 32-bit systems with 2GB of RAM (see Item 3), but as a side effect, there are now cases where it creates strange artifacts for both 32-bit and 64-bit users (such as green-glowing pixel issues on certain clothes) and we are researching if this addition is also potentially affecting frame rates on certain video card combinations. The first part of this issue will be solved in version 1.5.1 permanently (the version after Open Beta), but in the meantime we are working on this. In particular the White Purple Glow and Invisible Opponent problem is clearly an issue (and no it's not an exploit thankfully, just a graphics glitch confined to your local system).

Item 5: Various driver and sound issues causing potential crashes and other technical issues.

So above and beyond the top collection issues that we are working on there is also a secondary category of issues that are a bit harder to track down, for example an elusive sound driver problem for certain sound cards (and in some cases that's ALSO tied to the 2GB limit issue for 32-bit Windows). These issues will also be squared away one by one, and once we also will have much more extensive QA times for each subsequent patch.

Positive Game Play Enhancements - First Open Beta:

On the positive side of things, we are working on two critical game play enhancements for Open Beta, which we think will be great additions to the game; 

(1) Skill Ratings based on actual skill (has been detailed extensively in an earlier post: http://apbreloaded.blogspot.com/2011/03/missing-skill-ratings-and-thanks-to.html ) where we expect the Skill Rating part itself to become part of the Open Beta and 

(2) Meta-grouping for Level 9 players (aka "Rating 9" in APB terms). It turns out we have a GIANT group of players who got to level 9, which is the first group right after the tutorial, and then simply stopped playing. If you don't both find a group and also ready up ("K"), you COULD end up roaming the city for hours with nothing apparent to do. Before Open Beta we expect to have this enhancement added as well, which takes care of this particular issue.

Positive Game Play Enhancements - Later Open Beta

After the first OB build our focus is on two other items (to be included IN the Open Beta)

(1) The District segmentation by Threat and Skill Rating. This will basically allow you to "progress" to harder / more skilled districts over time. It also (potentially) allows us to ease up a little bit on the upgrade restrictions, since as you get better (or buy better stuff), you will enter in to better and better districts. It also means you can manually enter a higher ranked district, but not go to a lower district.

(2) This one is key: Purposeful End-Game PvP. Without going in to all the details here (but I will do that in a much longer post once we get to the Open Beta), we are working on a very significant improvement to the game that will give actual purpose to teams and clans, and more importantly, gives a purpose to continue challenging others to fight you in the game itself for a very very long time. We will share more about this once we get to the actual Open Beta, but the short version is that just like guilds are important in WoW we are making Clans important in APB, and the long term survival and glory of clans will be critical to the game's success.

Releases 79 and 80, plus US-East server move before going Open Beta

Hi guys

The blog went a little quiet last week as we were dealing with a lot of post monetization items and preparing to get ready for the big May 18 date. In the meantime the moderation team on the forums have been cranking out a lot of answers, so go check out the forums in particular if you are already in the Closed Beta.

This week we are going to release Patch 80 on Tuesday night US time, and we are also going to perform the "formal" move to the "real" US-East location, in Washington DC later in the week (probably on Thursday this week) - which should really improve latencies for everyone east of the (imaginary) Dallas / Oklahoma City / Wichita / Omaha / Sioux Falls / Fargo line across the US.

Or - put a different way - if you live in Dallas - you should have approximately the same latency to either side - but since local cable conditions vary wildly (for example - Comcast LOVES peering in Washington DC and if that's your provider certainly go that direction), it's entirely possible that you will in fact be much better to one location specifically (so test it out once it is live).

Patch 80 fixes a lot of little things that will make for a better game now that they are patched (see below) and improves a few stats for some Armas marketplace weapons, while the team continues working on the BIG patch items that are upcoming in the next release. After the "80" patch is out, the team is really focused on working on the formal OB patch, and we should know this week if all the features we are aiming to add can make it in to the final OB version.

And - I realized that I had not posted the Patch 79 notes on the blog from last Thursday night (though Vali and Nueme handled that on the blog), so I have added them here as well. We were unfortunately forced to perform a music wipe in that patch to fix long running music bugs that were unable to be addressed any other way. Going forward that should not have to happen again.

Looking forward to the Open Beta excitement that's just around the corner!

Cheers,
/TechMech



1.5.0.80 Release Notes (scheduled May 10, 2011)
Bugfixes
• Made an initial fix for black and dark clothing showing random colour patches on it due to DXT compression. We’ve still got a few fixes to make here but this will remove the blotches of green/purple.
• Fixed characters sometimes not being lit in the Wardrobe and appearing as a black silhouette.
• More spawning bug fixes.
• Renamed female Cowboy hat to Cowgirl hat.
• Made clothing items that are not available through progression non-tradable.
• Fixed an exploit: Tutorial missions are now only offered to solo players.
• Increased Prestige multipliers for Enforcers (as even with witnessing they're getting a much lower cash turnaround than criminals).
• Increased the VIP lives on VIP missions by a factor of 1.5.

Weapons
• Named the weapon presets for the ALIG 762 and the Colby PMG 28.
• Adjusted Effective ranges for several weapons (Joker down slightly, Obeya up slightly, Star LCR up).
• Adjusted damage falloff percentage for the N-HVR's, the DMR and the STAR 'LCR'.
• Improved accuracy on Obeya CR762.
• Reduced Recoil and FoV on the Scoped NTEC-5.  Also increased accuracy loss when moving in marksmanship.
• Increasing damage of Armas shotgun.  Now slightly less damaging than the JG rather than only half damage as previously.
o Tradeoffs:
 JG - Extreme damage, medium spread (good at close range, bad at medium range)

 CSG-20 - High damage, low spread (worse at close range, decent at medium range)


Known Issues
• Localisation/Translation
o Much of the new and edited in game text has still to be translated and entered into the game. Playing APB in languages other than English may result in English text being shown anyway in some places.

1.5.0.79 Release Notes (released May 5, 2011)

Bugfixes

• Moved broadcast messages to upper centre of screen rather than centre to avoid blocking line of sight.

• Fixed the one-slot van not having it's one slot

• Killing a prestige 5 enforcer as an enforcer drops you to prestige zero.

• Fixed a bug in the spawning algorithm.

Songs and Themes

• Corrected inconsistencies and bugs across several musical instruments (note sequences off, patterns of sounds out of order) making them easier to use correctly.

• Unfortunately this requires a wipe of all character-created songs and themes, since existing songs break using the corrected instruments.

Known Issues

• Localisation/Translation

o Much of the new and edited in game text has still to be translated and entered into the game. Playing APB in languages other than English may result in English text being shown anyway in some places.

Open Beta Date, Credit Bonuses, New Items, Character Restorations and Cheaters

Ok guys - today we have a lot of interesting things to talk about - so we'll hop straight to it;


Open Beta Date has been set
We have set our formal Open Beta date: May 18, 2011.

With the builds over the last week fixing most critical issues (still some PB optimization and hardware issues to iron out over the next couple of builds), plus the items currently in the pipeline that addresses additional items, and the near completion of the US-East datacenter, we are now heading forward with removing all restrictions to joining the game on May 18, 2011.

However, between now and the Open Beta date (ie for the next two weeks), we will progressively permit access to MOST PEOPLE (17 or older in the US and 18 or older in the rest of the world) who do two things (1) creates (or already has) a GamersFirst Account with the appropriate info (see the link to the right of the blog) and then (2) submits his/her email (matching the G1 account) at the front of the APB page (www.gamersfirst.com/apb) in the "Newsletter Signup Email form." This will let us slowly turn up "the heat" on our servers before going fully "open."

To that end, tomorrow during maintenance we will let in another batch of about 40,000 people or so.

There are still a few important features that are not yet in the game (new skill rating system probably the most critical and skill-based district segmentation and progression), and various tweaks to spawning but those will be added as we move along, and likely some of the new ones will be there before Open Beta.

Credit Bonuses have been processed
In order to fully test our credit charging features, we offered a 100% credit refund to everyone making an APB credit purchase before the end of May 1. Those have now been credited to the eligible accounts, and the notifications (or rather - reminders) will go out tomorrow morning confirming the credits (though you can see the credits in your accounts tonight). Thanks to all Closed Beta testers who helped us test this critical feature. Next we will be working on making some of those charging screens a little bit prettier :)

New Items ready to go
Over the next two weeks we will be starting the process of releasing a new group of items for lease or purchase. We will start slowly with a couple of new leased guns this week, and then will follow that up with actual permanently purchased items, customizations, symbols etc (those are all permanent to your character). We have several hundred items in the pipeline to release over the next several months, some which will be reserved for Premium players as we go along, and some that will be rare and only available for short periods of time.

Character Restoration
Tomorrow during the maintenance window (4:00pm PST) we will release the character restoration system, to reclaim old characters and customizations. We will send out emails to everyone that have characters that they can reclaim.

Cheaters (aimbotters)
It keeps baffling us that people are cheating in one moment, then paying for premium, and then continue cheating? Seems really odd, given the amount of money you will be risking for something that literally only will  help you aim. We have ON PURPOSE not kicked hackers for over a week to monitor what they are doing. That clearly will change. One semi-famous aimbot site realized that we had caught a slew of their users over the weekend (though we did so silently), then they stopped their own hack, and then earlier today re-enabled it. Sigh... when will they ever learn. Anyway - before Open Beta we will perform various changes, which will be good for the game, and bad for the aimbotters. Long term we have a more radical solution to the issue, which we will share when it's ready :). In the short term we also have what we think is a good solution, and you should see the beneficial effects of that by Open Beta.

Anyway - with that long list of items, we are steaming ahead with all the other long term changes, and I will update this as we progress on the critical long-term PvP and team improvement items.

One day the team will even schedule in some sleep time, but we are not quite there yet :)

Cheers,