Do we need some sort of 'cheater amnesty' as we crack down on cheaters?

Wow - so yes - we have been tracking cheaters for quite some time, and with the launch of patch 130 we basically KILLED almost every cheater from all three major cheatmakers - in spite of their claims to the PB shield and other BS that they have been working on.

I clearly will not tell you HOW you were observed. Let's just say you were observed. And we can say with unequivocal certainty that the account you used has now or in the past been used for cheating. Sorry.

Some people posting in reply to the last post clearly stated things like 'I borrowed the account from a friend' or 'I got the account from a cheater friend.'

So - what do we do now? And what do the cheaters do now?

Sure - as a cheater you can go make a new account and RAGE CHEAT on the servers this weekend. Sure you can get some cheat-time in. But for what? Of course it just means you will be banned again, and with the next couple of builds we are ratcheting up the cheat hunting in such a manner that you cannot return. Ever.

Come Clean Instead!
Instead I have a MUCH better idea. If you feel remorseful. If you think you want to come back to the game. Then uninstall the cheat (or in some cases - because several of the cheats are root-kits that have actually BORKED your machine, you may have to re-install Windows), make a new account - and then play nice in the community for the next few weeks while we roll out the next several changes, and then pray we MAY let you continue on the new account before we launch the next anti-cheat patch. If someone TRULY comes back and plays clean - we may consider it. However, under no circumstances will you get back anything from the banned account. Don't even ask.

There are some interesting stats for the crew we banned (and to give you an indication how this was NOT a 'public' or 'easy' cheat ban). Nearly 50% were paid accounts (!). Some had 1500 HORUS of game play (that's 240 hours a month for the period one of the players played!). Some had spent a lot of money. And they STILL could not keep themselves from breaking the rules and destroying the game experience for those around them. It's all VERY SAD.

Based on the bans we can now even compute the revenues of the major cheat makers. We estimate that the three cheat makers make from $15,000 to $50,000 in revenue per month (depending on the cheat maker) from their cheats from the APB portion of cheats. So - for those of you out there will to pay a farg-ing $30 PER MONTH for a cheat - AND as much for items in APB - WHAT ARE YOU THINKING? AND - why on earth do you support the cheat makers to begin with. THEY are the ones that just cost you a boat-load of money, time and effort.

No cheat will go undetected forever. So I say it again - just don't do it!

Here are some interesting screenshots from the 'top-three' cheatmakers. Anyone feel pity?

You have ONE LAST CHANCE to stay with this game before we go live. Don't F-that last chance up.

/TechMech

Quick "version 130" update

Today we released patch 130 of the game. This intermediate patches fixes various minor items, but also further tightens security in the game. We expect 1000s of accounts will be banned as a result of this patch.

And again, surprisingly, why people PAY for game items and STILL cheat blows our minds. You will be caught (eventually), and what fun can that ever be?

Patch 130 resolves one of the 'rubberbanding' issues that was introduced in patch 123, though further testing is required for a permanent solution.

We are continuing to rapidly move toward the critical 1.5.4 version of the game, which introduces the first iteration of the new maps and game modes. Also 1.5.4 will also further tighten cheat resilience another notch.

Finally, the game will be in retail in North America in December. This means that the 'LIVE' version of the game will be scheduled to coincide with the retail release.

/TechMech

Entering the Open Beta "Home Stretch" - Part I

Player Motivation, Player Control and Cheats


First a quick note; from September 15 - September 20 we are offering a 100% rebate G1C bonus for everyone who buys any items in the new APB Reloaded (version 1.5.3) which we launched September 15. Click here for some more information about this promotion. There are always issues with a major new version, and this promo will let people make purchases while knowing they get a complete 'redo' later on.

Hi everyone! I cannot believe it's been more than four weeks since my last post! 

A lot has happened in the past four weeks, so this post is going to be quite long (and hopefully filled with some great information). After the last entry, we left our offices and went on a pilgrimage to Cologne where the team participated in GDC Europe (we discussed APB in some of the seminars) and GamesCom (which is one of the largest consumer game shows in the world).

GamesCom was great, especially since it gave us a chance to meet players, many who offered us a lot of great feedback and good ideas for the development team to mull over (though it was also a little bit entertaining to watch some new APB players search for the XBox game-pad controllers), and it was also a chance for us to meet some of the more well-known APB players in person.

To make it even better, we gave away tens of thousands of codes to players who stopped by the booth, and a lot of other swag and items. It's always good when you get a chance to play Santa in the middle of summer.

Upcoming Development Roadmap

In August the development team focused on finishing up the first 1.5.3 version build (which we released today). Builds 1.5.3 through 1.5.7 represent the "Home Stretch" for the entire Open Beta process, and after 1.5.7 we anticipate considering making the fateful leap to the Live version of the game (wooo!).

Version 1.6 is scheduled as the first Live (non-Beta) release, but before we actually get to 1.6, we still have a lot of work to complete, so today I will cover the most drastic upcoming changes, discuss a lot of the overall issues in the game and what we are doing to address them, and finally next week I will share the 1.5.3 through 1.5.7 roadmap as a link at the top of this page so you can see some more details about the upcoming changes.

Issues spotted in Open Beta

The first 90+ days of Open Beta have been great in helping us identify how players actually interact with the game, where players (like yourself) go, what you do, what you are willing to pay for, what you like to spend time on, what customizations you like, what cars you drive, what players you group up with, how you make your clan, what you hate, what you like and what activities motivate someone to continue playing the game (and sending virtual love-letters to the staff), and what issues and problems drives someone to rage-quit (and makes them email us 15 tickets in a row that all start with FFFF---KKK). The amount of data collected in Open Beta has been staggering. Unfortunately we have to now prioritize and structure this insane amount of feedback into actionable things.

Issue (1): Lack of Player Control

One of the core issues the game suffers from is the lack of player control. Not 'control' in terms of changing your character or designing a clan (which obviously are great customization features in APBR), but simply the inability to pick the types of missions and gameplay you like to play when you want to play them.

Unlike traditional games, APB Reloaded uses the "Mission System" to assign you missions while you drive/walk/sprint around the city (of course only if you remember to 'press K' - which is another annoyance of those we observed from watching people play at GamesCom).

The main problem with this fundamental design is that you may not be in the mood for the next mission you will be assigned. Even if it's a good mission, you might not have the right crew, time etc. for that particular game type. But as soon as the mission is assigned - that's it - you are committed (or if you quit, you of course lose Threat Level and thus risking your "Motivation" - so the Threat Level system is ANOTHER problem altogether which I will address below).

How are we going to return player control to the game, while still letting people have a wide open 'Living City' experience?

The first answer we are pursuing is "Fight Club" / "Proving Grounds," now scheduled as our 1.5.4 release. These missions, which you will be able to trigger directly from the district select screen, are the first iteration of what will become a series of dedicated matches that teams and players can launch and get a pre-defined "tweak-able" experience. We will also be able to tune what is included in the pre-defined experience on a weekly or monthly basis until we have given it the right length/content/feel (and take player feedback on which encounters are the most engaging and fun).

The idea is to give you the chance to jump in to a violent 10 v 10 (or similar) death-match or territory capture in a very small isolated portion of a district (which - by the way - also solves some 32-bit and low spec machine issues by being much smaller than the standard districts) for intense short-burst action. Think of it as a PuG clan fight, but with open ended groups/teams on each side and without all the Clan Mechanics.

While this provides insta-match content for players to experience, it also helps us try out new map changes and game-types which if successful, will be added to the main game maps as well.

We will have a whole post about these insta-missions, and how it relates to other districts and features still under development. In the current schedule Fight Club / Proving Grounds are in version 1.5.4 and Clan Warfare is in version 1.5.7 (check back next week for the details of the roadmap). We are really excited about the possibilities of this system, and letting players use it to get to the 'meat' of various types of action in just one click (again - without removing the current open ended districts).

Issue (2): Cheaters - and the Perception of Cheaters

APB is a Server-Side game. This means that barring various programming snafu's, there really are very few things you can do in the game to actually cheat. Unlike peer-to-peer games or games that rely more on the client being authoritative, APB is impossible to cheat with things like 'one shot kills' or 'invincibility cheats' and similar cheats of those types. The server rules all. Therefore (again barring coding mistakes, which do happen on occasion) most of these types of cheats are useless against the game.

However - there ARE things that the game is vulnerable to - specifically two items; aimbots and wallhacks (opponent 'ESP tags') since those two types attack the local client and exploit locally hidden information (players behind walls etc.). There is nothing quite as frustrating as finding a good hiding spot, only to be 'magically' found by your opponent. We agree.

We have two ways of attacking this issue, one is to change actual game code that will make certain exploits much harder, and the other is to work with an enhanced cheat deterrent system. The pros and cons of PunkBuster (or according to the community - the Cons) are quite well known, even though just last week it banned 2,000 players in one swoop when it was able to suddenly detect the latest evasion attempt from a cheat maker. The issue with PB is that it remains one step behind the aimbot/wallhack makers. But there is a solution for this issue in the pipeline as well.

Cheat solutions in 1.5.3 through 1.5.7

Starting with 1.5.3 we are making various aspects of cheat coding much harder to perform and/or easier to detect. Because it's cheating we are talking about we cannot really disclose exactly what we are doing, but as of this patch key cheat functions are starting to break in really odd ways. There should be plenty of frustration in the cheat maker side over the next several builds, though 1.5.3 has only implemented a portion of these changes. The goal is to get to make cheating a non-issue by the time we go formally "Live" with the game. As you can imagine, that's a tall order, but based on recent progress (some not yet in the game), we are starting to feel better about the prospects of solving this issue.

In a later patch closer to Live we have a much more drastic change that will also be rolled out, in addition to the road we have started down in 1.5.3. The goal is to have the full suite of changes in place before we release 1.6.0. 

So - What do you mean by the "Perception" of cheating?

IF there are only TWO serious types of cheats that even work in the game, how about the 'perception' of cheating going on? Why is this perception rampant? (see Issue 3 for another critical problem - that also causes rage and/or cheat anger - so all these issues are tied together).

First Perception - Snapping
First - there is an oddity with the game itself. Snapping is part of the game (!). If you run in one direction, but are looking behind you, the moment you hit 'fire' your character on all opponent computers will appear to make an instant 180 and 'snap' in the opposite direction while firing the shot. Good players can make it appear they are 'multi-snapping' by being very exact. The effect is very easy to test with two computers opposed to each other.

The result is that in many matches you often perceive your opponent 'snapping' to you (almost magically), since you don't actually get a copy of the opponent's camera movement, only the end-shot (snap). That's not cheating. That's normal game play. When you face an extremely good player, it's very hard to NOT think they are cheating. Mostly because the visual appearance of the opponent (snapping around to different targets) makes it look like something is amiss (compared to playing MW2), but also because most of us think NOONE can possible beat us or have better map awareness :) . We know we are awesome :) (again - not saying that aimbotting and wallhacking cannot happen - just that often it also appears to be happening while it's really NOT taking place). Snapping CAN indicating cheating, but only when players target multiple targets that are large angles apart in a split second. Unfortunately the two types of snapping (legitimate and the real cheatin' kind) are hard to tell apart.

We will release some videos to show this in action as well.

Second Perception - Magic Bullets

Second - not all misses are noticed by opponents, often due to how bullet flight is handled. If you are being shot by an enemy all bullets that HIT your avatar will always be readily apparent. Bullets that MISS are still sent to your computer, but we often see test players in our focus tests that do not realize that a large number of bullets were fired in their direction during an intense firefight, and that many of them missed. This can lead to a perception that opponents magically HIT YOU with every shot they fire (and magically never missed!), though in reality your enemy is shooting more shots in your direction than you might realize. 

Third Perception - Cheats that Aren't

Third - there are some supposed cheats (some that are even sold to people) - that in fact are not cheats at all. The best example is the 'no-spread' supposed cheat. If you use a 'no-spread' cheat and fire at a wall - you will THINK that bullets are hitting a wall and strike a tight area (or even easier - you can muck with local INI files and see visual changes in your client without having any effect on the server or opponent games). That's because it's a client side change. But, in spite of what you might see on a wall, shooting another player is handled entirely on the server. Turning on a 'no-spread cheat' might make you FEEL you are aiming better (and maybe you DO aim better because you 'feel' better) - but you are actually just as spread-y against opponent players as you were before the 'cheat' from the point of view of the server and the opponent player. Ie the cheat has no effect other than on your computer.

Are there any cheats right now?
Yep. No bones about it - that some people are using wallhacks and aimbots (or after the current update, are certainly trying to use them with the 1.5.3 build), and there clearly is a delay before they get banned. Therefore we are very interested in seeing the changing pattern during this patch version (1.5.3) and then followed by the upcoming 1.5.4 through 1.5.7 series of builds. What will players do? What will the cheatmakers do? We will all find out!

Other possible solutions - player kicks?

We often get the request to implement cheat-vote-kicking. A small sidenote, when Zak/Qwentle jumped in to Combat Arms for some quick analysis on how they handle something he was trying to work on, he was quickly vote-kicked by the Combat Arms players for being a 'cheater' (something you can do in CA). He clearly was NOT cheating, but he was simply just very good at playing the game, even though he had a brand new character. So player-banning also have problems.

So - what are we going to do?

In 1.5.3 we have done some interesting stuff. In x.x.4, 5, 6 and 7 we are doing additional interesting things. Depending on how they work out, we will start disclosing exactly what has changed. Let's first see how people react to them...

Third issue: Wrong Motivation Goal - and Driving People Nuts (aka the evils of Threat Levels)

So - not only do you have a lack of player control (ie you get handed a random mission, and if you quit it you lose Threat Level) you also start thinking (incorrectly) that Threat Level is actually something you really want to care about as a progression metric and a community status symbol. It was never intended to be either.

We really apologize for not addressing this earlier... Threat Level (or ELO rankings or Glicko Rankings) should actually never be a driving factor for your enjoyment of the game, but partly because of how they are presented, it's become the ultimate status symbol.

To use a golf analogy, if I had to continually show my golf handicap (which is not great) and every day I played I was constantly shown Tiger Woods' handicap (which, even in his current slump is probably a sub-zero handicap), I would simply not feel all that great about myself. Or having my high-school football team compared to a professional football team every day (American football or European football - same issue :) )

In games like Golf (or Tennis, or Football) we get around this comparison problem by simply 'grouping' ourselves in logical cagegories, divisions and leagues. NFL teams (or Premiership / Bundesliga teams for the football games where you actually use your FOOT) are never ranked against or compared with high-school or low category club teams. And that's for a good reason. It doesn't make sense.

We are working on a drastically different way of representing Threat Level and progression in the game. Some aspects will be the same (a cool symbol), but what the symbol means will be radically different (and potentially more informative).

Here is our latest idea of how we will do this. You will be assigned a skill rating and a progression. The skill rating (based on the underlying system we have today) will give you one of four colors; Green, Bronze, Silver and Gold. Players will be assigned one of those four colors based on skill rank.

The actual symbol will be one of 20 different symbols (initially) representing the progression you have made in the game; 0-9, 10-19, 20-29 etc. So you will get a different progression symbol every 10 levels from 1 through 195 (until we expand the 195 to permit higher levels as well).

In all matchmaking locations where today you see the Threat Level symbol, you will instead see the new combined Skill / Progression symbol giving opponents information both about your general threat, as well as your progression through the game.

This way you actually get better information about the gear and content that a player is likely to have equipped. Especially for Gold players there is significant difference between a Gold who is progression 25 and one who is progression 125,since the higher progression player has a lot more versatility in equipping the right gear for the right encounter. With the current TL system, this critical information is never communicated.

Will you EVER get to see the 'real' underlying detailed skill information? Potentially yes - in particular in our upcoming Clan Warfare system you are likely going to have the ability to 'see' your mates detailed data (partly so you can build the most competitive clan fighting team to defend your clan ladder rankings). There is becomes important info (and not unlike how WoW deals with the ELO levels, and the same reason you can only see your own team's detailed rankings, but not your opponents).


New contacts coming soon


Now in order to expand the game for those player who are already high level, we are also working hard at adding new levels of progression above the current contacts.  Those who have completed both organisations for their faction will be given the chance to work with several specialists which we are adding in the next several versions of the game leading up to the Live Launch.

In addition to expanding the Progression range above 195, new rewards for working with these contacts will also include High level Vehicles, Symbols and the first parts of Tactical Gear / SWAT Gear (Helmets, Pads, Loadouts, Webbing Armor etc).

More importantly, they’ll also unlock a plethora of new Modifications that are not yet in the game, giving players new options and choices to make when deciding on the best mods for Weapons, Vehicles and their Character.

Back to the Grind-Stone...

These are the three areas where we are really putting a lot of focus as we race toward the first 'Live' finished game. As we roll these features out, we hope to continue getting your feedback (though, you may want to wait with final feedback until the features are live :) ), and then get to a successful 1.6 Live Launch.

Next week I will post the updated Roadmap at the top of this page, so, check back then for more details about the upcoming changes, as well as the design details of Proving Grounds.

Until next week,

Server lag = solved, TL-ratings changed, NVidia SLI support and patch 116 improvements

Today's patch caps lots of different tracks of optimizations and improvements that have been worked on in parallel for quite some time, so this post will outline all these changes.


Server Side Lag Solved

First up - as of this week we have officially sorted out all the items related to server side lag that were killing some servers at various strange intervals (5, 10 and 20 minutes generally). Without going in to extreme technical detail, the issue turned out to be the method used to invoke various external libraries that under certain conditions could cause network congestion on the server itself. That issue was fixed in the last 10 days, and combined with new networking equipment from Internap that optimizes the route selection from individual players, most server side lag conditions have now been completely eliminated.

The network routing optimization boxes are quite cool, as we were able to observe in real time some players connecting from Australia to Frankfurt (maybe not the best idea to connect that far) and see them drop from 1500ms ping times to 500ms ping times. Of course the 500 we cannot do anything about (given the physical distance), but the 1000ms improvement was very significant.

Will you or some of your friends still see game lag? Sure - under three conditions; (1) your home network runs in to congestion, (2) your ISP has a freak-out and messes up its routes to its peers or (3) your home computer is not up to spec and you end up lagging all the time (but that's of course client side lag which is a whole different beast).


NVidia SLI support in the new 280.26 WHQL drivers

NVidia notified us this week that the new WHQL drivers now have the APB profiles included. This is the link to the drivers. The SLI profiles are NOT listed in the release notes, but, if you install these drivers and attempt to run the game using NVidia SLI card config, you will see that the game FINALLY runs as expected.

Download the final release drivers from here:
http://www.nvidia.com/object/win7-winvista-64bit-280.26-whql-driver.html

(and again - the listed games in their notes is for NVidia 3D vision support, something we have not yet added, but is also on our list of optimizations to come at some point in the future).


Threat Level Revisions

A few weeks ago we released code which altered the matchmaking, but in doing so, the code also (actually at my request) included a new 'uncertainty' modifier value which basically made the matches made with new players slightly more 'uncertain' (and for a longer discussion about the matchmaking changes you can look at the earlier blog posts).

Due to some interesting math behavior - the rating system has been properly rating people (so everyone on a relative basis has the proper rating). Here is a histogram of a set of 15,000 ratings from one world from one day in the game using the RAW (underlying) rating data.

Keep in mind that this data DOES NOT equal the game Threat Levels (as you can tell from the above diagram, the ratings are from negative to positive rating levels and don't line up with TL values). However for those who like math, the ratings DO fall on a nice even bellcurve, which in fact IS the expected outcome of our rating system.

So if the RAW ratings look nice and curve-like, then what on earth went wrong when 'translating' the ratings to in-game Threat Levels? 

In short - the location of the middle of the bellcurve is NOT where we had expected it to be when we implemented the formulas. In fact - in the live data the bellcurve center appears around the value "5.5" and not at the expected value (which was expected to happen much higher up on the translation scale). There are a lot of math reasons for this 'real world' outcome (after all - that's what we use Beta to figure out), some of the reasons had to do with the formulas and parameters used, and some were the result of the APB mission system and how players of different ratings group for missions.

So what happened in the current translation in game? Well - we accidentally ended up with THIS lopsided translation to threat levels that people saw in the game:

What actually happened was that the translation from RAW rating data to visual game information (threat levels) pushed 35% of players into TL 1 (aka Bronze 1). In fact a total of 50% of ALL players were rated Bronze 1, 2, 3 (!). That means there is an ENORMOUS difference in skill between players within those three rating levels - and then everyone else were spread out across the rest of the spectrum. Or another way to think of it - the difference in skill between the lowest Bronze 1 player and the highest Bronze 3 player was just as great as the difference in skill between a Bronze 4 player and a Gold 10 player (!). 

Clearly that was NOT the expected amount of players in those threat levels, and the TL values became pretty useless to gauge overall skill between players. 

In spite of the above translation issue, even though the underlying RAW data was correct, the visual translation into the actual game certainly did not work as intended, and has lead to a LOT of concerns and complaints in the game.

As a result - with patch 116 we have re-translated all the RAW data, into new TL values. When 116 goes live, then the ACTUAL threat level distribution will look like this:

The idea with this distribution is that 30% of players will be Bronze (1-10 in the above chart), 60% will be Silver (11-20 above) and 10% will be Gold (21-30). In this rating system only 0.2% of players will be Gold 10.

The vast majority of players will end up in the range Bronze 6 to Silver 8 (80% of players), and then players above (Silver 9 and up) or below (Bronze 5 and less) will be quite rare (roughly 10% above and below that middle set of values).

Silver 5 will be the most common rating, with 9% of players being a Silver 5.

So when you log back in to the game tonight - don't be surprised that your level has changed. In fact - it may have jumped from a Bronze 1 to a Silver 2 (if you were at the very top of the Bronze 1 range).

Therefore the new threat translation should give you a lot more useful information about the matches you have with opposing players.

Will there be other TL related issues in the future? Potentially - but the plan going forward will be to re-balance the TL display at least monthly.


Hong Kong Premium Server Launch, and the upcoming Russian and CIS world.

Tonight we also enabled the "premium" server in Hong Kong. This is entirely a system meant to give low-latency access to a region that we don't really support yet (given the very high prices for connectivity in the region), and you can ONLY access this server if you are a premium player. That's a little counter-intuitive for a Free2Play© game, but, since you can continue playing for free on the Los Angeles server (where our cost of operations is 10X lower than in Asia), we felt that we would offer a premium access choice for those who really want low ping times to the games locally in that region.

If this experiment works, then we will also consider adding an Australian Premium server given the tremendous amount of interest we seem to have from Australia and NZ as well, where the cost of internet service is equally astronomical.

We have made progress on launching the game in Russia and CIS as its own separate territory, and I will update the blog when we have more details on that launch. When that happen we will also be able to re-enable the use of Cyrillic characters in chat, game names etc. for the Russian and CIS worlds and will migrate all Russian characters over to that world.


Going to GamesCom or GDC Europe?

We do have a contingent at GamesCom and GDC Europe. If you are going as a gamer (to GamesCom) or an industry person (to GDC), certainly feel free to come by.

We have a pretty small presence at GamesCom (mostly for 'fun') but we WILL be giving out quite a few free codes and other swag, so stop by the APB 'alley' if you plan to be at the conference. You can also meet with some of the APB Devs, Artists and Designers who will be there handing things out.



Patch 116;  Various fixes plus 'spammy popup reduction'

For patch 116, please see the patch notes on the forums, but one of the bigger changes involves reducing the annoyance factor of the popup we added in earlier builds meant to point out the clear difference between premium and non-premium players. We may have gone a LITTLE bit wild (yes I know - maybe VERY wild) with the earlier version of the popup with the sheer amount of notifications. In build 116 we relaxed the spammy nature of the notifications to a much more manageable level.

In the new build the popup does NOT appear in the Tutorial (which was a little over the top), and does not appear when you hit Heat 5, and it ONLY appears after 50% of the missions during normal play. Of course getting a premium account completely removes the popup (yay!), just like premium cable TV has no commercials.

However - for all the people that expressed their concern about this popup, it is worth noting that premium conversions JUMPED nearly 40% over the old conversion rate when we added this notification (potentially people didn't realize what they were missing :) ), so we are going to continue using the popup (albeit in a much milder form). After all, this increases our ability to launch more stuff by being able to increase our staff to work on all the remaining issues in the game. After the popup was added about 33% of all the CCU at any given moment on any server are now Premium players, and the number is growing.



Future Changes

Next up - we are continuing to work very hard on optimizations and exploit fixes as well as the next set of feature changes. Based on some really interesting work we are doing right now we have been making some tremendous progress and at some point (when things thoroughly QA-ed and ready) we plan to release of a lot of changes simultaneously, which likely will take the edge of many current and future exploits. I won't be sharing more than that until it's ready to go (given the enormous amount stuff involved), but as soon as we are ready we will share much more data.

Finally the new features (ClanWarfare, Racing and Asylum) all are progressing, though with a relatively small team progress is taking a little bit of time. Once these features are close to release, we will share that here and then we will start preparing for the final 'formal' (non-beta) release of the entire APB Reloaded game.

Stay tuned - and see you at GDC and/or GamesCom in case you are stopping by!

Cheers,

Open Beta 2 (version 1.5.2) live now

(plus Nvidia SLI support coming soon)


So - after some initial delays (recall we had targeted June 30), APB Reloaded Open Beta 2 launched this morning.

The initial patch had a few stability issues at full production load, which meant we had to quickly re-patch, and now the team is working round-the-clock on another patch to solve the remaining disconnection issues in the next 24-48 hours.

Open Beta 2 (aka 1.5.2) contains the biggest changes to Missions, Matchmaking, Skill Rating and District Recommendation to date, and it's very difficult to simulate some of the effects of these changes in QA or even in the smaller "Test World" - so for the next week we will be watching all the data coming in from these changes very closely, so everyone should expect there to be rather frequent patches over the next week.

In other words - YES - between now and next Friday I expected daily occurrences of "15 minute warnings" to take districts down for patching. If you see one of those, then please just log in after the patch and continue testing.

NVidia SLI support will be added in next NVidia driver set
We just received formal notification from NVidia that they are finally including a dedicated APB Reloaded profile in the upcoming beta driver-set 280 thus finally enabling NVidia SLI support. The current driver-set (275) was released June 22, so we expect 280 to appear on the NVidia site the next two weeks. Once the 280 beta drivers are live we'd highly recommend every NVidia gamer try them out (and report on them). This driver update should finally resolve the high end SLI related issues for APB Reloaded (!). Next up - getting AMD to do the same for CrossFire, and then finally fixing some lingering sound driver issues.

After these hardware issues are resolved we will take the decision if we are going to create an Open Beta 3 (aka v 1.5.3) version that will incorporate the next major feature update and optimizations, before we go to final Live/Production release (version 1.6.0). More info to come on this after we have patched and updated 1.5.2 and everything is stable, as well as a longer discussion about future optimizations to 32-bit implementations (which still is affecting about 24% of all our players).

Matchmaking and Threat Level Changes
The initial indications from this morning is the people are getting much better matches, but under some circumstances are now seeing a pretty large REDUCTION in threat level compared to what they used to have, even when they are winning.

We will analyze this data today and tomorrow as we get it from the live servers, since it's clearly possible that there could be a bug in the threat system. However (and this is important), the changes we made can also make you arrive at a NEW threat level in the new system even after winning and it may end up being a LOWER threat. So it's entirely possible the system is working as intended and just being re-calibrated, but we will know more once we have analyzed the live datasets.

Also due to the changes, it used to be that everyone basically started out as roughly a Silver 2, and then moved up or down depending on match outcomes. Now instead everyone starts out as a Bronze 1, and from there start moving up (you can technically still move down, but then you just stay as Bronze 1). This change alone essentially could be thought of as 'reducing' all threat levels by about 12 levels (though of course what actually is happening under the surface is much more sophisticated than that).

Basically the threat system is based on two parameters, (1) your expected skill and (2) the uncertainty about your skill. If the system is very uncertain about your skill level, it will use the lowest possible value to signify your new threat. This is all to say that if you were to beat someone of a very different rank than you or who has a very UNCERTAIN rank, even if you WIN it's possible the system will determine that we are now in fact MORE uncertain about your real skill rating even though you won (!). If that happens it will (paradoxically) appear that you are going 'down' in threat level, but in reality your skill level may have stayed static, but the uncertainty about your skill level most likely increased pushing down your threat level. If you continue playing the level will eventually stabilize (though at a new value).

One of the reasons for this current effect might also be because as of 1.5.2 new players have HIGHER uncertainties (by about 25%) than in the original system (which is what makes them start out as Bronze 1 instead of Silver 2), and as those new players play with existing players, their uncertainties are 'spreading out' across all the other players (of course the math behind this is a little more complicated than this). But it also means that over time the system will again stabilize. However - it COULD MEAN that if you used to be a Silver 6, going forward will now be called a Bronze 9 (or other generally lower rating).

In the end, since you will be put in districts with similar ratings, the change should NOT have a massive impact on your performance.

However, we continue looking in to all the data, and we will determine if in addition to the re-calibration there is another hidden issue we need to fix. If we discover this then we will make an update, and also change the current threat levels. More info to come on this over the next few days.

Cheating Stats
So we were going to post a name-and-shame wall here on the blog (there is the outline of such a list already on the forums), but ran in to technical issues with the sheer volume of accounts (and blogger not having a good method of showing 6000+ entries) so we are still working on a method to link this to the blog. In this and future patches there are also new methods of detecting cheaters, though we will 'crank up the heat' on cheaters slowly (but surely) as we head closer to live production.

Contrary to what people think, the cheating percentage is quite small (in percentage terms) and we are finding, banning or suspecting about 1.3% of the entire userbase (another way to look at this; 98.7% of players do NOT cheat). And yes - Punkbuster might be slow to catch on sometimes, especially right after there have been some significant updates to paid cheats, but it pretty much always gets the cheater in the end. We DO have a group of suspected cheaters that we have NOT YET BANNED on purpose for a variety of technical reasons (those are the 'I'm crashing my system rather than being detected' crew - and you know who you are), but I feel sorry for their asses for having spent that much money to cheat in a free game, especially after we do include them in our future "Name and Shame" campaign.

Of course even though a low percentage of players cheat, they also ironically are also frequent players, which means they show up in matches slightly more often than non-cheaters. The net result is that usually a cheater is able to play a series of matches before we catch him/her (our name-and-shame log has this data as well), and there are large differences in cheating percentages by country. For example, a Russian player is nearly 2X as likely to attempt to cheat as an American player. Also paid players are slightly less likely to attempt to cheat (though some paid players ALSO try it, which does boggle the mind).

Again - just don't do it. If you do - you might get lifetime banned from all of our current and future games, even if your cheating was in the past (and we haven't yet banned you). Probably not worth the trouble, since all it really will do in APB going forward is to match you with much better players, so the result (if you are NOT a good player to being with and you are RELYING on cheating to get by) is that you will need to continue cheating to maintain your higher rating (it's like a crack-cocaine loop). If you stop cheating, then of course your competition will get easier as well, but during your descent you will of course lose (like drug-withdrawal).

Future Servers - Russia, Hong Kong and Australia
None of these future servers are live yet - but this is where we are heading. For Russia, we are in discussions with various Russian publishers to take all Russian players and to have a Russian publisher host and operate the game from Russia. There are a lot of reasons we want to segregate the Russian community, but mostly because we don't have a good payment network for Russia, and also to handle this from Russia would provide some improved performance to that local market. We will share more information about Russian servers as we announce those partnerships in the future.

On a trial basis we are planning something else interesting; we are going to put District Servers in Hong Kong and in Australia. But - because the cost of hosting, power and bandwidth is INSANE in those areas (bandwidth in Australia is 50X - FIFTY TIMES the price in the US) - we are going to make those district servers Premium Only. In other words - just like today, the thousands of players from Asia play on the Los Angeles servers, can STILL play in LA for Free, but if they want to play on a local (Premium) server they will need to have a Premium account.

As we get closer to these new server configurations, we will share more details, but I would expect this to happen sometime before September (and when it does - the world list will simply show new 'Premium' worlds that you can select).

In Summary
The next couple of weeks are going to be incredibly intense for all the teams as we work through Open Beta 2, and prepare for Clanwarfare and other critical updates that are just around the corner. We really appreciate the support from the community as we continue working toward making APB Reloaded the best Free2Play(R) game on the planet :) ! Obviously some critical things still to do (including Optimization for 32-bit and others), but after all the fixes are in we will have a really strong foundation for the next big release of the game. With today's patch we have taken a big step toward that goal, especially once kinks have been worked out in Open Beta 2.

Waking back up?

Alright. We have had an incredibly intense three weeks, so really sorry for the long posting gap (while of course the team has kept updating Facebook and Forums regularly). This will just be a quick preview of what's coming up, and I hope to follow this up with much more detail at 'some point™' when some of the critical items we are working on have gotten sorted out, and at that point also post a brand new roadmap.

We just released patch 1.5.1.96 tonight, which fixes various issues, and adds more server-side controls for certain items to reduce some specific client exploits. That's the good news. Keep in mind patch 96 is a specific hotfix patch, and does not contain the large items the team has been working incredibly hard on for version 1.5.2 of the game.

The Next Big Thing - version 1.5.2

1.5.2 introduces many new items outlined in the roadmap, most importantly district recommendation and skill-rating based matchmaking and it ALSO introduces a lot of items we had not yet talked about; meta-grouping for better PuG-ing, action grouping relaxation (fancy way of saying you will be able to shoot right as you stop sprinting instead of being blocked from shooting), 70+ different mission fixes, new clothing, new items and a boat-load of various bug fixes and improvements based on much of the forum and tester feedback we have had. So there is much more to talk about once that version is ready to go live.

The plan had been to get the 1.5.2 build out by June 30 (per the roadmap), but right now it seems we will go past that by 'some period™', partly because of the massive amount of changes requiring a lot of testing. In the next few days we will also decide if we will use this build as the first one to go up on the Public Test World. We have recruited and selected about 500 volunteers (who do not yet know who they are I should add), and people with those accounts will be able to test builds going forward before going live to the public. The next few days will determine if the PTW system will in fact go live before 1.5.2 or vice versa.

We have also completed the networking upgrades that were scheduled on the roadmap. Now players in Europe will also be able to connect through Telia, which should give customers a drastically reduced ping when going to the servers especially from Northern and Eastern EU. We are in the next few weeks also adding TiNet (an Italian telecom) which has improved network coverage for Southern EU, for areas such as Spain, Italy, France and Greece.

Some Currently Known Issues in the Beta

Finally - we are fully aware of several annoying issues we are tracking and hoping to address shortly. There are a lot of specific or individual items that are on our list - but there are two in particular that are more general and requires a special mention here - Latency spike and 32-bit issues.

There is a latency issue that kicks in every few minutes or up to every 10 minutes for several players, where suddenly latency spikes to 800ms in the game. However, if you simultaneously happen to be running a long-running ping in the background to 173.195.41.254 (our ping probe in the EU) or 173.195.39.254 (ping probe in US-East) or 173.195.32.254 (ping probe in US-West) - you'd see that the network ping/latency actually stays low during the spike, and server side and client side frame-rates also stay normal during the latency spike. So from what we can tell the current spike is not a direct networking, datacenter or server issue and instead appears to be a client side issue, at least the most likely culprit at the moment. We are working to sort out what other dependencies trigger the random client latency spike, but because of the complex interactions of all the components it will take a bit of time to isolate the root cause. We will update as soon as we have a patch that addresses this particular issue.

And the biggest issue of them all continues being 32-bit optimization for those who do not yet have 64-bit systems. APB LOVES memory. The districts are VERY large (compared to other high end games APB's districts can even be described as HUGE), and therefore the game demands 1.8GB of memory or more (and especially after a few district switches and encountering 100 other players with their customizations, that usage easily hits 2GB plus). We are working to optimize this, but at the moment the game will continue running close or at the 2GB limit. Unfortunately 32-bit windows by default only addresses 2GB of RAM, so this can lead to crashes on Vista-32 and Win7-32 in particular (XP-32 tends to fare a little bit better).

For MOST Vista-32 and Win 7-32 systems, especially if you have 4GB of Ram we continue to highly recommend TRYING (at your own risk) the bcdedit /set IncreaseUserVa 3072 command (here is a great reference from Autodesk which uses the same trick for AutoCad and 3DS Max on 32-bit systems). This command will increase addressable application space from 2048MB (default) to 3072MB, minus any memory mapped devices (such as graphics cards). Unfortunately it also means if you have a large memory mapped device (such as 1GB graphics card) this setting could cause conflicts. Some players have reported success when using a slightly smaller amount - in particular bcdedit /set IncreaseUserVa 2560, which sets the user-mode (program) address memory to 2560MB, which often seems to be enough to keep APB and it's very large world happy for a few hours. Here is the Microsoft reference on the bcdedit /set command. Again - try all these commands at your own risk (or go the proven route and upgrade to 64-bit Windows 7, which is clearly the preferred route).

There is a pretty good collection of known workarounds by moderator nXe here, which we have not fully vetted - but certainly feel free to try these as well in case you have compatibility issues.

Expanding the Team; more C++ game coders needed!

Also - as a result of the great response the game has had in Beta, we are right now expanding the development team in Edinburgh, UK. If you (or someone you know) is a fantastic C++ coder interested in either game engine development or game play development you can send in your cover letter (describing  why you would be great for a critical job on the APB team), and your CV/resume to careersUK /a t / GamersFirst / d o t / com (split apart just to help with our spammier friends - so your first test is to see if you can figure out what email to send your resume too :) ). 

There is some additional info on this page (click on "UK" to see the description): http://www.gamersfirst.com/corporate/?q=Careers. Our team is located right at the heart of Edinburgh, one of the coolest cities in Europe, with many great direct connections to other key cities on the continent. If you or someone you know are up for an amazing job and an amazing challenge where you will have a lot of impact, certainly drop us a line and try to convince us why you would be a great team member. Only friendly, well adjusted, optimistic and nice people need apply :) . And we are pretty serious about that requirement, since the team requires incredibly tight collaboration between all team members around the planet at all times.

In Closing

So - in closing - this preview is intended to get everyone up to speed, and prepared for the next major build, which really is the foundation for the more complete game that we had envisioned when we set out on this journey many months ago. A huge thanks to everyone who have played, tested and engaged in the world of San Paro for the last three months. We expect the game will be that much better thanks to all those efforts.

See you in San Paro!

Cheers,

The Development Road Map

In our effort to "hyper-communicate" about what we are working on, today we have released the Development Road Map (also reachable as a tab on top of this blog page).

At the moment it only has the high level public items we are working on for the next 30 days (and a lot of low level details which are not yet published). However, later this week we are going to expand  include all the cool long-term features that we are working on, such as Clan Warfare, Asylum, Racing etc. which really will open the game up to a whole new set of experiences.

Instead of answering the same questions (and quite frankly - zany conspiracy theories) over and over again on forums and on the blogs, we figured we'd rather publish all the ongoing work in one place, so if you have any concerns or constructive suggestions, hopefully the most common issues (lag, queues etc.) are addressed as ToDo items in the Road Map; for example the upcoming Firewall rebuild happening this week in the EU, the new ISP peer and IP provider we are adding next week to improve latency for EU players, the launch of the CBAK (Closed Beta Appreciation Kit), the launch of car kits, the new matchmaking and the new district recommendation items.

 So - while the current list is just a taste of what we are hoping to share, at minimum our aspiration is to NOT promise things "soon" Wow-style, nor on "Valve-time"  but instead the plan is simply to share things that we feel comfortable promising within some reasonable time frame. Will we ever miss a date? You'becha. And occasionally we will beat a date or two by getting there a tad early. But the general development direction should certainly be very clear from the roadmap itself.

Pearlescent paint bug patch today, and introducing the TMC.

Alert: Emergency patch v.91 today. Estimated to be released at 1:00pm PST / 21:00 BST / 22:00 CEST. In patch 90 a bug was introduced when customized items with Pearlescent paint appears in very crowded areas, causing a GPF crash. We are patching to fix this asap.

More specifically it's the type of bug that can only be tracked down with 100 live players (a crowd) and a full district where people have a great variety of different items (given the trillions of combinations of items players create). Even a static load test has a hard time catching this particular issue (and required some creative load testing for the development team just to reproduce the issue).

But - to catch these issues more reliably in the future, the next step is the launch of our public test world and the TMC (our Test Monkey Crew).

We will start taking applications for how to participate in the TMC after the E3 show is over on Monday the 13th of June, and Neume will send out more details on how it will work (and how you apply). Some of you will literally be asked to participate (since we can "see" what you have been doing in the game), and for some slots we will simply take applications.

(I am also working on a bigger blog post right now about our upcoming roadmap, so stay tuned)

-------------

update: June 6 - Scratch the additional blog post with info. Instead we'll convert (and condense) our formal "Roadmap" page instead and make it public for everyone to read. That way you will get far more details about what's going on in the game for both the next few weeks, as well as for the next several months than what you'd get from the blog posts (and commentary).

Also - it's still Beta. For 80% of players it's in fact very close to production quality and an amazingly great game to play. For some however it is not at that level for various reasons (hardware compatibility, latency to their ISP or routing issues, queues, SLI comparability etc.). And - yes - it's a commercialized beta, since part of this open beta is to test what items actually sell and for how much. Wouldn't be much of a test if we didn't do it "for reals." So - please hold any additional commentary until we release the roadmap.

Player Explosion during Beta

(update May 30, 1:30pm PST; as a temporary measure, in order to reach the in game store to buy premium or character slots without first logging in to the game, you can access the store at the following location: http://www.gamersfirst.com/marketplace/apb/ingame_store.php . We will permit regular we access as long as the queue-ing remains in the game. Note:item purchases should be performed from inside the game.)
--------

Ok - so first off - I want to cautiously remind everyone that we launched APB Reloaded "Open Beta" only six (6!) days ago.

Open Beta means primarily that there will be no more character wipes, and while the game is getting very close to final production quality, and it's certainly already a blast to play APB Reloaded in it's current form, the game still needs some key changes, fixes and upgrades before it is fully production grade. There are code improvements to be released for matchmaking, more purchasable items to be completed for the item shop, and key upcoming infrastructure changes to be rolled out. In short - the game is still in Beta, and every week it will continue improving.

This is also why Premium accounts (ie upgraded accounts that accelerate game progression, unlock customizations, increase drop rates and gives players item purchase discounts) are only priced at $9.99 to start, and $7.99 for renewals. This rock bottom "beta pricing" includes a good 33%-50% discount compared to other MMOs.

Now don't get me wrong - while still a little rough around some edges (such as a few pesky sound bugs for some specific sound cards, some occasional district DC issues, and the fact that the large map size dislikes 32-bit Windows, but LOVES 64-bit Windows, as well as more cheat related changes still to be made, and the fact that we are working to add more things for people to do, game modes to use and districts to explore after they get really high level) - APB Reloaded even in Beta is STILL a really great game, with a ton of wild fun to be had in the city of San Paro.


Premium holders getting a leg up, because they are doing their part to help rebuild San Paro
Getting a premium account supports the development team, and also ensures updates are rolled out even faster since more support means we can invest more in speeding up the development effort. And - best of all - if you don't want to upgrade, you still don't have to. You can still play the game for free. But I do want to take a moment to THANK all the thousands who have already jumped on board and gotten premium accounts! Your support is invaluable. And therefore, we are of course going to let you "cut in line" when things get into a capacity squeeze.

Having a huge volume of free players means that free players will occasionally have to queue for slots at peak times, since it's important we provide priority access to paying players. Even so, our explicit long-term strategy is actually to make sure that everyone (include all free2play players AND all paid players) get basic game access without major game queues, especially when the game goes "Live" after the current Beta phase is over.

But during the massive growth phase we are in right now, or during large upcoming game changes, priority certainly always goes to Premium members, and others will be placed in queues only when network conditions require it, to make sure the game experience is not destroyed for everyone else trying to play the game.


Records set during the 6 days since start of Open Beta, and a blog post "Stat Scoop"
The purpose of the Open Beta is to test the limits of all the equipment and systems. On that score, the Reloaded Open Beta has been a smashing success so far :) !

The good news is that today we again set new records for beta players in the game, and the bad news is that during the beta phase there are some limits on network capacity, with increased capacity originally scheduled to be ramped up over time (which of course - given the response for the game, will now be sped up - a lot).

To give you some inside stats (we can call this the exclusive blog post "stat scoop") even though I will not expect to normally share many (if any) realtime stats. But given the week we have had I will certainly share some of that; at peak today, after only six (6!) days in Open Beta, we had just under 24,000 peak concurrent players connecting to the game globally (aka "Peak CCU") with a somewhat larger number of players logging in to the game in Europe than in the US, with more than four times that number of unique players logging in to the game during the last 24 hours, each player with an average playtime well over 3 hours per game session (average!). This load also meant the system automatically fired up over 200 district server instances around the globe to handle the load from the gamers.

To put that in some perspective with some of the great games out there (some F2P, some subscription); Eve Online announced back in 2007 that they had reached 35,000 peak CCU, which was 4 years after the game was released. World of Tanks announced in April this year that their PCCU reached 25,000 after a year of Beta testing. Nexon announced that Combat Arms beta had reached 12,000 PCCU in Europe in their open beta running for 10 months from December 2008 to September 2009.

So for us to get to 24,000 PCCU in six days (and WELL over 12,000 PCCU in Europe) is certainly a very good start, especially when you consider its still only Beta, and it has not yet been a week :) (and we have now also capped our servers to boot).

That also means the game is consuming A LOT of servers and network capacity to handle the growth, since unlike games that are super-low-server-impact-per-player (FarmVille, Kingdoms of Camelot etc), APB Reloaded consumes an insane amount of processing resources to keep track of 100 players, 250 cars and over 1000 NPCs real-time per district, and performing a large-scale physics simulated district at 30 frames per second server side. To continue the comparison with Eve, the sever side frame rate of APB is an order of magnitude higher than the server side framerate of Eve. Granted, it's not a fair comparison (Eve is a viscous liquid equation system with click to move mechanics that update rather infrequently, and APB is a frame-rate dependent shooter, so both are great games - just of course doing it very differently, even though CCP's upcoming games might change that). But you guys get the point; there are a LOT of sensitive computations going on in this game at a rather massive scale in our datacenters, which also makes the game somewhat "fragile" to large network issues.

Game Signups and Social Media Growth
The game has also settled into a steady growth pace by adding an average of nearly 1,000 new signed-up game accounts PER HOUR for the past several days, and our Facebook page is getting an average of 6,000 new likes PER DAY (which is great, since we now are using Facebook to post all key game status alerts, which gets info out much faster about game status than just using email, forums and other old skool formats). The page has grown from 48,000 on May 17 to 120,000 on May 29. In fact our facebook page has now surpassed the pages of other great games out there, or is heading toward the same like counts as other games that we all admire. Clearly that increases the pressure many-fold on our teams to live up to the qualities of the games that we are now going to be compared with.

Growth Strategy and Issues
The strategy on the GamersFirst side has been to roll out the game with a large but reasonable amount of investment to avoid starting the project from a big financial hole during the redevelopment phase (since this game certainly had a VERY bad history of not doing too well previously), in particular because this game has massive server side hardware requirements, far beyond most normal MMOs, with several hundred servers to execute the district logic for this volume of players.

But interestingly enough, we actually have plenty of server capacity to spare on hand. In fact right now we have sufficient server capacity on standby that we could power up that will give us 40,000 Peak CCU capacity if not more for the game.

Instead the issues we have seen for the last 6 days in Europe have all been firewall related. Even though the best firewall sales people on the planet will tell you that THEIR firewalls (Ciscos, Junipers, you name them) can handle XX Gbps of traffic, as soon as we throw millions of packets per second of encrypted UDP at any firewall, chances are that the "specs" go out the window. This resulted in the theoretical 7Gbps firewall pair we had in the EU croaking at 737Mb/sec (or less than 15% of the "theoretical" limit) and at much fewer pps than ever "rated."

So, alternatively, we COULD run the servers without any firewalls (that would actually solve the packet loss and firewall congestion issue instantly (!) ). But that would be religiously a pretty bad move. And certainly other game companies have shown that would be a rather bad idea. But, that's often the way this type of issue gets solved (throw the servers out on the net without firewalls). The network performance would surely go up. Be so would the risks to the game and the players.

Therefore, what we are doing instead is a measured increase over time, and also rolling out various changes to our core systems that will permit us to slowly increase all the server caps.

Also, we are cognizant that players can come and go, so we are going to continue performing somewhat deliberate growth during the beta phase and plan to pull out all the stops when all final changes, updates, fixes and features are fully in place.

See you on the streets of San Paro!

Cheers,
Bjorn / TechMech

-----------------

(updated: Sun May 29 11:59pm to add):

Sidenote to clear up minor mis-information on the interwebs?

One of our forum members pointed out a video segment by some random reviewer that got a lot of things wrong about APB, and to top it off referred to Gamersfirst as an Asian company "doing mostly things in the Asian markets." Let me quickly clear that up; Gamersfirst was created in California originally under the name K2 Network, is based in Irvine, and its largest investor is Intel Capital. While we have global operations all over the planet, we only publish games in the regions of North America, South America, Europe and the Middle East. So while that maybe "touches" Asia a little bit, we are actually an entirely a western market focused company and currently do not publish any titles in East Asia. Hope that clears it up (if anyone were to ask :) ). Also - all this information is easily google-able too (for example google: k2 network investment), but I think that would be too much trouble for some people.

Temporary player limits and game queues placed on European APB worlds during network maintenance

A quick update, we are preparing to perform network maintenance in Europe this evening EU time to deal with network packet loss, and while we prepare for those changes we have enabled Queue-ing on the European servers in order to improve network performance for those that are already in the game and playing it. It will take about one hour right now until the player levels have dropped to the point where the network performance should return to normal.

After the network maintenance is finished late tonight, at that point we will remove the limits again. For the future if we ever have to use queuing and limits again, then we will let Premium players bypass them (that is not the case at the moment unfortunately, but will be in place as of tomorrow).

So apologies for all EU players placed in a queue at the moment while we rapidly work to expand network capacity to handle the massive influx of new players.

Cheers,

Patch 88 success, EU network packet loss spikes last night, Premium Re-Ups for Only $7.99 and New items on sale shortly

Today we continue the "blog of full disclosure"

Patch 88 release successful

Last night PST / morning EU time we released patch 88 which performed a lot of minor tweaks to the game. The patch launch went well, and we saw quite an uptick of players after the patch. In fact - we ended up setting a record in EU today with tens of thousands of players logging in to Open Beta simultaneously from Europe alone. The game has been growing at a rate of 50% per day globally for the past 5 days, though we expect that pace of growth to taper off rather soon.

Packet Loss on Europe 1 & 2 networking equipment last night

That giant Open Beta turnout wave caused some unexpected network behavior during EU peak times yesterday. The servers were unaffected (since they scale horizontally and new districts will just start firing up), but we did run in to an unexpected problem with our main network edge devices that caused sudden severe packet loss conditions in Europe, even though the devices were in theory running at only 10% of capacity. 

The result was that between 16:00 and 22:00 CEST on the networking gear connected to Europe-1 and 2 (ie Patriot and Obeya) we saw several sporadic but big spikes of up to 40% packet loss during those hours. 

Inside the game those packet loss episodes manifested themselves as severe rubber-banding, even though ping time and server load would actually remain very low (however, you would be able to see the packet loss conditions if you happened to use the test ping IP we provided on the Forums).

We are working closely with the manufacturer of the networking gear overnight to see if they can resolve the issue before tomorrow's peak time. They are potentially able to provide a fix and if it can be confirmed, I will post the information here as an update to this post.

US-East and US-West did not experience the same issue, partly because those locations have of course been split between two different physical datacenters and in effect have double the amount of networking gear infront of the game servers at each location.

Again, we hope this issue will be resolved very swiftly, but keep in mind this type of testing is what Open Beta is all about - to test the crap out of all the equipment at full throttle. So, while we apologize for the un-fun experience in Europe during that time window, we hope people continue logging in and playing since that will help us troubleshoot the problem in much more detail tomorrow night.

New Items on Sale Tomorrow

This week we are going to start posting a lot of cool new items in the store, and we are going to start introducing some permanent items as well on a rolling schedule. We will announce the new items on Facebook tomorrow so stay tuned (or simply log in to the Armas Marketplace when we launch the items).

Renewal of Premiums during Beta - only $7.99 USD (£4.99 GBP / €5.99 EUR) !

So in the next two days our first batch of Premium Players are going to get reminder emails that their first 30 days are almost up. As we have said before - during the Beta period the cost of premium is only $9.99 for all of the cool premium benefits that let you vastly accelerate your progression and item drops, and gives you complete artistic freedom when you design your characters plus a whole range of other benefits including a 20% item purchase discount. 

Thanks to this discount when Premium Players re-up (adds 30 additional premium days) before the end of his/her premium period, those premium extensions get an automatic 20% off (!) off the Beta $9.99 price. 

The result is that premium renewals are only $7.99 in the US, £4.99 in the UK and €5.99 in Europe. And - any benefits that you accrue, including standing, achievements, customizations, permanent items or permanent weapons you buy during Open Beta carry over to the live production launch of the game as well.

And finally - a big thanks for Open Beta testing our game!

And a final word to everyone trying the game out; a really big THANKS for Open Beta testing our game (and liking us on Facebook!). 

While we will try to keep the game as near production quality as possible at all times during OB, there will of course be some hiccups along the way, and that's expected at this stage of Beta. But we are working around the clock (litterally) to ensure the best possible "near production" experience along the way. In the next days and weeks we will continue performing network updates, traffic improvements, game tweaks etc., while also working on the major upcoming changes that we expect to launch before the big formal release of the game!

Happy Testing!

More Vivox servers and patch 88 tonight Maintenance at 12am PST / 3am EST / 08:00 GMT / 09:00 CEST

Two Wild Days of Open Beta Fun!

Open Beta so far has gone extremely well in most areas, and now we are heading in to the "cleanup phase" for several parts of the game, including post beta optimizations, tweaks and preparation for new features and items.

In a very short time we saw a huge number of players in game to the point where we accidentally overloaded our voice over IP systems earlier today (which is part of tonight's maintenance).

We have also seen our Facebook group rapidly reach 80,000 likes (now beating out other great games such as Star Trek Online, Champions Online, Runes of Magic, Dungeons and Dragons Online and World of Tanks), while setting our sights next on Eve Online (104,000 likes)!

Of course Facebook likes are vanity items and not great indicators of good games (after all Charlie Sheen has 1.3 million likes, not sure he is a "good person"), however for us Facebook is clearly another important channel we use as a communication tool with our rapidly growing community (and you guys of course know how much we DO value our ongoing communication with players!).

So a big Thank You for all the FB likes (!), and please do continue telling your friends about our Facebook Page, and if they have not yet heard of APB then certainly tell them everything you know about APB and San Paro as well!


Maintenance tonight

Tonight at midnight PST (aka early am in Europe) we are performing a quick maintenance to continue rolling out tweaks (patch 1.5.1.88), that will further improve the game.

First - we will be adding more Vivox (Voice over IP) servers tonight on our end, since the incredibly high load and demand for the game had us running short on voice channels today. After the additional systems are added, we should see fewer problems with people unable to launch team based voice chat due to unavailable voice connections.

Second - the new threat levels and matchmaking systems have started working out much better game in the last 24 hours. Still a few more tweaks to go, but now we can see clear behavior trends in the data, and what appears to be some really great matches.

Specifically - Patch 88 has the following changes:

Version 1.5.1.88

Threat Level and Matchmaking Updates

• Fixed an exploit where a leader could gain large amounts of Threat by kicking their team just before winning the mission.
• Fixed a bug where a kicked player could still have their Threat updated as if they had won.
• Fixed the Max Threat Player Notification referring to a ‘Threat Level 30’ player instead of ‘Gold Level 10’.
• Adjusted Matchmaking formula to better fit the new Threat system.
• Added a new ‘Threat Training’ system:
     o    New characters will see their Threat level as ‘In Training’, with a unique Threat icon for up to the first ten opposed missions.
     o    This gives the Threat system some time to home in on a characters skill level before displaying it on the UI and to other players.

Going Forward

Over the next few weeks we will continue tweaking and optimizing the game, and among other things we will be adding more network providers to give people more optimized network routes to reach our servers, and also laying the foundation for the next big wave of core game additions that will give people many more choices in the types of gameplay and engagements they can pick in APB. But we will share more about that after the post Beta cleanup.

And - before I go - we DO hate cheaters (when they are in fact able to bypass things for short periods of time). As I responded in one of the earlier Blog Responses; after this first wave of OB-launch-craziness subsides this week, we are going to start a much more aggressive campaign of public naming-and-shaming of cheaters when we catch them. 

The exact community policing methods are still under discussion, but we CLEARLY want people to realize that they are playing with fire when they are drawn to the "dark side of the force." In fact, we will extend the system to friends of hackers, where if your friend gets caught cheating, you will instantly be a suspect too. The motto is "Friends don't let friends cheat in APB" - and we will work on new methods of Community Policing to deter cheaters in conjunction with the "normal" technical policing we perform all the time. 

As hackers attempt new cheats that work for limited periods of time (and idiots pay the hackers for the cheat - which basically lets hackers install a keylogger on each machine running the cheat), we will eventually detect it (we always do), and at that point we will invalidate everything you have achieved in the game, and eventually we will also suspect anything any of your friends have also done or achieved in game.

Nueume will shortly share more information about the upcoming City Council program and other exciting community events in the near future, and as all these initiatives get underway, we look forward to growing a great San Paro community that can thrive and prosper as a self-governing city in cyber space...

Threat Level update, and matchmaking update at 2:30pm PST / 5:30pm EST / 22:30 GMT / 23:30 CEST

In Saturday's post we noted that the visual threat levels were off given the current curve. We also discovered another critical issue with low-level matchmaking that will be fixed with this current patch as well (fewer wild 1 v 4 matches).

One of the temporary side effects of the change though is that new players will appear to be Silver 2 during their first match before moving around properly. In fact, what SHOULD happen is that for the first few matches the player has "NO" rating, until we have some idea of what direction he/she is moving.

We probably are likely to perform yet another update later this week as we continue tweaking the system.

Today we are patching the game at 2:30pm PST / 5:30pm EST / 22:30 GMT and 23:30 CEST, and this would take about 1 hour.

----- (update) -----
The servers went live again 4:15pm PST after a slight security patch delay. Enjoy the game!

Visual adjustment to Threat Levels coming on Monday

The skill rating system is actually working very well at the moment, HOWEVER, we have realized that the end-user display of the data (the Threat Level Symbol - like "Bronze 1") are out of alignment, in particular since they visually show a very large number of players as Threat 1 for much too long while the system is learning the player's skill. In fact we will start changing the TL symbol much earlier in the process.

It's important to know that a player's REAL skill rating are going to be unaffected by this upcoming change, and this just alters the translation from the real system wide skill rating to the Threat Level display since the current translation skews too low to be practical (in other words, right now there is a VAST difference  between people at the very start of TL-1/Bronze-1 and at the very end of TL-1/Bronze-1. In fact the difference between start of TL-1 and end of TL-1 is greater than the distance between TL-3 and TL-11, which clearly is not an appropriate alignment to communicate how "dangerous" or "skilled" a particular player is).

In the higher threat levels the alignments are a bit more appropriate right now, and therefore will only be slightly adjusted by the upcoming change. And yes - we already have a few players (well under 0.5% at the moment) who have reached TL-30 (aka Gold 10). And they seem like the correct players in that category.

When the new threat level visual chart gets implemented Monday, everyone should see drastic jumps in their visual threat levels at that time (though - again - the underlying skill rating stays the same).

The other tweak we are working on a little longer term is modifying the matchmaking system. Because the new skill rating system produces numbers of a whole different magnitude (and accuracy), with the increased player volume they also don't translate very well into the original match making system. It works, it's just not terribly optimized for the new data, and seems to produce too many 4v3 and 2v3 and 2v4 matches at the moment for lower level players, leading to large backup call sequences. Interestingly, the skill rating gets accurately computed in the end no matter who was in the match (ie - if the match was uneven against you, but you still won - then you will clearly gain a lot from the match, in fact, losing a bad match under some conditions can make you move up in skill rating).

Therefore the other upcoming priority is to tweak how the matches are set up, especially now that we have far more information about a player's REAL skill level than we ever had before. Which is a very nice basis to start this change from. Based on this information we will be able to tweak the matchmaking to create better matches in a district at both low and higher levels.

And, there has been a lot of discussion about how we compute Skill Ratings. Without digging in to extreme detail, the basic paradigm of the new system is that at the end of a match the system takes all the ratings of the various team members, then determines a particular person's "participation" quotient in the match (if you don't participate much, you don't get much of the effect), and then the system updates players ratings in accordance whether they were EXPECTED to have wont the match or not. If one group of players were expected to have won, but still lost, then the penalty is much greater than the award they get if they were expected to have won and did win. If they were expected to lose, but still won, then they are going to get a much higher boost to their skill rating.

We will make a future dedicated post to skill ratings, and also start posting some of the data from the real live games so people can get a sense of how these matches play out and how players are shifting in skill ratings over time.

Finally - today for the past several hours we have had well over 250 registrations PER MINUTE for the game, and we expect all these new players to arrive in full force on Monday (except of course for the super-double-secret-early-invites we are doing before Monday morning - but those are of course a big secret). But Monday is also the day we will be formally reaching out to new players, so this coming week certainly should be very interesting.

Cheers!