Server Side Lag Solved
First up - as of this week we have officially sorted out all the items related to server side lag that were killing some servers at various strange intervals (5, 10 and 20 minutes generally). Without going in to extreme technical detail, the issue turned out to be the method used to invoke various external libraries that under certain conditions could cause network congestion on the server itself. That issue was fixed in the last 10 days, and combined with new networking equipment from Internap that optimizes the route selection from individual players, most server side lag conditions have now been completely eliminated.
The network routing optimization boxes are quite cool, as we were able to observe in real time some players connecting from Australia to Frankfurt (maybe not the best idea to connect that far) and see them drop from 1500ms ping times to 500ms ping times. Of course the 500 we cannot do anything about (given the physical distance), but the 1000ms improvement was very significant.
Will you or some of your friends still see game lag? Sure - under three conditions; (1) your home network runs in to congestion, (2) your ISP has a freak-out and messes up its routes to its peers or (3) your home computer is not up to spec and you end up lagging all the time (but that's of course client side lag which is a whole different beast).
NVidia SLI support in the new 280.26 WHQL drivers
NVidia notified us this week that the new WHQL drivers now have the APB profiles included. This is the link to the drivers. The SLI profiles are NOT listed in the release notes, but, if you install these drivers and attempt to run the game using NVidia SLI card config, you will see that the game FINALLY runs as expected.
Download the final release drivers from here:
http://www.nvidia.com/object/win7-winvista-64bit-280.26-whql-driver.html
(and again - the listed games in their notes is for NVidia 3D vision support, something we have not yet added, but is also on our list of optimizations to come at some point in the future).
Threat Level Revisions
A few weeks ago we released code which altered the matchmaking, but in doing so, the code also (actually at my request) included a new 'uncertainty' modifier value which basically made the matches made with new players slightly more 'uncertain' (and for a longer discussion about the matchmaking changes you can look at the earlier blog posts).
Due to some interesting math behavior - the rating system has been properly rating people (so everyone on a relative basis has the proper rating). Here is a histogram of a set of 15,000 ratings from one world from one day in the game using the RAW (underlying) rating data.
Keep in mind that this data DOES NOT equal the game Threat Levels (as you can tell from the above diagram, the ratings are from negative to positive rating levels and don't line up with TL values). However for those who like math, the ratings DO fall on a nice even bellcurve, which in fact IS the expected outcome of our rating system.
So if the RAW ratings look nice and curve-like, then what on earth went wrong when 'translating' the ratings to in-game Threat Levels?
In short - the location of the middle of the bellcurve is NOT where we had expected it to be when we implemented the formulas. In fact - in the live data the bellcurve center appears around the value "5.5" and not at the expected value (which was expected to happen much higher up on the translation scale). There are a lot of math reasons for this 'real world' outcome (after all - that's what we use Beta to figure out), some of the reasons had to do with the formulas and parameters used, and some were the result of the APB mission system and how players of different ratings group for missions.
So what happened in the current translation in game? Well - we accidentally ended up with THIS lopsided translation to threat levels that people saw in the game:
What actually happened was that the translation from RAW rating data to visual game information (threat levels) pushed 35% of players into TL 1 (aka Bronze 1). In fact a total of 50% of ALL players were rated Bronze 1, 2, 3 (!). That means there is an ENORMOUS difference in skill between players within those three rating levels - and then everyone else were spread out across the rest of the spectrum. Or another way to think of it - the difference in skill between the lowest Bronze 1 player and the highest Bronze 3 player was just as great as the difference in skill between a Bronze 4 player and a Gold 10 player (!).
Clearly that was NOT the expected amount of players in those threat levels, and the TL values became pretty useless to gauge overall skill between players.
In spite of the above translation issue, even though the underlying RAW data was correct, the visual translation into the actual game certainly did not work as intended, and has lead to a LOT of concerns and complaints in the game.
As a result - with patch 116 we have re-translated all the RAW data, into new TL values. When 116 goes live, then the ACTUAL threat level distribution will look like this:
The vast majority of players will end up in the range Bronze 6 to Silver 8 (80% of players), and then players above (Silver 9 and up) or below (Bronze 5 and less) will be quite rare (roughly 10% above and below that middle set of values).
Silver 5 will be the most common rating, with 9% of players being a Silver 5.
So when you log back in to the game tonight - don't be surprised that your level has changed. In fact - it may have jumped from a Bronze 1 to a Silver 2 (if you were at the very top of the Bronze 1 range).
Therefore the new threat translation should give you a lot more useful information about the matches you have with opposing players.
Will there be other TL related issues in the future? Potentially - but the plan going forward will be to re-balance the TL display at least monthly.
Hong Kong Premium Server Launch, and the upcoming Russian and CIS world.
Tonight we also enabled the "premium" server in Hong Kong. This is entirely a system meant to give low-latency access to a region that we don't really support yet (given the very high prices for connectivity in the region), and you can ONLY access this server if you are a premium player. That's a little counter-intuitive for a Free2Play© game, but, since you can continue playing for free on the Los Angeles server (where our cost of operations is 10X lower than in Asia), we felt that we would offer a premium access choice for those who really want low ping times to the games locally in that region.
If this experiment works, then we will also consider adding an Australian Premium server given the tremendous amount of interest we seem to have from Australia and NZ as well, where the cost of internet service is equally astronomical.
We have made progress on launching the game in Russia and CIS as its own separate territory, and I will update the blog when we have more details on that launch. When that happen we will also be able to re-enable the use of Cyrillic characters in chat, game names etc. for the Russian and CIS worlds and will migrate all Russian characters over to that world.
Going to GamesCom or GDC Europe?
We do have a contingent at GamesCom and GDC Europe. If you are going as a gamer (to GamesCom) or an industry person (to GDC), certainly feel free to come by.
We have a pretty small presence at GamesCom (mostly for 'fun') but we WILL be giving out quite a few free codes and other swag, so stop by the APB 'alley' if you plan to be at the conference. You can also meet with some of the APB Devs, Artists and Designers who will be there handing things out.
Patch 116; Various fixes plus 'spammy popup reduction'
For patch 116, please see the patch notes on the forums, but one of the bigger changes involves reducing the annoyance factor of the popup we added in earlier builds meant to point out the clear difference between premium and non-premium players. We may have gone a LITTLE bit wild (yes I know - maybe VERY wild) with the earlier version of the popup with the sheer amount of notifications. In build 116 we relaxed the spammy nature of the notifications to a much more manageable level.
In the new build the popup does NOT appear in the Tutorial (which was a little over the top), and does not appear when you hit Heat 5, and it ONLY appears after 50% of the missions during normal play. Of course getting a premium account completely removes the popup (yay!), just like premium cable TV has no commercials.
However - for all the people that expressed their concern about this popup, it is worth noting that premium conversions JUMPED nearly 40% over the old conversion rate when we added this notification (potentially people didn't realize what they were missing :) ), so we are going to continue using the popup (albeit in a much milder form). After all, this increases our ability to launch more stuff by being able to increase our staff to work on all the remaining issues in the game. After the popup was added about 33% of all the CCU at any given moment on any server are now Premium players, and the number is growing.
Future Changes
Next up - we are continuing to work very hard on optimizations and exploit fixes as well as the next set of feature changes. Based on some really interesting work we are doing right now we have been making some tremendous progress and at some point (when things thoroughly QA-ed and ready) we plan to release of a lot of changes simultaneously, which likely will take the edge of many current and future exploits. I won't be sharing more than that until it's ready to go (given the enormous amount stuff involved), but as soon as we are ready we will share much more data.
Finally the new features (ClanWarfare, Racing and Asylum) all are progressing, though with a relatively small team progress is taking a little bit of time. Once these features are close to release, we will share that here and then we will start preparing for the final 'formal' (non-beta) release of the entire APB Reloaded game.
Stay tuned - and see you at GDC and/or GamesCom in case you are stopping by!
Cheers,
