The Missing Skill-Ratings, and a Thanks to Hackers

Ok - so time has come to lift the covers off one of the core problems in APB; the lack of meaningful ratings that underpin the "threat" and matchmaking systems.

There are various important core issues that the team is working on (such as spawning in your enemy's face in particular, which the team has been working on some really elegant solutions for), but the lack of real ratings is critical since it causes a ripple-effect of problems. Matchmaking now consist of a very smart and sophisticated matchmaking algorithm, paired with a very "basic" (and maybe even "dumb") rating algorithm.

We are fixing all this as one of the reasonably early Open Beta tasks (not the very first OB release, but in the second round of OB or so), and at this point one of the core feedback items from Closed Beta is - "that's some crappy matchmaking." We agree. Let me explain. 

No Real Skill-Rating
For a game that has some really cool and well engineered systems, it was a bit of a shocker to discover that in spite of relying almost exclusively on computer generated match making (unlike games like Counter-Strike where you make your own matches), no real thought was made to create a decent skill-rating system to feed those match-making decisions.

APB has a slew of different level and progression measurements visible to the player with titles like "threat", "rating" etc. This gets terribly confusing since "rating" as used in the APB UI actually means "progression" (or in MMO terms really "Level," not to be confused with APB's use of the word "Level"), and "Threat" is actually supposed to be a rough representation of "skill-rating" in more traditional game language.

Right now "Threat" - ie what 'should' be rating and what is the critical value used by the matchmaking system - is computed using a mechanic that tries to emulate Premiership League standings (or "soccer" to all the Americans reading this). A win gives you 3 points, a draw 1 point and a loss 0. Then APB takes that score over your last 50 matches, with a focus on the last 20 and figures out if you are above or below your opponents using this crude straight-win system. No regard is paid to how many matches you have played (is this match 52 or match 952?) and no regard for what level of player you have beat/lost to in the past. All wins and losses are treated equally. Even if you were to beat uneven odds (like 1 v 3).

Thus if you have 20 wins in a row, your "threat" (your skill-rating) balloons. Even if all you did was beat 20 newbies. And you will then be tossed into 1 v 3 or even 1 v 4 scenarios as you have a high threat level.  Killing newbies still shoots your "threat" level through the roof. If you had close fought losses against true veterans, those losses are still considered as bad as if you had lost to some really bad players.

Since there is no consideration for the skill of those you beat, the number of people you beat, nor the manner in which you beat them, you are not ever going to hone in on a TRUE skill rating in the current system, and it becomes easy to game the system. For anyone familiar with the XBox True Skill system the above system probably doesn't make a whole lot of sense. We agree.

Chess, Glicko and TrueSkill
The fundamental idea behind a good rating system is to let people slowly hone in on the rating they truly should be over a long period of time, and then try to predict matches that will provide the best experience to the players involved.

I have been involved in many different game companies in the past, and we always ended up going back to Chess ratings as the foundation for our rating algorithms ( http://en.wikipedia.org/wiki/Chess_rating_system ). People have been extending chess ratings for a long time, most recently Microsoft's implementation of the TrueSkill rating system, which is a rather complex system, but fundamentally is an expression of Chess ratings applied in online matches.

There are less computationally intensive rating systems such as Glicko ( http://math.bu.edu/people/mg/glicko/glicko.doc/glicko.html ) which strike a balance between the "drifts" that tend to occur in pure Chess ratings and the complexities of the TrueSkill implementations.

So part of what we are trying out, will be a system that uses a LOT of the different characteristics of the above rating systems. It is still going to be a little while with the current wild Threat system (sure - go ahead and exploit it, lose 20 matches in a row, and then go kill newbies if you like since your threat will suddenly seem "low") - but clearly it's a top priority to get this worked out in Open Beta.

This leads to another requrement...

User Segmentation
In Most MMOs (like in Knight Online, another game that we publish) people tend to get grouped into distinctly different combat segments. In KOL the groupings are for levels 0-35, 35-60 and 60-83. The groupings divide players out based on the types of equipment they have and the amount of progression they have made in the game.

In APB this isn't strictly applicable given the hybrid nature of the Shooter/MMO and the greater emphasis on skill rather than progression, but at minimum "protecting" lower rated players from the being "hammered" turns out to be really important.

Therefore the next big thing is going to be creating some basic "skill-level districts." If a player doesn't 'volunteer' to join a higher skill-rated district, then he/she will be kept in the district with the most appropriate skill-ratings for his/her current skill-rating. We'll share more details on how this will be handled, as we get closer to releasing game changes that incorporate this.

Sure a lower level player can VOLUNTARILY join a higher rated district (maybe his/her friends are there), but the default action would be to put you in a district that most closely matches your current skill.

User Control Debate
One of the key items we keep debating internally is the amount of end-user control we should expose to the players in parallel with the Open World systems that will always be the core of the system.

In other words, how "counter-strike-like" we should make the start up of some matches. In Counter-Strike, since you can determine the exact nature of the team and your opponents, you actually are less dependent on a good rating system. So, one potential fix is to permit players the ability to queue-up and start their own fights (of course NOT removing anything from the current city or mission systems, just adding this ability in addition to the the mission systems).

Even though that might seem counter to some of the original APB ideas of launching things under the veneer of the living city, giving people direct game control would potentially let players set up matches that the system would never really promote on its own (who would NOT like 14 v 14 fights if they were available?) Our addition of Chaos rules and Turf Wars certainly adds some elements of session fights, and Chaos and Turf Wars are certainly our next big focus, but in addition to those enhancements, we are also debating if we should add direct match tools to the game. That way you can decide to play Open Missions (the current system), Chaos/Turf Districts or Player Matches.

We are happy to take feedback on this point, though I have a feeling that the true feedback would come in terms of a user trial (ie - we might put some of that in there and see where players go). The timing however, is obviously a little foggy, given the enormous amount of changes we are already working on and the huge backlog of game changes we are working on right now.

Finally - Thanks to Hackers
For the past 3 weeks we have been watching and observing user behavior in Closed Beta. We'd like to extend a "thanks" to the 60 odd players that have been toying around with various hack tools (about 0.4% of the players). Thanks to your hard cheating work, we are now much better equipped to deal with you going forward. How? I guess you will find out.


Cheers,