Colliding with the GTA V train (and getting back up again) - plus - details of APB's upcoming massive engine upgrade.

First let me get this out of the way; we think GTA V is a very good game. We always look for great competing products when we figure out what we should be working on next, and clearly GTA sets the bar very high.

The GTA express train
The September 17 release of Grand Theft Auto V, and the follow on October 1 release of GTA Online had a massive impact on the whole game industry.

Using Steam Data as a general PC-activity level sampling tool (keeping in mind Steam only tracks activity for those game installs that originated from Steam itself and not from publishers' websites, which means it only shows a small portion of gamers in non-Valve games, but shows 100% of gamers in Valve created games) it appears Online-PC games in Steam's "western-market areas" took an average 30% percent hit in normal average peak CCU levels when players jumped onto the GTA V bonanza.

Given GTA V broke the record for top grossing 24 hour sales in the history of ALL entertainment, one would expect the wind to get knocked out of the rest of the game industry for some period of time, especially for action-oriented games like Saints Row, All Points Bulletin and Planetside.

Steam Peak CCU for players accessing PC Online games through Steam as a before-and-after
comparison of GTA V's impact on the overall online game market.

More specifically, because APB Reloaded is also very similarly themed to GTA V with the same player demographic (even though APB is far more skill and map knowledge based, and far less 'arcad-y' and 'aim-assist-y' than GTA), we were pretty much standing on the GTA train-tracks waiting for a possible collision (and GTA of course having a much 'shinier' and newer engine, much larger development resources, deeper content and years of iterative development under its belt made it a hard challenge to face).

When the dust finally settled last week, it actually turned out that APB Reloaded was not hit any harder than the average Online PC game (again based on before-and-after comparisons of average PC game Steam numbers for the top-100 PC Steam games). Not to say we didn't take a dent. But we are still here. Still plugging away. And still working on our ambitious long term plans.

Slowly but surely, starting last week, we also started seeing players returning to APB, which has helped us breathe a little bit easier this week. Our giant super-secret engine update that we have been working on since July depends on players continuing to support our game, so we are incredibly thankful that players are returning to APB once they have gotten their GTA fix.

Introducing Tiggs

For those who have followed APB Reloaded closely over the last several months, we recently made some significant team changes.

In September we introduced the APB community to a person we think is the best community manager on the planet (Tiggs, with 20+ years of online community management experience dating all the way back to Ultima Online and Star Wars Galaxies) and we also appointed Joe "Linus" Willmon to be the the new head of APB's production team. We also concentrated our internal efforts on the newly begun stealth engine upgrade mission, and numerous improvement and performance items.

Tiggs and her new team has also done an AMAZING job of keeping the community up to date, and leading the efforts around tracking down performance issues and continually updating the players of ongoing bug fixes and updates.

Our massive "Secret Engine Upgrade Mission"
APB clearly was born with tech in an era that pre-dates GTA V by one full ‘game and engine generation.’ As those who have followed us for the last few years know, APB Reloaded uses Unreal 3 version 2007 and 2008 as its primary game engine (basically the version of the engine used by the original Unreal Tournament 3 that came out in 2008). Therefore the upgrades mentioned on the Unreal 3 Wikipedia Page do not apply to APB's original engine.

Due to the vintage of the engine, we keep bumping up against performance limitations, and also severe limitations in our product development speed. Because we use many custom systems in the game, it takes a very long time for our team to make material changes to the game, resulting in a lot of features being shelved or delayed.

An upgraded engine would let us tackle important things like min-spec system performance, lighting system improvements, net code improvements and new graphic card optimizations and it would likely lead to a faster content creation pipeline by taking advantage of new systems available in the updated engine.

The team performed small upgrade tests throughout 2012 and then Johann "Aphadon" took a serious stab at the task in April 2013 to test if we may be able to fix just 'some' of the older rendering code that causes some of the frame stutter (for example with some particle effects that have been shown to have some serious performance problems). But taking that approach turned out to cause other big follow-on problems when fixed in isolation. So to combat the overall problem, the drastic decision was then made to let a part of the team go off and map out what it would take to upgrade "everything." And with that, we really do mean "everything"

July 17, 2013 marked the kick off of the massive engine upgrade to move the engine forward in time by 5 years, from the early 2008 version of the engine to the 2013 version of the engine, which is the version that contains all the improvements Epic created for Gears of War 3, plus 2 additional years of engine refinements after those changes and optimizations. Clearly Epic has worked very hard in recent years to allow really small targets (even targets as small as cellphones as evidenced with the release of Infinity Blade for iOS/Android using a modified Unreal Engine)

Gears of War 3 version of Unreal 3, circa 2011 (Image: IGN)

The 2013 version of the 'vanilla' Unreal engine contains about 25,000 source code files of various types. The original APB game and custom engine contains 61,609 files in total, out of which nearly 35,000 were source code files (the other tens of thousands of files are content and config files). The mission; merge these terrifyingly massive projects. To complete the merge it turns out that our team has to merge roughly 10,000 source files, representing well over 1,000,000 lines of code. And then get them to compile. And then fix all the dependencies between these files, and fix features and functions that end up breaking as a result of the merge.

By end of last week (October 11, 2013) the team had finished the basic merge pass of 9,393 files, which leaves them only a couple of hundred files to go. Once the team gets everything to compile, then the long and painstaking effort of solving for issues and dependencies get underway (not to mention the massive QA and testing effort).

An upgrade of such a customized game as APB presents some particular issues. The first one is that RTW originally created a ton of code that added custom versions of features that were not natively supported by the engine. Many of those features (day-night cycles, building systems etc.) were later independently created (and optimized) as engine features by Epic, but by then the RTW code path had already diverged from the original engine code. In 2007-2008 adding these items (like day-night cycle to textures, and city building tools) were huge engineering feats. Some of the custom code lead to the development of some really amazing functions that remain top of the line, like the character and vehicle customization systems. Of course it also lead to being dependent on certain ‘dead-end’ custom features, that are hard to break the dependencies for after the fact.

The new Lighting Systems supported by the upgraded Unreal 3 engine

The other big issue is that Epic traditionally performs a massive amount of optimization and performance work around each of their own releases for their own games. Most notable Gears of War 3 that came out in 2011, several years after the APB engine version we use today. The GOW-3-optimized engine is also sometimes referred to as "Unreal 3.5" given the very large amounts of code (network code, light systems) optimized in preparation for Epic's Gears release.

Of course the APB code base (to date) contains none of those upgrades (bar a few items we added such as some net code from a later engine and of course the singular integration with Scaleform). And in the last few years as Epic began to start supporting mobile and tablet devices (as evidenced by Infinity Blade) the engine started supporting incredibly optimized and compact build targets, something that was not really a focus in the original engine.

The upgrade timeline, and opening the door to next gen options
Aphadon (our lead developer on this task) will pen a more detailed technical post about the upgrade in the next month or so when we are getting closer to something that's actually testable. Also, as we get closer to launch we should be able to predict the performance (and optimization) characteristics of the upgraded engine.

The current roadmap puts a full 'upgraded APB engine' sometime at the end of Q1, 2014, but as with all roadmaps, that could potentially change. The team is clearly working insanely hard on this particular task, and we think it will finally allow us to also unleash a lot of potential that has remained locked away in the custom code the game has been saddled with. As an added bonus, by upgrading the engine, we also open some interesting possibilities, including a path to launch on other platforms in the future.

Next Up
Linus will be penning a separate post about the ongoing server-performance testing that's led by Tiggs and her team, as well as the Halloween stuff coming very soon.

Also, I am personally very curious about Rockstar's decision to sell in-game currency directly in GTA Online, especially given the many reported GTA Online glitches that basically could make the whole currency purchase process moot. It's a very interesting idea, so I may pen a whole post just about their add-on transaction system. It's something we are looking at very closely.

And as always, thanks for continuing to support APB for the past several years, and with these massive coming changes, we hope to continue keeping APB strong for years to come.

/TechMech

End of Week 5 Update : A GTA Style Story, or a Counterstrike Style Match?

Games modes - who needs em?

Let's talk a moment about the fundamental concept of APB, and how it is NOT like a multi-player Grand Theft Auto game (aka how it is NOT "GTA Online"), at least not yet.

When the team designed APB back in the day it appears one of the goals for the game was truly to create a Grand Theft Auto "Online" inspired game with two opposing sides duking it out for supremacy. No problem with that in theory. Except that a multi-player GTA clone that preserves any semblance of story progression (or Red Dead Redemption clone for that matter) is a really hard thing to pull off for one basic reason; people on the "opposing" side will never do anything that resembles "story-line-reenactment," and more fundamentally, the whole GTA concept, while open world-ish in nature, is still at its core a story-driven progression with a logical beginning and end. The great stories themselves is really what make the GTA games brilliant on their own (after all, how many old ladies can you run over before you want to put the rampage behind you and start experiencing some story progression?). As an aside, I am still a bit confused about the story-value of the Zombies in RDR's Zombie update after the brilliant first act of RDR - but that's a whole different discussion.

So to put it in perspective, imagine yourself being one of the NPC characters in one of the GTA games, ie - not playing a protagonist like Niko Bellic in GTA 4 or Claude Speed in GTA 2, but instead playing the guy who works with Governor Johnny Rotten, or working for Jerkov in the Russian Mob. If you in fact PLAYED that character, you would basically sit around and do nothing 99% of the time, while waiting for your "scene" to occur, at which point you would add your cool flavor to the game and then get shot. No sane gamer would actually want to act out that role (it would be like a really boring version of a Civil War reenactment). But that kind of single-player progression design is what makes those games brilliant from the protagonist's point of view (and duds for the NPCs). One of the complaints about APB was the fact that sometimes one side in a conflict were basically asked to act as story progression for the other side, and that didn't always seem to work out very well.

On the opposite side of story driven games are the hard-core Player vs. Player games that basically throw away the story. Counterstrike, Team Fortress 2, Unreal Tournament 2004 even our own War Rock and to some extent Knight Online. Is there a story in any of those games in their primary game modes (ie the mode that 99% of players use daily while whacking away at each other with guns or swords)? Neh. Not really. Sure there are some allusions to a world out there (what's the Tournament thing in UT anyway?, and why is the Spy in TF2 French? and what's the "War Rock" meteor that apparently crashed to earth from the heavens? - who cares?), so basically the storyline is almost completely irrelevant to the core game play and the core fun. But those are all still brilliant games, thanks to tight level design and the ability for players to have a completely new game experience with each respawn and by focusing on fast-action strategy, the game becomes more akin to chess on steroids (you know the board, but you don't know what your opponent will do), rather than a cinematic story progression.

That creates a dilemma for the current incarnation of APB. Because it tried combining what is really two separate types of games into one game mode (story plus PvP; where PVE style NPCs give you orders to do stuff that lead to PvP confrontations), the game originally felt like it hadn't quite made up it's mind yet on what type of gamer interaction it expected (either the ultra-detailed player-and-team-control demanded by PvP, or the progression driven feel of story-focused PvE?) .

APB actually has the beginnings of a lot of the elements that will make it a really successful game in the long term, and could in theory exploit (maybe "complete") both types of interactions; there is ACTUALLY a giant back-story to the entire city (100+ pages of it), there are deep anthologies of the various factions in the city, and there is even the foundation for a single-player mission mode, which would in essence turn one of the training missions into something more akin to GTA. And if we were to go further in that direction, then there is really nothing stopping us from going "Law and Order" on the whole thing (ie - episodic story-driven content meant to be experienced either by an individual in single-player mode, OR by groups of people in a more traditional PVE/collaborative mode where teams or clans go "raiding" together and battle AI opponents that do more things than just run away like the current inhabitants in the city).

On the other hand, there is no reason the PvP fighting shouldn't also be extended to it's logical conclusion, and we are considering starting a new mode where we hand back direct control to the player, a la Counterstrike. Potentially we would call that "Session Mode," "Clan Mode" or "Team Mode" or something similar. Of course this type of mode risks breaking the design-continuity between the Social Districts and Action Districts, but fundamentally, there is no good reason you should not be able to have more of a lobby-type function where your clan can take on a competing clan (even Crims on Crims), and have a show-down in a new (presumably smaller) map until one clan or team stands victorious. It would be like challenging someone to a duel at dawn, posse vs posse. And if you don't yet have a clan, what better method to find future clan mates than by permitting a more user-controlled joining process that enables short, fast paced game instances that can be completed in ten to twenty minutes.

There is even a third design philosophy; PvE collaborative mode using fast-paced (minimal-story) PvP mechanics. The closest here would be a Zombie style mode (hah - didn't I just question Zombies earlier?). The idea would be that you and some buddies should be able to show up at a building in the Social District, and upon entering you descend into a Resident Evil style nightmare clearing Zombies (and not a whole lot of story progression beyond shooting). This third mode is really just a PvE event with minimal story line, but would be something that provides more variety for the gamers, since it's a really great "introductory" collaborative mode mechanic (everyone is a winner, except the dead zombies who are all losers).

Our end goal is clear - we want people to become so engaged in the activities we will make available in San Paro that gamers want to "hang out" in some part of the city almost all the time. There should be a huge variety of interactions (PvP, PvE, collaborative, single-player, storylines, social interactions etc.) and  players should really drive what modes we spend our time expanding and increasing in size (and what modes we abandon).

For the first release we will clearly not have had a chance yet to implement much, if any, of the above philosophies, however, this type of expanded game play is what we are planning out for the next 12 months.

Next week there will be no Blog Update (since it is Christmas week after all), but I will pen a short New Years update, and then we are planning to publicly release the first official launch schedule during the first week of January, as well as giving you more details of what will be in the various CB, OB and Live releases.

Cheers,