Colliding with the GTA V train (and getting back up again) - plus - details of APB's upcoming massive engine upgrade.

First let me get this out of the way; we think GTA V is a very good game. We always look for great competing products when we figure out what we should be working on next, and clearly GTA sets the bar very high.

The GTA express train
The September 17 release of Grand Theft Auto V, and the follow on October 1 release of GTA Online had a massive impact on the whole game industry.

Using Steam Data as a general PC-activity level sampling tool (keeping in mind Steam only tracks activity for those game installs that originated from Steam itself and not from publishers' websites, which means it only shows a small portion of gamers in non-Valve games, but shows 100% of gamers in Valve created games) it appears Online-PC games in Steam's "western-market areas" took an average 30% percent hit in normal average peak CCU levels when players jumped onto the GTA V bonanza.

Given GTA V broke the record for top grossing 24 hour sales in the history of ALL entertainment, one would expect the wind to get knocked out of the rest of the game industry for some period of time, especially for action-oriented games like Saints Row, All Points Bulletin and Planetside.

Steam Peak CCU for players accessing PC Online games through Steam as a before-and-after
comparison of GTA V's impact on the overall online game market.

More specifically, because APB Reloaded is also very similarly themed to GTA V with the same player demographic (even though APB is far more skill and map knowledge based, and far less 'arcad-y' and 'aim-assist-y' than GTA), we were pretty much standing on the GTA train-tracks waiting for a possible collision (and GTA of course having a much 'shinier' and newer engine, much larger development resources, deeper content and years of iterative development under its belt made it a hard challenge to face).

When the dust finally settled last week, it actually turned out that APB Reloaded was not hit any harder than the average Online PC game (again based on before-and-after comparisons of average PC game Steam numbers for the top-100 PC Steam games). Not to say we didn't take a dent. But we are still here. Still plugging away. And still working on our ambitious long term plans.

Slowly but surely, starting last week, we also started seeing players returning to APB, which has helped us breathe a little bit easier this week. Our giant super-secret engine update that we have been working on since July depends on players continuing to support our game, so we are incredibly thankful that players are returning to APB once they have gotten their GTA fix.

Introducing Tiggs

For those who have followed APB Reloaded closely over the last several months, we recently made some significant team changes.

In September we introduced the APB community to a person we think is the best community manager on the planet (Tiggs, with 20+ years of online community management experience dating all the way back to Ultima Online and Star Wars Galaxies) and we also appointed Joe "Linus" Willmon to be the the new head of APB's production team. We also concentrated our internal efforts on the newly begun stealth engine upgrade mission, and numerous improvement and performance items.

Tiggs and her new team has also done an AMAZING job of keeping the community up to date, and leading the efforts around tracking down performance issues and continually updating the players of ongoing bug fixes and updates.

Our massive "Secret Engine Upgrade Mission"
APB clearly was born with tech in an era that pre-dates GTA V by one full ‘game and engine generation.’ As those who have followed us for the last few years know, APB Reloaded uses Unreal 3 version 2007 and 2008 as its primary game engine (basically the version of the engine used by the original Unreal Tournament 3 that came out in 2008). Therefore the upgrades mentioned on the Unreal 3 Wikipedia Page do not apply to APB's original engine.

Due to the vintage of the engine, we keep bumping up against performance limitations, and also severe limitations in our product development speed. Because we use many custom systems in the game, it takes a very long time for our team to make material changes to the game, resulting in a lot of features being shelved or delayed.

An upgraded engine would let us tackle important things like min-spec system performance, lighting system improvements, net code improvements and new graphic card optimizations and it would likely lead to a faster content creation pipeline by taking advantage of new systems available in the updated engine.

The team performed small upgrade tests throughout 2012 and then Johann "Aphadon" took a serious stab at the task in April 2013 to test if we may be able to fix just 'some' of the older rendering code that causes some of the frame stutter (for example with some particle effects that have been shown to have some serious performance problems). But taking that approach turned out to cause other big follow-on problems when fixed in isolation. So to combat the overall problem, the drastic decision was then made to let a part of the team go off and map out what it would take to upgrade "everything." And with that, we really do mean "everything"

July 17, 2013 marked the kick off of the massive engine upgrade to move the engine forward in time by 5 years, from the early 2008 version of the engine to the 2013 version of the engine, which is the version that contains all the improvements Epic created for Gears of War 3, plus 2 additional years of engine refinements after those changes and optimizations. Clearly Epic has worked very hard in recent years to allow really small targets (even targets as small as cellphones as evidenced with the release of Infinity Blade for iOS/Android using a modified Unreal Engine)

Gears of War 3 version of Unreal 3, circa 2011 (Image: IGN)

The 2013 version of the 'vanilla' Unreal engine contains about 25,000 source code files of various types. The original APB game and custom engine contains 61,609 files in total, out of which nearly 35,000 were source code files (the other tens of thousands of files are content and config files). The mission; merge these terrifyingly massive projects. To complete the merge it turns out that our team has to merge roughly 10,000 source files, representing well over 1,000,000 lines of code. And then get them to compile. And then fix all the dependencies between these files, and fix features and functions that end up breaking as a result of the merge.

By end of last week (October 11, 2013) the team had finished the basic merge pass of 9,393 files, which leaves them only a couple of hundred files to go. Once the team gets everything to compile, then the long and painstaking effort of solving for issues and dependencies get underway (not to mention the massive QA and testing effort).

An upgrade of such a customized game as APB presents some particular issues. The first one is that RTW originally created a ton of code that added custom versions of features that were not natively supported by the engine. Many of those features (day-night cycles, building systems etc.) were later independently created (and optimized) as engine features by Epic, but by then the RTW code path had already diverged from the original engine code. In 2007-2008 adding these items (like day-night cycle to textures, and city building tools) were huge engineering feats. Some of the custom code lead to the development of some really amazing functions that remain top of the line, like the character and vehicle customization systems. Of course it also lead to being dependent on certain ‘dead-end’ custom features, that are hard to break the dependencies for after the fact.

The new Lighting Systems supported by the upgraded Unreal 3 engine

The other big issue is that Epic traditionally performs a massive amount of optimization and performance work around each of their own releases for their own games. Most notable Gears of War 3 that came out in 2011, several years after the APB engine version we use today. The GOW-3-optimized engine is also sometimes referred to as "Unreal 3.5" given the very large amounts of code (network code, light systems) optimized in preparation for Epic's Gears release.

Of course the APB code base (to date) contains none of those upgrades (bar a few items we added such as some net code from a later engine and of course the singular integration with Scaleform). And in the last few years as Epic began to start supporting mobile and tablet devices (as evidenced by Infinity Blade) the engine started supporting incredibly optimized and compact build targets, something that was not really a focus in the original engine.

The upgrade timeline, and opening the door to next gen options
Aphadon (our lead developer on this task) will pen a more detailed technical post about the upgrade in the next month or so when we are getting closer to something that's actually testable. Also, as we get closer to launch we should be able to predict the performance (and optimization) characteristics of the upgraded engine.

The current roadmap puts a full 'upgraded APB engine' sometime at the end of Q1, 2014, but as with all roadmaps, that could potentially change. The team is clearly working insanely hard on this particular task, and we think it will finally allow us to also unleash a lot of potential that has remained locked away in the custom code the game has been saddled with. As an added bonus, by upgrading the engine, we also open some interesting possibilities, including a path to launch on other platforms in the future.

Next Up
Linus will be penning a separate post about the ongoing server-performance testing that's led by Tiggs and her team, as well as the Halloween stuff coming very soon.

Also, I am personally very curious about Rockstar's decision to sell in-game currency directly in GTA Online, especially given the many reported GTA Online glitches that basically could make the whole currency purchase process moot. It's a very interesting idea, so I may pen a whole post just about their add-on transaction system. It's something we are looking at very closely.

And as always, thanks for continuing to support APB for the past several years, and with these massive coming changes, we hope to continue keeping APB strong for years to come.

/TechMech