The Ups and Downs of Updates

Update 2 is coming soon… Read our first code dev blog as Simon discusses the ups and downs of updates!

With all the work that’s been continuing at Stainless Games to improve Carmageddon: Reincarnation since its release, we thought it was time to also resurrect our blog. We wanted to give you a clearer idea of the challenges we’ve encountered along the way and explain why it’s not as easy as some of our critics seem to think to increase fps or improve the look of the game. As a small indie dev team everyone has been pulling together magnificently to fix reported issues and help as many players as possible to enjoy the game. Our first blog comes from Simon Lacey, as he explains how we worked to increase the fps significantly between release and our first update. Simon is pictured below, testing the motion blur code…

Hi, I'm Simon, the lead programmer on Carmageddon: Reincarnation. I've been asked to write a blog entry describing a bit of the story behind the poorly received performance of the 'FULL RELEASE DAY!' version of the game (Build 7039) and the big performance step in 'Post Launch Update #1' (Build 7470).

It can be quite tricky developing on PC. You never know exactly what you are going to be running on. The best you can do is to run the game on as many different machines as possible, and hope the coverage is broad enough to minimise how many people are left with a crappy experience.

We have a range of machines at work; we have a couple of dozen developers regularly playing the game, but as developers, quite a lot have pretty decent machines, especially the artists that have to work with large uncompressed assets. In addition we have a selection of machines specifically chosen to cover the range from minimum spec (or indeed a little below) up to Titan powered machines. There's a good mixture of AMD, NVidia, and Intel hardware, and these machines are regular used for benchmarking.

We chose quite early on to have a minimum frame rate of 30 FPS, and following the optimisation in the final early access update and the subsequent final release, benchmarking was showing that we were typically exceeding this and by a pretty decent margin. Unfortunately this didn't match up with your expectations, and you let us know! It was always our intention to optimise further as part of our continuing development of the game, but the release feedback made it our number one focus.

Whilst we were busy on our release branch for launch, Rosario was busy modernising our renderer. It was too risky to include for our initial release but it gave us a nice head start on our optimisation work, and also had the benefit of improved lighting. From there it was just a case of doing the standard optimisation things, essentially various combinations of doing less and/or doing it more efficiently. Profiling, looking for bottlenecks, and recoding the hell out of them, aiming not just to improve minimum frame rate but to also increase the consistency of that frame rate.

In professional development with the need to include time for planning, review, testing, etc., three months quickly becomes two. Patrick spent most of this optimising the damage code, then tuning the new damage code and fixing the resultant bugs. This work combined with caps on expensive features like pinball mode lead to greatly minimised performance lows.

A lot of the other work involved knocking off half a millisecond here and there. Things like the UI only processing stuff that has actually changed, improving unexpectedly expensive string building, reducing data transferred between C++ and LUA, rationalising some vehicle construction, among an awful lot of other things.

This got us to a point where the game was running significantly faster, and a little smoother than our full launch, but lead to an issue on one specific min spec machine. Frame rate had improved significantly on this machine, but had become horrendously, unplayably inconsistent. Unfortunately our user friendly performance graphs were hiding this behaviour, and it wasn't until we saw it with our own eyes that we started to understand more of the issues that had previously been reported. So with just 2 weeks to our intended launch date for the update we had a massive new optimisation issue to get to the bottom of. I can't express enough how 'cool' it is to have the machine with the problem actually sat in front of you. Trying to debug/diagnose issues remotely through conversation and limited logs is often frustrating and frequently futile (for all parties). We still try, but I really wish every time someone reported a problem, we could just pop around their house with a laptop and sort it out. Although I suspect the ones that would be happy with me doing that, are the ones my mum warned me about.

Anyway, hardware with a problem, on my desk, go fix! Turns out there were two issues. The first issue that was identified was that we'd exceeded the VRAM on a 1GB card. Not typically a massive issue, drivers tend to do a decent job of swapping stuff out to main RAM for you, and as long as you're not too much over budget you should just get the occasional micro-stutter. Still, with limited main RAM as well as VRAM, it seemed like an ideal candidate for optimisation. We were able to come up with a mostly win-win for this, reducing our vertex size saved a significant amount of VRAM and gave a minor performance improvement.

The second issue wasn't quite so straightforward to track. Profiling was just showing the machine as doing a lot of work, but mostly because running a game is quite lot of work. After some investigation, the issue disappeared completely with virtual textures disabled. It turned out the VT system was trying to load/decompress too many images simultaneously, and whilst each task was relatively self-contained and quick to process, the sum total of them put too much of a drain on the system as a whole. Especially on lower end systems. A drop to the number of concurrent tasks gave us a functional VT system, removed the horrendous stutter, and had an equally significant impact on frame rate consistency and performance across the full spectrum of PCs. There was a slight increase in texture pop-in, but the benefit vastly outweighed this.

This left us with 3 days to test and get it live on Steam. Fortunately, for the most part, it seems to have worked out well. And we all hope that the next update (currently in its final QA/test phase) will be received with greater enthusiasm still.

Discuss this in the forums

Comments (11)

Mr.Smiley's picture
Submitted by Mr.Smiley on Mon, 26/10/2015 - 18:53

Stainless Why are the cars so shiny? :)

Edited by: Mr.Smiley on Mon, 26/10/2015 - 18:53
MrDeviance's picture
Submitted by MrDeviance on Wed, 28/10/2015 - 04:29

Because the look better. They reflect more now.

elsenator's picture
Submitted by elsenator on Mon, 26/10/2015 - 19:04

I love reading this stuff! Thank you very much for the behind-the-scenes look at what goes into an update like that. Update 1 was a huge achievement when it comes to raw performance, no doubt about it. On my powehouse of a machine it went from "ok" to "super".

I am eagerly awaiting the new update. Looked really, REALLY good on the Twitch stream, even with the crappy video compression artifacts.

Epitaph's picture
Submitted by Epitaph on Mon, 26/10/2015 - 19:46

Great to meet Simon and some very insightful information about that part of development!

By the way, when i saw this

Quote:
Simon is pictured below, testing the motion blur code…

and then instantly saw Nobby i thought to myself "WTF!?!?! Has he cloned himself to fill dev positions?"
:D

Harmalarm's picture
Submitted by Harmalarm on Tue, 27/10/2015 - 08:57

Interesting read ! I'm familiar with last minute changes combined with tight deadlines, so I totally understand what you guys must be going through. Same for the shitstorm that can follow when releasing work because of unexpected errors on different machines... ugh. Anyway, the 2nd update seems like a big improvement from what I can see in the twitch feed. Keep up the good work!

matias-uy's picture
Submitted by matias-uy on Tue, 27/10/2015 - 19:14

Thanks for the excellent work but please add wheel support (mine is Logitech G27) for the next update!! Around 30% or more users wants to play this game using a racing wheel rather than a joystick or keyboard (there is a poll out there, I do not remember exactly the URL).

Freeride Forever's picture
Submitted by Freeride F... on Wed, 28/10/2015 - 01:35

Did anyone think it was easy to increase performance? I thought everyone just knew it was necessary, not necessarily 'easy'.

Can't test every system combo, but you can consult something like the Steam Hardware Survey to get an idear of what's out there. Duals & quads make up around 98% or something like that, at no more than 3.69 GHz. 1 GB video cards reign with 2 GB not far behind. That tells me enough about the common GPUs out there. 8 GB system RAM followed by 4 GB. Whatever the reasons may be, final EA build on a lean & clean Win 7, 3.9 GHz quad, 8 GB RAM, R9 290X 4 GB, 120 GB SSD system in my house never breached 30 FPS on all MINIMUM settings! The range was 20-30. Not just pitiful, but totally unacceptable & unplayable.

Thanks for the informative blog. Thanks even more for making the unplayable playable. Couldn't play it before, now I can't stop. I'm now confident in & stoked for more improvements & content in the future. My Steam review has been changed to reflect my experience with the results of your efforts. I hope others do the same.

Edited by: Freeride Forever on Wed, 28/10/2015 - 01:35
MrDeviance's picture
Submitted by MrDeviance on Wed, 28/10/2015 - 05:08

This blog post should have come much earlier to help save the ship from sinking.

Where Stainless fucked up ROYALLY is when they decided to release a broken, laggy, outdated looking game and call it the FINAL RELEASE...
That mistake earned Reincarnation very bad reviews and scores from both the press and the fans..
So now when almost everybody that is not from this forum, forgot about the game and dismissed it as ugly broken crap, Stainless comes and they fucking fix it and hope that somehow everybody that was disappointed with it will find out just because it's posted here, in one youtube video and in a twitch stream?
I am so sad to say it but the reviews will not get redone nor patched just because the game has changed a lot after launch, which is indeed a shame.

I do however have a crazy idea that might just help repair the damage that has been done with the Final Release's bad rep and reviews.
I believe that all of these updates that are coming in the near future need to be piled on top of the game and then, the game needs to be given a serious amount of new content and have it called a different name, in order to get rated again by everybody that shat on it.

That's right Stainless, you read that right! Carmageddon Reincarnation needs to be RENAMED after you release all of the performance, graphics, gameplay and content updates.
You should either call it, Carmageddon Reincarnation: Fixed Edition or something completely different like Carmageddon 2016 HD edition or something of the sorts.
If you change enough stuff in it, you might just get away with calling it a new game that can be given for free to those that bought Reicarnation and sold to those that haven't yet purchased it.
It's really the only way to get your fixed edition some publicity and have some of the displeased fans, come back and check it out again...
And if you still have any plans to release it on the ps4 and xbox one, make sure you definitely rename it over there at least.
Take the example from Team17 that released Worms 2: Armageddon for the consoles and one year later they released the same game again for PC and they decided to randomly call it Worms Reloaded for no apparent reason.
I believe that fixing Reincarnation in both graphics and peformance and adding some more content, is going to justify changing the game's name and have it re-released

Edited by: MrDeviance on Wed, 28/10/2015 - 05:08
matias-uy's picture
Submitted by matias-uy on Wed, 28/10/2015 - 18:14

MrDeviance I cannot agree more with your comment. Perhaps you might not sound nice, but everything you said is pure game marketing strategy.

Edited by: matias-uy on Wed, 28/10/2015 - 18:14
HYLK's picture
Submitted by HYLK on Sun, 10/01/2016 - 11:34

Well here i am once again. And can see that the game hasnt improved for the better. Either u go for what the 10 fans here want, or u go for the people u dont have. Seeing these small fixes dont make the game better, it only make the game run smoother.

Some easy tips would be that u need:
a better career mode, one that make sense and has a challenge and goal to it.

more cars, with different handling, mass and armor to it.

Realistic upgrade, more armor = slower acceleration and more defense, greater engine = more speed and less handling, more offense = greater weapons less armor and better handling.

Harder difficulty to gain upgrade tokens. (now i just crash up some time, and let a girl of age 10 run around collecting coins. And u would be surprised of how easy she finds it to be)

And don't bother to remake another carmageddon game, People will have this game in mind and not bother testing or even give something thats a shadow of this game a chance.

I know it will hurt some of the egoistic people here, but thats the truth. Whether u like it or not. To make a change, its time to please those u dont have instead of giving something to those u already control.

Edited by: HYLK on Sun, 10/01/2016 - 11:34