The author mentions Sea of Thieves and Mindustry, but I think Factorio is worth a mention too (seemingly as usual when something game related comes up on HN). They do TDD and talk about their automatic tests in several blogs:
I still think the best combo is a mix of automated and QA. Software in general went through a large decline in quality after QA was laid off across the industry, and in my experience, automated testing just tends to find different kinds of issues. Also, playtesting is also not the same thing as QA, but I digress :)
What a lot of people don’t realize about QA is that it’s also a point of Empathy to your end users. Some might think it’s the PMs job to think as an end user but in my experience PMs are so bogged down with other sales adjacent duties that they rarely if ever can provide that pillar.
This is a core issue with for example Google products where MANUAL PROCESS BAD seems to be a religious bastion.
Recently Inherited a code base with almost 20000 tests. Many very detailed. Most of them were so mocked out that at end of day it was just ensuring the mock library was well tested.
> Most of them were so mocked out that at end of day it was just ensuring the mock library was well tested.
But what's the alternative here, when your code depends on other services and other logic that might return complex data structures, or need to do dozens of database calls?
Should you go for primarily integration tests, which might not only slow everything down, but also force you to think about the possibility of either needing to mirror your current environment per test run (e.g. launch a database, run all the migrations, generate needed test data, launch your app against the database instance, run tests against it, clean everything up), or being limited to a single test run at a time with no parallelism?
It depends on the codebase I think, some codebases are spaghetti mess that really benefit from being well integration tested. Other codebases are really well organized, atomic and have really clearly separated concerns. I think you can afford to run smaller less integrated unit tests. I don't think there's a right or wrong answer, just a balance of the two depending on your project.
Wonder what their coverage is in this - or if they even still use it. There are many occurrences of "how does this ship" bugs in League where damage numbers don't add up or status's have no actual effect, etc - sometimes hanging around for many patches. Not that I don't understand how hard it is to test the big ball of mud that (10+ years old!) game engines end up being.
I strongly agree with this. You need someone around that actually uses what you develop, be it dogfooding or structured QA.
I wonder if there is any books on game/software QA akin to Masters of Doom or Stay Awhile and Listen? Those pretty much skip over QA, it's usually just the devs playing their own game (which anecdotally is usually biased and of limited use).
The industry is changing towards more automated testing, I heard of a lot of AAA studios implementing/creating testing frameworks. It was very uncommon back in the days, they just threw a bunch of badly paid and overworked QAs at new builds. Games are very complex, covering a wide range of behaviours in environments with a lot of randomness is difficult. With automation, the predictable behaviours (for example, testing that climbing a ladder works) can be offloaded from QA and they can focus on finding more random issues with playthroughs. I guess there was also a push from engineers without a gaming background (and being more used to TDD) that have been recruited in the industry throughout the years.
https://factorio.com/blog/post/fff-366 https://factorio.com/blog/post/fff-288 https://factorio.com/blog/post/fff-186 https://factorio.com/blog/post/fff-62