In previous articles we’ve looked at which of the platform holders is the most consistent, the rip-roaring momentum of Switch, which games are the most controversial, and which have produced the largest gulf between critics and players. There was one more question burning in our minds: which of the big third-party publishers is the best?
Obviously this needed some judgement calls, such as what counts as a ‘big’ publisher? We decided that first of all meant global publishers with their own development studios, who support multiple hardware platforms simultaneously publishing games globally.
For this article we’re also ignoring Sony, Microsoft and Nintendo as publishers (having covered each in detail elsewhere), and looking at which of the remaining multiplatform publishers produces the most consistently great games. In the end we came up with a list of ten to compare: Koei Tecmo, Bethesda, Capcom, Konami, EA, Bandai Namco, Sega, Ubisoft, 2K and Activision.
The results? Well, we never would have guessed, but the real dark horse was Konami – which did incredibly well on both metrics. That may sound odd, but the maths don’t lie.
How We Did It
As with our previous pieces on the best first party games and the Switch vs WiiU comparison, we took the data in this piece from Metacritic: We downloaded data on over 18,000 games and over 200,000 games reviews (across all systems) to draw these conclusions. Obviously this means there’s a number of important things to bear in mind - and so with which to calibrate the number of grains of salt you take this analysis with.
For example, our data is only as complete as Metacritic’s was when we took our snapshot of their data the end of last year. This means that if a game isn’t correctly listed as not being multiplatform, it would impact the data here. With this said, we’re fairly confident that Metacritic’s data is robust – especially for Xbox One, PS4 and Switch, given they are the newest and most closely watched consoles.
When it comes to scoring and working out averages, Metacritic normalises scores from different publications on to a scale of 1-100 (so that 5/10 becomes 50, and 3/5 becomes 60, and so on). We took all of the scores and worked out a mean score for each game (which is slightly different to Metacritic’s weighted Metascore), which is what we used in our comparison below.
On top of this, for this particular investigation we limited our dataset to each publisher’s PS4, Xbox One and Switch releases in order to keep the data timely and accurate, but did not place a limit on the number of critic reviews needed to make the dataset. Additionally, any multiplatform game was only calculated in the dataset once, with its highest score counted. If a game received a 96 on PS4 and a 95 on Xbox One, the 96 would be used as the game’s score when calculating publisher averages.
What The Numbers Say
First up, here’s the raw data. For each publisher the first number is how many games total it released, the second number is average score out of 100, and the third number is the standard deviation, a measure of how much their scores tended to fluctuate from that average number. A low standard deviation means that a publisher’s scores were stable and tended to be closer to the average score, whereas a publisher with a higher standard deviation has games that fluctuate in quality more.
When viewed in a random order it can be a little tough to see exactly what’s going on, so first up we reorganised the data so that the entries ran from highest to lowest average (mean) review score across their release libraries.
Sorted by average score, highest to lowest:
Konami – 80.36
2K – 78.79
Sega – 77.85
Bethesda – 77.83
EA – 76.27
Capcom – 74.96
Ubi – 72.35
Koei – 71.49
Namco – 71.36
Activision – 69.03
Among our chosen publishers, Konami has released the fewest games so far this generation with 12 releases – but it also has the highest average score with an incredibly impressive 80.36 out of 100. Activision released far more titles, 59 games total, and had the lowest average score for their games at just 69.03 out of 100.
If you throw these datapoints into a graph, here’s what you’ll see (number of releases on the horizontal axis, mean on the vertical axis).
This does suggest a very broad inverse correlation between number of games released and the average quality of those games, though there are notable exceptions. An example like Ubisoft, however, released the most games at 74 games, and is in the bottom 40% when sorted by average review score. The numbers suggest that the more games you release, the higher the chance of their average quality dipping, and the fewer games released the more likely they are to trend higher in quality. The correlation is by no means a slam-dunk, but it does seem like an observable trend that might reward more detailed analysis.
With that out of the way, we decided to reorganise the data so that we started with the most reliable publisher (lowest standard deviation) to least reliable (highest standard deviation).
Most to least consistent (Lowest SD to highest SD):
Koei - 7.11
Bethesda - 7.54
Capcom - 7.58
Konami - 8.75
EA - 9.12
Namco - 9.67
Sega - 9.81
Ubi - 10.22
2K - 10.28
Activision - 14.51
The fact that Koei Tecmo appears at the top of the list for high consistency (low standard deviation) isn’t necessarily surprising. While it releases a high number of games, those games all tend to follow a very solid and consistent formula. Most are hack and slash titles made using the Dynasty Warriors engine, where players battle an army of small weak foes to feel like they’re all-powerful, then face off against a single strong boss enemy, controlling territory along the way. While critics may penalise Koei for not changing much from game to game, the quality of those titles stays consistent due to that same lack of changes, resulting in a reliable average score. Koei Tecmo games score reliably, in other words, because they tend to make very similar games over and over – an observation, not a criticism.
It’s again not a particularly strong correlation but, when you compare number of games released and standard deviation, you’ll find that a rise in number of games tends to go alongside a raise in standard deviation. What this suggests is that, if you release more games, their quality will be less consistent, which seems logical.
A Nerdy Diversion
So overall it’s pretty clear that Koei Tecmo are the most reliable publisher, which makes sense for all the reasons above. Konami was a surprise to see as having the highest average score, but when you consider those titles are the likes of Metal Gear Solid V: The Phantom Pain, Metal Gear Rising: Revengeance and Pro Evolution Soccer, maybe it shouldn’t have been a surprise. But there’s a big caveat here, which is that this is very much global Konami – the publisher also releases many games in its home territory of Japan which never find their way overseas, and thus are not included in this analysis.
This flags up one of the real things to bear in mind with this data. Most publishers are very different from one another, and each has grown to its current size in a different way. Some are still run by the founders – others are controlled by shareholders – others have some combination of both. It’s not that we’re comparing apples to oranges (they’re all in the same business, at the end of the day) but that each publisher has its own priorities and production processes which, of course, are not a part of this analysis.
The general trends between number of games published, average quality and consistency of quality seem to make a lot of sense. Koei came out on top for consistency, and there seems to be a good explanation for that, while Konami’s Schwarzenegger-like average score shows a publisher that focuses its global releases on what it knows to be extremely high-quality titles.
Just as interesting are the major publishers floating somewhere around the middle. Ubisoft is impressive in this respect, having scaled enormously over the last decade-and-a-half and now producing many dozens of titles across all major platform holders – with more consistency than a competitor like Activision.
Speaking of Activision, it may be tempting to jump on these figures and say something nasty about Bobby Kotick, but we’d be wise to think about the story the data tells. Here? It could be suggested that Activision’s large portfolio of licensed and children’s titles is counting against them – these games tend to review badly, but sell very well.
Major props are due to 2K, Bethesda and SEGA. In particular, Bethesda’s first years as a major publisher were something of a disaster – low quality titles like Rogue Warrior sat alongside creative misfires like Wet. But in the last decade it has put together a truly impressive slate of games, and become one of the industry’s major players.
And SEGA? The grand old warhorse, six decades in and counting, still treading a measure with the gods? Come on, you know the drill –
To be this good takes ages.
If you enjoyed this article, check out our comparison of the big three platform holders' software production, and our breakdown of the Switch's library compared to the Wii U's.