Starcraft Statistics: An Introduction into Terminology and Methodology
Published on 01/05/2017 11:59 PST by ROOT Gaming
When I first started this series I wanted to replicate the meta data that I assume Blizzard has internally. So often within the depths of reddit or Team Liquid we see balance and design discussions devolve into comments and posts that suggest "this unit is the never used" or "No one researches this upgrade" and "if this race goes this tech route they always win" without much evidence to back those ideas up.
Now I am assuming Blizzard has internal data on all this data but since that is private, like most companies do, we are left to our own devices. But I hope we can rise above these debates and try and put some numbers behind these ideas. Today I want to lay out the foundation for my research.
This is largely a project of database building. What I mean by this is my goal is to catalog as many games as possible to build a large sample of games that we can draw on, update and rely on to look at when these questions arise. I obviously won't be able to gather the entire population's data but by collecting a sizable amount I should be able to replicate the population.
I should note that while my main focus will be on players of Masters and above, I will be collecting data from players of all leagues in case I ever wish, or am asked, to look at variance in unit usage or some such thing between leagues.
In traditional sports this type of data is common place. There are hugely popular sites that have powerful tools that let you search through historical data to answer almost any question you may have. In video games this is harder, the vast majority of games played are not public and many more games are played. In American football for example there are only 523 games played a year, compared to video games where thousands are played every hour. While I may not be able to perfectly replicate this style of database, I hope that my sample is large enough to overcome sampling error.
I already possessed a large replay cache of thousands of replays for a previous job but they were primarily from Heart of the Swarm so I began this project anew with the Legacy of the Void beta. My methods for gathering replays is Four-fold:
1. Replay packs from streamers. These I gather from either being a subscriber, asking the streamer, being a moderator or friend of that streamer.
2. Community outreach to clans, teams, and groups for replays.
3. Tournament replay packs.
4. Replay upload sites such as GGTracker, Spawning Tool and SC2 Replay Stats.
Through these methods I have over ten thousand replays across all leagues from the first iteration of the Legacy of the Void beta up until this most recent patch. The nice thing is that this is a growing database that gets added to monthly so we can see how things change with patches, meta changes and other factors.
Before we go a bit deeper let me point out the biggest weakness I have found with my methodology so far. It is that all four of my sources for replays tend to be of the higher leagues so while I do have replays from Bronze players my goal of achieving the same league percentages that Blizzard has isn't going to be feasible because players from lower leagues just seem less likely to upload their replays.
Linguistics and Terminology:
It's important when we begin this project to make clear what words mean so I felt it was important to include this section. Let me pose three hypothetical comments to you to think about:
- "Reapers are built in 93% of Terran versus Zerg games."
- "An average of 2.4 Reapers are built in Terran versus Zerg games."
- "In Terran versus Zerg games where Reapers are built the Terran builds an average of 3.7."
All three comments are talking about Reapers in TvZ but how they are phrased is key. Let's look at these different ways of discussing unit usage.
- This first comment look at what percentage of games Reapers are built in, it says nothing about the volume of that unit's production, rather it focuses on the proliferation of the unit. This is a good metric for seeing how often you can expect to face a unit but does little to show you that unit's role or usage throughout the game. In this example we learn that Reapers are a very common unit built in almost every TvZ game but we don't know if it's one Reaper or 20.
- For this comment we get a bit different information, we have no idea what percentage of games Reapers appear in but we do know on average how many Reapers are built in TvZ games. We have no idea if one game had 5 Reapers and another game had zero Reapers, it doesn't let us know the reality surrounding the number. Compared to the last it just provides less insight.
- The third comment is interesting because unlike the first one we don't know how often Reapers are used but we do learn the usage rate of Reapers when the Terran does build Reapers, something more useful than the second comment. This is more valuable information than the previous comment because we don't care about the average of Reapers in games where Reapers were not potentially built but if we see our opponent going Reapers it's helpful to know what they usually go to.
When we have these discussions we should combine usage rates (Unit ____ is built in __% of games) and unit volume metrics (Unit ___ is built ___ times per game on average in games they are built). By utilizing these two metrics we can get a clearer picture of a unit's or upgrade's role. Another thing I should point out is the more specific a query is, the better. Knowing how many Carriers are built on average isn't useful information but knowing how many Carriers are built in Protoss versus Zerg in games going longer than 15 minutes by Platinum players is a very useful piece of information.
Going forward my posts in this series will still be a combination of metadata research which I have introduced here as well are larger projects like you've seen me do in past weeks. For my research posts on metadata they won't be huge spread sheets and total openness of my database since that is neither interesting nor a good way for me to make this worth my time. Instead I'll be looking at specific ideas. Here are two examples I am currently working on:
- Comparing the roles of the Zealot and the Adept and when each unit is used and in what situations.
- In Terran versus Zerg what composition has the highest win rate once Ultralisks enter the game with Chitinous Plating?
About the author:
Topher is an American football and eSports writer with a focus on statistical metadata research. You can follow Topher on Twitter