Building a Player Ranking Model

Published on 07/08/2017 09:00 PDT by ROOT Gaming

This week TheRedViper on reddit and TL came forward trying to develop a system of ranking players with points based on tournaments won. I come from a sports statistics job and didn't, and still don't, like the idea as a complex and indepth way of ranking players since it lacks context of who competed in each tournament, the meta, level of competition and tournament format but I think it had merit as a nice foundational beginning for a Greatest of All Time conversation. Now again, this isn't meant to replace stuchiu's list, which I think is more complex, indepth and full of context. I know that list is two years old but I still think the top 3 remains unchanged with a few players having won more tournaments like herO, Dark, Polt and INnoVation who would move up but not like an entire rewriting.

Now let me explain my thinking here. TheRedViper suggested a point system where each tournament had a generic point value with tougher, perceived, tournaments getting more points than the ones that are perceived easier. I had a few issues with this:  

1. It put more weight on tournaments like GSL over an IEM despite for many years them having the same players for the majority of the tournament. There is no factual reason for this and this idea should be discarded. No system should be based on tournament name only.  

2. It doesn't account for era and rule changes. For example WCS events during most of HotS had a mix of foreigners and Koreans while the modern WCS has far fewer Koreans. Shouldn't those earlier WCS events carry more weight due to stiffer competition?  

3. Shouldn't the number of players in a tournament factor in? Beating your way through a 64 or 32 player tournament (Take a DreamHack or MLG for example) be viewed as a greater accomplishment than winning a four or eight player tournament (The Korean Cross Finals or WCA Global Finals)?

Methodology

With these issues with TheRedViper's model in mind, I set out to create a model that solved these issues, at least in part, even only at a basic level. To keep it simple this is ONLY PREMIER TOURNAMENTS, if this is seen as a weakness I may add Major class as well but I feel this is a good limit. So I looked at each of the three issues I identified:  

1. I decided not to put value based on the name of the tournament. Instead I based a tournaments difficulty on two factors. The first is what percentage of the players were Koreans. This way an IEM that is 95% Korean players won't be viewed as less difficult than GSL purely because two or three foreigners made it to the round of 32.  

2. My solution also fixes problem #2 because WCS tournaments are given points based on era or their title, instead if a WCS took place during 2013 and it was 50% Korean it gets more weight than a WCS this year when there is only one Korean. It removes the bias factor of the name WCS or the era in which the tournament took place.  

3. I mentioned in my solution to problem #1 I had two solutions for adjusting the difficulty of a tournament and this is the second solution, adjust for the number of players. The more players generally means it is a harder tournament since to win you have more rounds to fight through.

With these three POTENTIAL solutions I set out to make a basic formula. I break tournaments down in three way:  

- List how many players competed in that tournament. This is only the amount of players ALL potential winners had to go through. So a DreamHack win is only viewed as a best of 32 since not all players have to fight their way through the other group stages. I'm fine with changing this if I need to. As with all factors in this, I'm creating a model and I expect it to be tweaked.  

- Next I looked at each tournaments population and broke them into three categories: Foreigner (less than 25% of players were Korean), Mixed (between 25% and 74% of the players were Koreans) and Korean (75% or more players in a tournament were Koreans). By doing this we get tiers to a tournaments difficulty. Once I made those tiers I gave them a point value: Foreigner (2 value or difficulty multiplier), Mixed (3 value), Korean (4 value). What this means is a Korean tournament is twice as valuable as a foreigner tournament. Again willing to change, just baseline to start discussion.

- Finally we combine them. We take the player count and multiply it by the difficulty multiplier.

Let's look at three examples:  

- IEM Season XI - World Championship: 24 players, Mixed category (3 difficulty multiplier) = 72 points
- 2017 WCS Austin: 32 players, Foreigner category (2 difficulty multiplier) = 64 points
- 2017 GSL Super Tournament #1: 16 players, Korean category (4 difficulty multiplier) = 64 points

So here we have examples of each type of tournament producing similar results through different player counts and difficulty. Overall I am fairly happy with this since it is very easy to understand (player count * difficulty multiplier = points) while also making basic sense. 

The final issue I needed to solve was how to reward the winner, runner-up and other competitors who finished well. I decided to make the cut off at round of 8 finishers and above. While consistently finishing top 16 is a nice way to make some money and all it's not where greatness comes from. Obviously the winner should get more points than the runner-up who would in turn deserve more points than a 3rd-4th place finisher or a 5th-8th finisher. With that in mind I did a quick test to see what percentages created the most balanced results. After Excel spit it out I just rounded things off (we don't need 23.73%, just rounded up to 25%) and here were the results:

- 1st: 100% of the tournament's point value
- 2nd: 75% of the point value
- 3rd-4th: 50% of the point value
- 5th-8th: 25% of the point value

An example of a 150 point tournament would look something like this:

- 1st: 150 points awards to the winner
- 2nd: 113 points goes to the runner up
- 3rd-4th: 75 points awards to these finishers
- 5th-8th: 38 points go to these finishers

As with all variables, I don't mind changing these if presented with a good reason to do so. The formula is what matters to me not the specific input variables.

Here is a Keep It Simple Stupid version of what we just talked

- A tournament's difficulty is determined in two ways: player count and percentage of Koreans
- This creates a tournament difficulty system that is swayed by the difficulty of winning the tournament rather than basing a tournaments value purely on its name.
- This still ignores meta, what level of Koreans are competing in each event, and unique formats such as round robin, multiple group stages, etc.
- This isn't a measure of skill, a player could be unstoppable for two years straight and never lose in a tournament then stop competing and wouldn't rank #1 because this is a count statistic, it rewards players who win consistently but isn't a measure  of win percentage.

The Results

Having covered how the model works, let us take a look at the results. First we'll look at the top 10 players. To read the table it goes like this, MC has 2150 points, 6 first place finishes, 10 second place finishes, 10 third and fourth place finishes, etc:

Player
Points
1st
2nd
3rd-4th
5th-8th
MC
2150
6
10
10
18
Polt
1972
8
5
8
11
TaeJa
1916
11
2
11
9
MMA
1688
9
4
7
8
Life
1512
10
4
6
4
Mvp
1420
9
2
6
1
PartinG
1356
5
7
6
15
HerO
1289
5
2
5
17
Rain
1250
5
3
5
12
INnoVation
1226
6
2
7
11

Right away I'm sure there will be contention that a player like Life, Mvp or another of the dominant players isn't in the top 3 but remember the strength of this is measuring success rather than dominance. MC, Polt, TaeJa and MMA all may have lacked the peak of the regularly considered "best players" but they all had more sustained success. Polt and MMA are the only two players to win a Premier tournament in all three expansions and Polt is the only player to win a Premier tournament in six separate years.

Another factor that may need to be looked at is volume of tournaments a player participated in. MC won less tournaments than anyone in the top six but he had the most top 8 finishes, as his 18 5th-8th results show. While those may not be worth a lot of points they do add up when he earns 50 points here and there. The same can be said of players like HerO and PartinG who never had the championships other top players did but they did have 15 or more 5th through 8th finishes.

Life and Mvp do stand out with their 10 and 9 championships respectively with far, far fewer low ranking finishes. Mvp only brings in one 5th through 8th place finish. He doesn't have the longevity of someone like Polt but his win rate was amazing and that should be noted.

Next we'll examine the top 10 foreign players for a bit of fun:

Player
Points
1st
2nd
3rd-4th
5th-8th
NaNiwa
924
2
4
2
13
Stephano
843
4
2
6
8
HuK
758
3
0
5
10
Nerchio
744
3
4
3
13
Snute
686
2
4
5
13
MaNa
529
2
4
3
5
Ret
454
1
0
4
6
IdrA
440
2
1
3
7
Scarlett
416
0
2
4
8
Neeb
374
3
2
3
3

This one is a bit closer to the consensus surprisingly. NaNiwa and Stephano top almost every person's top 2 foreigners of all time list, though maybe in different orders. After that HuK makes sense, he ranked top 5 in stuchiu's ranking. Nerchio ranked a bit lower in his list but that was two years ago and since then he's had an amazing 2016 that would make sense to boost him up. The same can be said of Snute who hasn't had the peak of NaNiwa and Stephano but has had probably the most consistent career of any foreigner with success sustained for multiple expansion. MaNa and Ret seem like odd results for the middle of the pack but both had success early in Starcraft 2's lifespan and sustained that while MaNa even had success in 2015 with his 2nd place finish to Lilbow. IdrA and Scarlett seem a bit low but since this doesn't note that much of their success came during some of the most difficult eras for foreigners I see that as a failing of the model. Neeb will probably be seen as the biggest insult but remember that prior to 2016 he had almost zero success and a year and a half of being a top 3 foreigner can only lift him so high. If he can sustain his 2016 level of play through the rest of 2017 and into 2018 he'll move up the rankings quickly. But again he had virtually no top 8 success for five years, it makes sense he is this low.

Help Me Improve This

I don't claim this is perfect or even finished, I just want to present it to you so you can help me adjust it to make it better. My goal here is make the most balanced and accurate model, if you have ways to help me improve then please share. I'd rather the model improve and me be wrong than never change and be stubborn and inaccurate. With that in mind here are the areas I think we should focus our discussion to improve this but if you see issues elsewhere, please let me know.

First off let me note the biggest issue I want to address in future model adjustments:

Problem: That this does not really display dominance and rather rewarding players for longevity. Now that may not be an issue, we may feel it is better and easier to create a ranking of Most Successful Players rather than Best Player since success is easier to measure than best-ness. But if I do decide to shift the goal from success to dominance my best solution would be two fold.

Solutions: First I'd weaken point total for 5th through 8th place finish to not reward players who rarely win but consistently finish in the top 8. This would be a pretty easy fix. The second fix would be much harder to do but would be more accurate when looking for the Best Player. I would look at average results per tournament. Player A competes in 20 tournaments but averages only top 8 finish would average lower points than a player who competes in only 5 tournaments but averages a top 2 finish. This would reward players with shorter careers but who had dominant results during that short career such as Mvp, Life, or someone like Stats, Dark, Neeb or TY who may still be at the beginning of their dominance.

- Adjusting the tiers by adding more tiers by breaking down the percentage of Koreans or keeping three tiers and adjusting the percentages for the existing tiers. Think there should be five tiers instead of three, give me a good reason and we'll adjust.
- Adjusting the difficulty variable for each tier. If you think a Korean tier tournament is actually three times as valuable as a Foreign tournament, explain why and we'll adjust.
- Adjusting the point payout based on finish. Think the 5th through 8th finishers only deserve 10% of the points rather than 25%, let's talk about it.
- Adding in Major class tournaments. I think this has pros (more results) and cons (many are online and subject to more upsets and less consistent results). Let me know your thoughts.

Again, I never claim this is perfect, accurate or anything of the like. I realize this is going to be fairly shallow but sometimes that is all that is necessary. I'm also not claiming this is finished, I want to improve this though I doubt it will ever be perfect. 

I will likely do a few followup articles by breaking down this data in different ways to find various things out such as which player was most successful in each expansion, which player had the most successful year ever, etc. Let me know if you have any requests and I'll respond.

About the Author

Topher is an American football and eSports writer with a focus on statistical metadata research. You can follow Topher on Twitter

RotterdaM 414
ViBE 24