How we rank

Best-of lists disagree, boost recent releases, and are published faster than anyone can read them.

Ultimate Ranks reads them all, corrects the bias, and finds the consensus.

Drawing the line somewhere

We only include publications with editorial oversight: outlets like IGN, Polygon, and Edge for games; Pitchfork, NME, and Rolling Stone for music; Sight & Sound, Empire, and Cahiers du Cinéma for film.

We generally follow Wikipedia’s reliable-sources criteria — it’s the closest thing the internet has to an agreed standard for what counts as a credible source, and it’s maintained by thousands of volunteers with nothing to sell. Thanks, Wikipedia.

Correcting for the internet

Without correction, modern titles would dominate simply because more lists are published nowadays and they favor recent releases. This infuriates people who have played Super Mario Bros. 3.

To fix this, we weight articles according to the number of years they cover and the number of articles covering the same period. The more time an article’s list covers, the more it counts, so all-time lists count much more than end-of-the-year lists. The more articles cover the same period, the less each counts. This applies due weight according to how comprehensive a list is, and normalizes the increasing number of articles published over time.

Here’s what that looks like in our games data. Every list gets a bonus for the span of time it covers, and a deduction for how many other lists cover the same period:

Span	Years covered ÷ List count = Weight
2024	1 ÷ 54 = 0.02
1996	1 ÷ 2 = 0.50
2010s	10 ÷ 44 = 0.23
1990s	10 ÷ 9 = 1.11

Everything versus everything

Ok, now we’ve got a zillion reliable lists squished into fair time-cubes. To combine the rankings on each list, we treat the lists as ballots in a ranked-choice election. The math on the exact method we use is truly gnarly, but once the lists are weighted, the effect isn’t far off from just counting them up.

The intermediate explanation is something like:

1. Compare an entry to another entry on every list and count which is ranked higher more often
2. Repeat this head-to-head matchup for every single entry
3. Whichever wins the most overall is ranked higher

The actual method is difficult to describe in prose, but Wikipedia saves us again with this very simple table illustrating the process:

Diagrams by Markus Schulze, from Wikipedia’s Schulze method article, recolored and reused under CC BY-SA 3.0.

If you really want to know: we solve the all-pairs widest-path problem over a (max, min)-semiring using Floyd-Warshall relaxation.

$$P_{i,j}\leftarrow \max (P_{i,j},\min (P_{i,k},P_{k,j}))$$

Setting graph theory aside and pivoting to the equally accessible social choice theory: we chose this method because it can combine lists of any length or format to fairly express the original critics’ views.

A perfect mess

Unfortunately, the completely perfect lists from Ultimate Ranks are based on flawed lists created by flawed people whose lists are likely influenced by other flawed lists. Life is messy. Math can’t fix sociology.

We’ve applied as much honest data smooshing as we can to extract a signal from this noisy world, and we hope it’s useful, but really: Watch what you want. Read the book your landlady recommended. Play Final Fantasy 25 because you are who you are. Maybe just go outside for a walk.