How we rank

Best-of lists disagree, skew toward whatever came out recently, and multiply faster than anyone can read them.

Ultimate Ranks reads all of them, corrects the bias, and finds the agreement.

Well, not all of them.

We only include publications with editorial oversight: outlets like IGN, Polygon, and Edge for games; Pitchfork, NME, and Rolling Stone for music; Sight & Sound, Empire, and Cahiers du Cinéma for film.

We generally follow Wikipedia’s reliable-sources criteria — it’s the closest thing the internet has to a consensus on what counts as a credible publication, and it’s maintained by thousands of editors with nothing to sell. Thanks, Wikipedia.

Correcting for the internet

The 2020s aren’t over yet, but they have already produced over 20 times more game ranking lists than the entire 1990s. Without correction, modern entries would dominate simply because more lists exist for recent years. This infuriates people who have played Super Mario Bros. 3.

The 2020s don’t have more lists because they have more great games or better albums; they have more lists because there are more publications writing them.

To fix this, we group articles by their scope: single years, decades, or all-time. Each category gets a fixed budget of influence tied to how much history it evaluates: one unit per year of scope. That’s a budget of 1 for a single year, 10 for a decade, and one unit per year of the medium’s lifespan for all-time lists.

Each period’s budget is split equally among the articles covering it. If 56 articles cover “Best of 2023,” they each get a tiny slice of that year’s budget; if only one article covers 1994, it gets the full amount. These weights scale each article’s contribution to the pairwise tally in Everything vs. everything.

The alternative (weighting every list equally regardless of scope) is simpler, but it lets publication volume dictate what’s considered the best instead of, you know, how things were actually ranked.

We correct for the underlying cause of recency bias: the growth in publication volume over time. The goal is to give each calendar year comparable total representation without forcing it.

Not even worth mentioning

What to do when an article could have ranked something and didn’t? On a “Best Games of 2015” list, leaving off Bloodborne is a deliberate omission; on a “Top 100 of All Time,” leaving off Bloodborne might just mean the writer ran out of slots (still: no). We handle these a little differently depending on how crowded the field is for the medium.

If a medium has a relatively concentrated critical canon (like major video game releases), we can treat an entry that’s left off a list as a last-place vote and use that assumption to sharpen the rankings. For mediums with vast catalogs like music, an omission is just an omission.

Everything vs. everything

With that in place, we run a Condorcet election using the Schulze method. This method:

• Handles messy data – Unranked entries can be treated as tied; partial lists keep whatever order they have.
• Resists vote splitting – Revolver, Abbey Road, and Sgt. Pepper’s don’t dilute each other the way they would under a points system.
• Respects majority preferences – If most critics rank Pet Sounds above Thriller, Pet Sounds finishes higher (this seems obvious, but other methods can screw this up).

The algorithm has two stages:

1. Pairwise tally – Compare entries directly by counting how many times A beats B.
2. Strongest paths – Compare entries indirectly using dark math. Basically: if A>B and B>C, then A>C. We solve the all-pairs widest-path problem over a (max, min)-semiring using Floyd-Warshall relaxation. Too easy!

Once that’s done, you get a respectable list and a lot of lingering questions about how you’re spending your free time.

j/k lololol

Unfortunately, the completely perfect lists from Ultimate Ranks are based on flawed lists created by flawed people whose lists are likely influenced by other flawed lists. Life is messy. Math can’t fix sociology.

We’ve applied as much honest data smooshing as we can to extract a signal from this noisy world, and we hope it’s useful, but really: Watch what you want. Read the book your landlady recommended. Play Final Fantasy 25 because you are who you are. Maybe just go outside for a walk.