The popularity baseline was harder to beat than expected

Running three approaches against the same play-count data produced a result worth stating plainly: the naive baseline — recommend whatever is played most — came close enough to the tuned collaborative filtering models that the difference did not clearly exceed the variation between evaluation runs.

The finding

Both similarity-based models improved on the baseline, and the improvement was small. Tuning the hyperparameters moved the error more than the choice between user-user and item-item did, which is the opposite of how the two are usually discussed.

The predictions themselves show the same thing from another angle. On the user-item pair the notebook inspects, the actual play count was 2 and both tuned models predicted between 1.3 and 1.5 — close, but both biased low in the same direction, which is what a model trained on a long tail does.

Why that is the interesting result

A baseline this strong tells you something about the dataset rather than the algorithms. Play counts concentrate heavily: most titles are played a handful of times and a few account for most listening, as the distribution in the notebook shows. When most of the signal is concentration, recommending the concentration captures most of the available accuracy.

Which means the honest reading is not “collaborative filtering does not work”. It is that this evaluation cannot distinguish the approaches, because the metric rewards agreement with a distribution all three models can reproduce. Evaluation covers why that distinction matters more than the ranking it replaces.

What would settle it

A metric that penalizes recommending what someone would have found anyway — coverage of the tail, or improvement on users with short histories specifically. That is a different evaluation, not a different model, and it is the thing worth building next.