- Reading behaviour carries a signal for newsletters: every model beat a random ranking at the top 5, and the collaborative model got 2.84 of 5 right.
- The biggest gap was the newsletter data. Link clicks in the sent emails were the only signal, of very low quality, which more or less made unbiased recommendations from interaction data unfeasible.
- Next time: dense embeddings of users and newsletters or another kind of model, or first a better way to track how newsletters are read.
Newsletter recommendations
Bachelor's thesis at KTH with Osvald Svenaeus, done with a Swedish media company. Its articles were already personalised; its newsletters were the same for everyone. We tested whether what people read on the site can predict which newsletters they engage with, using three kinds of recommender systems compared offline on 28 days of user interactions and metadata.
thesisDiVA ↗
levelBSc · KTH 2025
withOsvald Svenaeus
toolsSurprise · scikit-learn · PyTorch
text modelKB BERT (Swedish)
~/conclusion
~/results
Bold: best on that metric. Random ranking: what picking newsletters at random would score, 5 of 17–20 on P@5 (computed here, not in the thesis); it has no defined value for the other metrics. Table I in the thesis.
| System | MRR | P@5 | RMSE | MAE | Built by |
|---|---|---|---|---|---|
| Neural networkbehaviour, demographics, article sections | 0.666 | 0.362 | 0.158 | 0.071 | Osvald |
| Neural network + BERTplus embeddings of read articles | 0.636 | 0.352 | 0.161 | 0.075 | Osvald |
| Collaborative (Funk SVD)users × newsletters and articles | 0.564 | 0.568 | 0.200 | 0.104 | Olle |
| HybridSVD prediction as an extra network input | 0.647 | 0.366 | 0.160 | 0.072 | both |
| Random ranking | – | 0.25–0.29 | – | – | – |
- The collaborative model was best at the top 5: 2.84 of 5 recommended newsletters were relevant, against about 1.8 for the others and 1.3–1.5 for a random ranking.
- The neural network had the lowest error and the best MRR: the first relevant newsletter came at position 1.5 on average, out of about 20.
- Adding BERT embeddings of the articles a user read added noise and made the network slightly worse; the hybrid scored like the plain network.
~/my part: the collaborative model
The research angle: can interaction data across item types feed one traditional collaborative model, where users and items are vectors of latent features? To test it I built one users × items matrix in which the items mix two content types, newsletters and articles, to see if the model could learn latent features that span both.
Rows are users, columns are newsletters and the articles read in the 28 days. Article cells hold read time, capped at 9 minutes. Newsletter cells hold a rating built from the emails: 1 point for subscribing plus 10 per link click, normalised so each user's ratings sum to 1, then scaled to the same 1–9 range so the two kinds of item sit side by side. Most cells are empty. Each user and each item gets a short vector of latent features (the small squares); a hidden newsletter rating (?) is predicted from the user's vector and the newsletter's vector. 20% of user–newsletter ratings were hidden and predicted this way. Illustration, not real data.
how the vectors are learned
Every user and item starts as a short random vector. Training uses stochastic gradient descent: for one known cell at a time, the model predicts the value from the two vectors, measures the error, and nudges both vectors to shrink it. Only the known cells are used, which is what lets Funk SVD work on a mostly empty matrix where classic SVD cannot. Repeated over all known cells, the vectors settle into latent features: directions nobody labelled, such as a taste for sport or for long reads. It is the same idea as word embeddings such as word2vec, where words start as random vectors and gain meaning from the sentences they appear in; here users and items gain meaning from the cells they share. Because articles and newsletters share the user vectors, a user's reading shapes the vector that predicts their newsletters.
newsletter ratingrui = (10cui + sui + 0.01) / Σj (10cuj + suj + 0.01)each newsletter's share of a user's points: 10 per link click (c), 1 for subscribing (s), plus 0.01 so no rating is zero; then scaled to 1–9 to sit next to read time
predictionr̂ui = μ + bu + bi + qi⊤puaverage rating + how high this user rates + how high this item is rated + how well the user's vector matches the item's
erroreui = rui − r̂uithe real value minus the prediction, for one known cell
updatepu ← pu + γ(euiqi − λpu)move the user's vector a small step (γ) towards the item if the prediction was too low, away if too high; the same for the item's vector qi
penaltyλ(‖pu‖² + ‖qi‖² + bu² + bi²)keeps the vectors small, so the few known cells are not memorised
Equation 9 (rating) and equations 2–4 (model) in the thesis; the model as implemented in the Surprise library.
~/limits
- Limited newsletter data: no cookie tracking in the emails, only clicks on the links in each newsletter sent out.
- That favours newsletters with many links, and many users never click at all.
- Read time favours long articles.
- Being subscribed says little: many users are subscribed to most newsletters.