A recommendation row can introduce a customer to something new while sending thousands of other customers toward the same popular products. The customer’s experience and the store’s sales mix can move in different directions.
The management decision is what you want recommendations to improve, and how you will know. Product variety on screen, variety in one basket and variety across the business are separate outcomes. Each needs its own evidence.
🔎 What research says
Evaluate recommendation designs at the level of the list, the customer and the whole catalogue. Follow changes in viewing through to purchasing, and inspect mobile and desktop results without confusing device choice with a randomized treatment. Research supports measuring these distinctions; it does not support a universal promise that recommendations make sales more diverse.
🗺️ In this guide
1. Separate personal exploration from store-wide variety
📈 Recommendation
When comparing recommendation algorithms, evaluate the range each customer buys and the distribution of purchases across your whole catalogue. One customer discovering a new category does not establish that the store is selling a broader mix.
For a homeware seller, compare the existing algorithm with a candidate while keeping placement, prices and eligible products consistent. Track categories bought per customer and each category’s share of purchased units. This is our proposed test, not the study’s exact homeware intervention.
For an executive, ask which outcome matters: suitable purchases, profitable sales, broader discovery or less dependence on a few categories. Set that objective before selecting the algorithm. A diverse-looking row is not an agreed business result.
🎓 Findings
Lee and Hosanagar (2019) ran a two-week randomized experiment at a North American retailer with 1,138,238 users and 82,290 products. The experiment compared no recommendation panel with view-based and purchase-based collaborative filtering: suggestions derived from other customers’ views or purchases.
In the accessible author manuscript’s Table 5, the five leading subcategories accounted for 10.0% of purchases in the control group and 14.5% with purchase-based recommendations. That is 4.5 percentage points more concentration, with a permutation-test p-value of .016. Expressed on the same denominator, those shares are 100 versus 145 of every 1,000 purchases. The extra 45 is arithmetic illustrating the reported shares, not a forecast for your store.
Aggregate purchases became more concentrated. Individual diversity did not move consistently across categories; the paper does not establish that every shopper’s basket became narrower.
🧠 Why it works
People can each branch out while branching toward the same popular destination. As a hypothetical example, customers who usually buy different kitchen products may all add the same popular mug. Each basket broadens, while store sales become more dependent on that mug. The study’s cross-purchase analysis supports this distinction; the mug example is ours.
✋ Limitations
The numeric table comes from a June 2017 author manuscript associated with the 2019 publication; final publisher-table equivalence still needs checking. The five leading subcategories can differ between groups. This is subcategory purchase share, not the share of five individual products, revenue or profit. Two conventional algorithms at one retailer do not establish the effect of every modern recommender.
2. Follow variety from product views through to purchases
📈 Recommendation
Keep discovery and purchased variety as separate outcomes. Compare them within mobile and desktop traffic, while preserving a randomized recommendation comparison in each group.
For a fashion seller, record which products are displayed, viewed and purchased. Calculate the distribution across the same eligible catalogue and time window for each stage. A product-view report and a sales report must identify their different denominators.
If discovery broadens but buying does not, inspect suitability, stock, delivery and product information before buying more discovery traffic. Test the suspected obstacle; this is a proposed diagnostic, not a proven remedy from the experiment.
🎓 Findings
Lee, Gopal and Park (2020) randomized personalized recommendations against an equal-sized best-seller panel at a South Korean fashion retailer. Over three days, product-view Gini was 0.498 with recommendations versus 0.674 in the control, with a statistically clear difference (p < .01).
Gini measures how unevenly activity is distributed across products: zero means an even distribution; values nearer one mean stronger concentration. The observed difference was 0.176 index points, calculated as 0.674 minus 0.498. It does not mean 17.6% more sales. The experiment found no statistically clear change in aggregate sales diversity (p = .279). That result leaves uncertainty; it does not prove the groups were identical.
🧠 Why it works
Visibility is one step in choosing. A shopper can inspect an unfamiliar item and still buy a familiar one. An operator therefore needs evidence at the purchase stage before claiming a more varied sales mix.
✋ Limitations
Device use was not randomized, so mobile-versus-desktop differences cannot establish that a phone caused different behavior. There was no clear recommendation-by-device interaction on sales diversity. The experiment was short and involved one fashion retailer; its null result is not proof that recommendations never affect sales diversity.
3. Check perceived variety against actual choices
📈 Recommendation
Ask whether recommendations feel meaningfully different, then compare that response with actual exploration and purchases. Keep visual variety, useful variety and sales variety distinct.
For a coffee seller, show the same eligible range through two truthful recommendation approaches. Ask whether shoppers can find a useful unfamiliar option; separately record products inspected and ordered. This is our proposed store test.
Keep a recommendation-level measure, a customer-level measure and a store-level sales measure. A colorful row, a shopper’s favorable answer and a broader sales mix answer different questions. Do not substitute one for another.
🎓 Findings
Matt, Hess and Weiß (2019) distinguish algorithmic recommendation diversity, perceived diversity and sales diversity in an online experiment using a realistic shop. The publisher’s abstract reports that perceived diversity did not always match actual diversity effects, and that outcomes depended on the recommendation algorithm and available preference information.
The accessible primary abstract provides no verified effect size, comparison rates or uncertainty interval. It therefore supports this measurement distinction, not a numerical prediction or a claim that one algorithm will broaden your sales.
🧠 Why it works
A row can look different without offering useful alternatives. Conversely, shoppers may not recognize a meaningful difference between products. Combining a direct question with observed choices helps an operator locate that mismatch. The proposed question and coffee example are applications, not tested wording from this paper.
✋ Limitations
Full methods and tables were unavailable after checking the publisher and searching author and institutional sources. Sample details, exact treatments and effect sizes remain unresolved. Treat this as a conditional measurement recommendation until full-text verification is complete. It does not establish a paid-order improvement.
💡 The takeaway
Define which kind of variety matters, then measure whether the algorithm changes it.
Start with one recommendation placement and a controlled comparison. Read individual exploration beside store-wide sales concentration, and retain the distinction between views and purchases. Choose a rollout using suitable orders, economics and your stated assortment objective.
📚 Sources and study types
Lee, D., & Hosanagar, K. (2019). How Do Recommender Systems Affect Sales Diversity? A Cross-Category Investigation via Randomized Field Experiment. Information Systems Research, 30(1), 239–259. Randomized retailer field experiment; numeric table checked in the linked institution’s June 2017 author manuscript.
Lee, D., Gopal, A., & Park, S.-H. (2020). Different but Equal? A Field Experiment on the Impact of Recommendation Systems on Mobile and Personal Computer Channels in Retail. Information Systems Research, 31(3), 892–912. Randomized recommendation-versus-best-sellers experiment; mobile/desktop use was not randomized.
Matt, C., Hess, T., & Weiß, C. (2019). A factual and perceptional framework for assessing diversity effects of online recommender systems. Internet Research, 29(6), 1526–1550. Online shop experiment; this guide uses the primary abstract directionally because full methods and estimates were not recovered.
