A search ranking decides which products get a chance. The awkward part is that yesterday’s chance becomes today’s evidence: an item placed first may collect more clicks simply because shoppers saw it first.
For marketplace operators, the decision is which signals deserve to influence the next ranking. Separate position-driven attention from relevance, give new products a fair way to be assessed, and check whether the resulting purchases bring customers back.
🔎 What research says
Measure what position itself changes before learning from clicks. Test informative inputs for new products, then judge seller-quality ranking against both current purchases and later customer return. These findings support measured changes to ranking inputs; they do not establish one universal ordering rule.
🗺️ In this guide
1. Separate position-driven clicks from product relevance
📈 Recommendation
Before promoting a product because it attracts clicks, check how much of that attention came from its existing position. For teams that control site search, test position effects within a relevant set of products rather than treating the current order as neutral evidence.
For a marketplace selling headphones, compare equally relevant and available listings through a bounded ranking experiment. Keep query, price and delivery information comparable. Record where each item was actually shown, then follow impressions to clicks and paid orders. This is our proposed application, not the hotel study’s exact test.
Compare the probability of a click with the probability of an order after a click. Let the executive rollout decision depend on suitable purchases and customer outcomes, alongside attention. A busier first slot can simply redistribute visits.
🎓 Findings
Ursu (2018) studied Expedia hotel searches with randomly ordered results. In Table 2, the fitted linear click probability fell by 0.19 percentage points for each position farther down the list. That equals about 1.9 fewer clicks per 1,000 displayed hotel observations in the retained sample. It is a regression estimate for that sample, not a store forecast.
Positions affected which hotels were inspected, but the study did not find a clear position effect on purchase probability after a hotel had been clicked. More prominent hotels could still gain purchases by gaining inspection first. The paper also estimates an average $1.92 extra search cost for one lower position using a computer model. That is a dollar valuation of search effort, not a fee, a hotel-price increase or seller revenue.
🧠 Why it works
A visible item gets a chance to be evaluated. Feeding its extra clicks back into the ranking can mistake exposure for product appeal. Separating those two influences helps the operator learn which headphones shoppers value when given a comparable chance to see them.
✋ Limitations
Expedia’s data exclude searches without clicks and oversample searches ending in bookings. The displayed-observation arithmetic therefore applies to the retained sample; it is not the response of 1,000 ordinary visitors. The dollar estimate depends on the model and four hotel destinations.
Harris, Novarese and Wilson (2022) also found that people’s inspection order in research-paper email alerts was more complex than simply moving down the list. That adjacent setting warns against one universal position curve; it does not prove where to place ecommerce products.
2. Use early signals to assess new products
📈 Recommendation
When a new product has little sales history, test whether information associated with advertising improves your estimate of its relevance. Use that information as an input to evaluate, while maintaining a route for products with little or no advertising history.
For a marketplace launching new kitchen products, compare a candidate ranking model with the current one on held-out traffic. Evaluate new-product exposure, useful visits, paid purchases and the performance of established products together. The launch example and evaluation plan are our proposal.
Ask the team to explain what its advertising signal represents and whether it adds information beyond product attributes and exposure. Validate whether it predicts buyer outcomes; do not substitute the seller’s advertising budget for relevance.
🎓 Findings
Yang and colleagues (2024; first published online in 2023) tested using advertising-propensity information when ranking new products on JD.com. The published abstract reports benefits for consumers and the platform. It concerns learning about new products for organic search listings.
The accessible primary summary does not provide a verified effect size, baseline, confidence interval or exact definition of the advertising input. Those details could not be recovered from the publisher and author-linked full-text routes used here. This supports a conditional test, not a numerical uplift target or a rule that sellers should purchase organic priority.
🧠 Why it works
A new product has not had time to collect the sales history an established product enjoys. An additional informative signal may help the platform decide which new options deserve inspection. Whether a particular signal is informative is a measurement question; spending more is not evidence of a better product.
✋ Limitations
The result comes from JD.com’s ranking system and new-product setting. It does not validate a store’s own proxy, establish long-term effects, or show that the highest advertiser should rank first. Without the full methods and tables, do not copy an algorithm weight, spending threshold or expected sales gain.
3. Test seller quality against repeat purchasing
📈 Recommendation
For a marketplace with reliable transaction data, test whether ranking on seller quality improves customers’ later buying as well as the current visit. Check information hidden by a near-perfect public feedback score before choosing the ranking input.
For a homeware marketplace, compare the current ranking with one that gives validated transaction-quality information additional weight among relevant listings. Check delivery failures, complaints, current conversion and subsequent paid purchases. These are proposed local measures, not all outcomes tested in the paper.
Define the follow-up period and customer groups before the test. For an executive, a rollout needs evidence that any gain in future buying justifies the effects on relevance, seller exposure and present purchases. Keep a control group long enough to observe that tradeoff.
🎓 Findings
Nosko and Tadelis (2026) report an eBay experiment that gave more prominence to sellers with a higher share of transactions generating positive feedback, counting all transactions in the denominator. This differs from the usual positive share among transactions with feedback.
With the quality-weighted ranking, 61.85% of users purchased again within 180 days, compared with 61.55% under the standard ranking. The gap was 0.30 percentage points: about three additional returning purchasers per 1,000 users, with a reported 95% confidence interval of roughly two to four per 1,000. This is the experiment’s overall assignment comparison, not a forecast for another marketplace.
The study found no reduction in conversion during the experiment. The repeat-purchase effect was smaller for highly experienced buyers than for less experienced buyers. The researchers tested that difference directly; it is not inferred merely because one subgroup was statistically significant and another was not.
🧠 Why it works
A completed order can look successful in the current-session report while leaving a customer reluctant to return. Incorporating a useful quality signal into ranking may improve the match with a seller. This is a reason to judge the ranking over a customer relationship, rather than award success to the first click.
✋ Limitations
The experiment ran on eBay’s US site in December 2011–January 2012. Assignment used browser cookies; the analysis retained users linked to a single browser identifier to reduce mixed exposure. The adjusted estimate was smaller, so three per 1,000 is not an invariant effect. The tested feedback measure is a proof of concept, not a universal quality score or a recommended cutoff.
💡 The takeaway
A useful ranking learns from comparable exposure and earns its place through buyer outcomes.
Start with a bounded test in one relevant category. Record the complete route from display position to inspection, purchase and later buying. Decide which input to change only after distinguishing those outcomes and checking who benefits.
📚 Sources and study types
Ursu, R. M. (2018). The Power of Rankings: Quantifying the Effect of Rankings on Online Consumer Search and Purchase Decisions. Marketing Science, 37(4), 530–552. Published study. Randomized hotel rankings plus a computer model of sequential search; the dollar amount is a model estimate.
Yang, J., Sahni, N. S., Nair, H. S., & Xiong, X. (2024; online 2023). Advertising as Information for Ranking E-Commerce Search Listings. Marketing Science, 43(2), 360–377. Published study. JD.com field experiment; primary abstract reviewed, quantitative methods and results not recovered.
Nosko, C., & Tadelis, S. (2026). The limits of reputation in platform markets: An empirical analysis and field experiment. Quantitative Marketing and Economics, 24, article 9. Open-access published study. eBay transaction records and browser-randomized ranking experiment; the reported effect here is the assignment comparison.
Harris, M. N., Novarese, M., & Wilson, C. M. (2022). Being in the right place: A natural field experiment on the causes of position effects in individual choice. Journal of Economic Behavior & Organization, 194, 24–40. Published study. Randomized research-paper email order and downloads; adjacent evidence used only to limit a universal position rule.
