Measurement and applications of position bias in a marketplace search engine
Search engines intentionally influence user behavior by picking and ranking the list of results. Users engage with the highest results both because of their prominent placement and because they are typically the most relevant documents. Search engine ranking algorithms need to identify relevance while incorporating the influence of the search engine itself. This paper describes our efforts at Thumbtack to understand the impact of ranking, including the empirical results of a randomization program. In the context of a consumer marketplace we discuss practical details of model choice, experiment design, bias calculation, and machine learning model adaptation. We include a novel discussion of how ranking bias may not only affect labels, but also model features. The randomization program led to improved models, motivated internal scenario analysis, and enabled user-facing scenario tooling.
Code (0)
등록된 구현이 없습니다.
Tasks
PositionSimilar Papers 제목 키워드 기반
Causal Estimation of Position Bias in Recommender Systems Using Marketplace Instruments
Information retrieval systems, such as online marketplaces, news feeds, and search engines, are ubiquitous in today's digital society. They facilitate information discovery by ranking retrieved items on predicted relevan…
Information RetrievalPositionRecommendation SystemsRetrievalUnbiased Platform-Level Causal Estimation for Search Systems: A Competitive Isolation PSM-DID Framework
Evaluating platform-level interventions in search-based two-sided marketplaces is fundamentally challenged by systemic effects such as spillovers and network interference. While widely used for causal inference, the PSM …
Causal InferenceMagentic Marketplace: An Open-Source Environment for Studying Agentic Markets
As LLM agents advance, they are increasingly mediating economic decisions, ranging from product discovery to transactions, on behalf of users. Such applications promise benefits but also raise many questions about agent …
Valid and Unobtrusive Measurement of Returns to Advertising through Asymmetric Budget Split
Ad platforms require reliable measurement of advertising returns: what increase in performance (such as clicks or conversions) can an advertiser expect in return for additional budget on the platform? Even from the persp…
validSkillFuzz: Fuzzing Skill Composition for Implicit Intents Discovery in Open Skill Marketplaces
Large Language Model (LLM)-based agents increasingly automate software engineering tasks through reusable skills, natural-language instruction documents that guide planning and execution. Open skill marketplaces enable u…
Intent Discovery