Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog.
Experimentation: Putting Research Papers Into Production
Debiased balanced intervealing at ...
Interleaving for faster results
- Technique comparing ranking systems more sensitive
- control ranker by generating a ranked list that mixes results
- Requires less users and different metrics O_O
Interleaving is hard to implement
- user preference metrics are different
- lots of event logging to understand engagement
Not enough users
- Interleaving (potentially less users)
- Testing strategies
- isolated
- user is exposed to one treatment / experiment
- overlapping
- many experiments
- requires partitioning the product
- ((we don't do this, things mostly work))
- Papers by google / timothy chan
If the metrics don't move
CUPED variance reduction
Move to more sensitive metrics
— Kunal