Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog.

Experimentation: Putting Research Papers Into Production

Debiased balanced intervealing at ... Interleaving for faster results

• Technique comparing ranking systems more sensitive
• control ranker by generating a ranked list that mixes results
• Requires less users and different metrics O_O

Interleaving is hard to implement

• user preference metrics are different
• lots of event logging to understand engagement

Not enough users

• Interleaving (potentially less users)
• Testing strategies
  • isolated
    • user is exposed to one treatment / experiment
  • overlapping
    • many experiments
    • requires partitioning the product
      • ((we don't do this, things mostly work))
• Papers by google / timothy chan

If the metrics don't move CUPED variance reduction Move to more sensitive metrics

— Kunal