Experiment Writeup Template

‘Find the Edge’ Framework

• Be aggressive: find the edge and work backwards:
  • Oftentimes when you're working through solution designs there's a range of possible effect:
    • High Bound; Solution A: high effect on metric a and b
    • Low Bound Solution C: low effect on metric a and b
    • And a million possibilities in between.
    • Opportunity cost of starting anywhere in the middle— maybe the negative effect on the high bound solution was same as medium.
    • Similarly, testing low bound is a failsafe— we may find that even the low bound has a high negative effect on metric b. At which point, we may need to say this approach to solving the problem may not be viable; and we need to go back to the drawing board.

Solution Design Guidance

• Think big and small
  • At this point our sample sizes are pretty low, so we're going to be looking for features with large detectable effects.
  • In experimentation, generally speaking, small changes have small effect and large changes have large effect— in both directions.
  • That means, we pursue large scale changes to the UI/UX. That said, if they hypothesis is strong that a small change could have a big impact, by all means.
• Set specific metrics goals: Understand cause and effect
  • These should be very specific— often times you simply hear that we must drive metric N x% points. That's good, but as we know, with every action is an opposite and equal reaction— so one KPIs gain might be another's loss— so we need to give the team clear guidance around loss allowances ratios: e.g. 10% increase in signups against 1% decrease in engagement.
• Build for launch: buid an experiment you can graduate on the fly
• Bucketing and Exposure:
  • Going forward, when assessing bucketing logic, it’s not simply about identifying what page a user sees the change on, but also which audience properties are required to have the option to see the change. Then narrow to a range of funnel steps (if more than one) that we we’re looking to drive and isolate the experiment just to them.
• If possible, do not run multiple experiments on the same surface targeting the same KPI without making the audiences mutually exclusive. You wont know which led to which without down stream isolation of one experiment behavior when users in other experiment’s control.