Example: per week we take 40,000 unique home improver
search engine click thrus. Around 3,000 jobs submitted per week.
original conversion 7.5%. If you do A/B testing on a solution
which has a 4% better conversion rate (7.8%) for one week your
conclusion as to which is the better solution would have around a 5%
chance of being incorrect
NB - works less effectively on lower Click Thrus, as this is judging improvements, not identical results
|