It doesn't really have anything to do with the site... it has to do with what the difference is vs. the total size of the sample. If you see a 50% increase but only had 3 people in your sample, that is not statistically significant because the probability of that happening by chance is so high. To make this concrete, if the actual proportion was 4:1 against that figure (as in, you "should" have measured a whopping decrease to 25% the original conversion), you'd still measure the totally incorrect 50% increase in one out of every ten attempts at that measurement.

I failed to make the point of my question clear enough. Nickpotier critiqued the results without offering up any numbers. I wanted to see what he thought the numbers should have been for each example given.

Roughly a good sample size is around -log(d/2)/(2 e a^2) where d is the significance you want and a is the difference in conversion rates you hope to measure. See http://www.win-vector.com/blog/2013/03/a-bit-more-on-sample-...

I read that and I have no idea what those values are. Is there an example of this with actual real-life data?

He gave you a reference. Google "statistical sample size". There are several calculators in the results that you can play around.

Everybody should take a college level course in statistics, even poets and painters.