Split testing is treated as the mature way to make product decisions, and for a consumer product with substantial traffic it is. For a B2B product with four hundred accounts, or a young marketplace with a few thousand monthly users, the arithmetic does not work: detecting a realistic improvement requires far more observations than you will collect in a reasonable period, and a test stopped early because the numbers looked good is not evidence of anything.
The specific trap is stopping when significance first appears. With small samples, the difference between variants fluctuates widely, and if you check daily you will eventually see a result that crosses the threshold by chance. Teams that do this accumulate a series of wins that do not compound into any improvement in the overall metric, which is the observable symptom of testing that is not working.
So calculate the required sample before running anything. Take your current conversion rate, the improvement you would consider worth shipping, and your traffic, and work out how long the test must run. If the answer is eleven months, you have learned something useful: this decision cannot be made this way, and pretending otherwise wastes the eleven months as well as the decision.
What works instead at low volume is qualitative and comparative. Watch eight people attempt the task and you will usually find the problem without any statistics. Ship the change to everyone and compare a fortnight before against a fortnight after, accepting that this is weaker evidence and being explicit about the confounders. Or make the change where the reasoning is strong enough that you would ship it without a test — which describes most interface improvements honestly assessed.
Reserve the statistical machinery for the decisions where it genuinely pays: pricing, a headline on a high-traffic landing page, an email subject line, anything where you have volume and the change is easily reversible. And be honest in the write-up about what a result does and does not support. A team that says we believe this is better and here is our reasoning is more trustworthy than one presenting a p-value derived from two hundred users.
