The test was meant to split visitors equally. B received 9,346 vs. 10,454 for A, or 47.2% instead of 50.0% (χ² = 62.0, p < 0.001). The whole gap comes from a single browser: Safari iOS, where B is missing 1,108 visitors.
Current data will be discarded on relaunch. A skewed split can’t be corrected after the fact.
When visitors are missing on one side only, the groups no longer match. B has most likely lost the visitors for whom the page failed. Any measured difference would mix the variant’s effect with the failure’s.
Split checked for each browser at a 0.001 threshold to avoid false alarms, with p adjusted (Holm) across 4 checks.
| Browser | A | B | B share | p |
|---|---|---|---|---|
| Safari iOS | 2,960 | 1,852 | 38.5% | < 0.001 |
| Chrome Android | 3,380 | 3,379 | 50.0% | 1.00 |
| Chrome desktop | 2,640 | 2,641 | 50.0% | 1.00 |
| Safari macOS | 1,474 | 1,474 | 50.0% | 1.00 |
Split checked for each device type at a 0.001 threshold to avoid false alarms, with p adjusted (Holm) across 2 checks.
| Device | A | B | B share | p |
|---|---|---|---|---|
| Mobile | 6,340 | 5,231 | 45.2% | < 0.001 |
| Desktop | 4,114 | 4,115 | 50.0% | 0.99 |