The unbearable randomness of fuzzing

Fuzzing has become a cornerstone of automated software testing and vulnerability discovery. However, its inherently stochastic nature poses serious challenges for evaluation and reproducibility. Moreover, the underlying sources of this randomness, often unforeseen and uncontrollable, remain largely unexplored by the scientific community. In this paper, we present the first comprehensive study of the different sources of randomness in fuzzing, analyzing its root causes, impact on key performance metrics, and implications for experimental design, revealing several key f indings that can guide future fuzzing practitioners. We systematically analyze the sources of variability, including target behavior, environmental conditions, and fuzzer internals, and quantify their effects through large-scale experiments across diverse configurations. Using fuzzer throughput and code coverage as primary indicators, we reveal that variability persists even under tightly controlled conditions, and that conventional practices, such as fixing the PRNG seed, are insufficient to ensure consistency. Through statistical power analysis and meta-evaluation of real-world benchmarking data, we demonstrate that commonly adopted experimental setups often miss the minimum number of repetitions to produce statistically robust conclusions. We provide empirically grounded guidelines for the number of repetitions needed to reliably compare fuzzers and offer practical insights to improve the rigor of future fuzzing research. Our work serves as a foundational step toward more proper fuzzer evaluations by offering insights into how to handle the unbearable randomness of fuzzing.