Interesting. I recall how we had to decrease the clock speed of the processors on a high-performance machine in order to reduce errors - these were detected by simple reproducibility tests. With code optimised to sustain a huge percentage of peak speed and a thousand or so processors, such problems can become manifest, even in the chilly environment in which the machine was housed.