I think the branch predictor can be primed with well-crafted data? (start off with a long series of 0s for X)

For the other aspects like getting the cache timing, that does seem harder. I don't really know if it's feasible in practice.