What I've read is most of the lag in the more optimized touchscreen stacks (read=iOS) is due to filtering and smoothing, which takes a few samples to do. The worse stacks (early android) had poor drivers and a lot of layers of abstraction, or worse, Java (maybe this is what you mean when you say "UI Graphics", but the raw numbers we're talking about here are on a far lower layer than scrolling in lists).
Graphics rendering lag is at worst 2 frames of video (double-buffering).
If you had read the PDF, 5ms is the "minimum touch time", which is not latency, but a minimum touch time it can register.
The same software has no problem rendering complex games and doing all kinds of game logic, all in 16.7ms. What makes you think asking for a touch input will take 15 times longer?
I already gave you an example of software stack impact on latency - Android sound lag went down from >250ms to respectable 10ms for some Nexus devices. Blame buffers.
Remember Androids ~>200ms sound lag?