Slightly OT, but the I/O inflation caused by raising read ahead to 2048K seems to be pretty huge. For a DB server that might only be reading a few K, I can see why it caused an issue.
Anybody knows why Ubuntu did that? I would have expected that in the age of SSDs, read-ahead becomes less useful.
I don't think he says anywhere in the article that Ubuntu default was to use 2048Kb RA. On my 14.04 LTS installation all block devices are set to use 128Kb by default. Maybe they were experimenting with it after the upgrade.
In 2014 this should all really be auto tuned - how hard it is for a user space tool to figure out that a disk is SSD, do a seek latency and read speed test to conclude it will be better off without RA? (Thinking of it though RA brings data in RAM which is still orders of magnitude faster than reading from SSD. So may be there is _some_ benefit if you don't increase the IO overhead enough.)
Wow great debugging article from the performance guru B.Gregg. I'm also not comfortable tracing kernel functions as of yet and remain in user space so far with Sysdig,strace,ltrace.. but a tool such as perf(1) I definitely need a more practical understanding of.
Try to look closely at the methods used to correctly apply perf(1) to the problem at hand. If it feels like you understand _why_ it was used, you'll easily be able to determine if it's the right tool for the job.
The article was really interesting for my developing work, but at the final conclusion I felt as if he oversold it as a tool to diagnose system problems: The issue was that he did not restart the database after tuning readahead, right?
So instead of the "have you tried turning it off-and-on-again", which takes next to no time, we venture off into tracing kernel code?
Read ahead is a kernel level parameter; having tuned it before (also for a Cassandra ring) I would not expect to have to restart any user level processes for it to take effect.
But the real issue is understanding the platform that you are building on, and having the tools to explore how that platform works. I am so thankful that Brenden is out there doing this work and sharing it for the rest of us to benefit from!
Anybody knows why Ubuntu did that? I would have expected that in the age of SSDs, read-ahead becomes less useful.