The problem is two fold. The first is that most authors and publishers are not very technical. The second is tools like InDesign created EPUBs that would fail the idpf epubcheck tool with errors [1]. Adobe has fixed some of these issues in later versions, but it's expensive for a publisher to re-export all their books.
What this leads to is basically fixing random, one off issues depending on publisher and book. I would definitely suggest not writing your own reader and instead looking at something like Readium (and contribute if you have time!).
I oversee the epub production for an academic publisher, and the epub conversions are created by our typesetter at the end of production. They do indeed do a bit of custom programming (and sometimes manual labor) to make the files come out decent. They're supposed to run epubcheck on every file before delivery. I'm the guy who ends up fixing all those little random errors, so I have to insist that the code be reasonably clean and orderly (they work to a spec document that I prepared). Plus, accessibility is a concern. Poorly coded and disorganized files don't play well for people who need assistive reading systems.
