Since you mentioned the Eigen library: I'm sure it's an extremely elegantly designed library, and I know it's super powerful and efficient. But for someone like me, with fairly limited C++ experience, it was absolutely painful to learn and understand how to use it. Even though it's fairly well documented, I had to really dig through the sources whenever bugs arised. Understanding all these templating abstractions was a huge pain.