There is Telegram-FOSS [0]. I believe they have removed the binaries. It's also the version you can find on F-Droid.
Edit: Also, the maintainer of the Android app is making all his changes locally and then submits all of them in one giant commit. There are pull requests, but he just includes these in his giant commits.
People do have complained about this [1][2], but he obviously doesn't (want to) understand git.
Looking at it, the libs/arch/.so files are just compiled JNI/NDK libraries compiled from source of jni/* files (that's completely standard for native code on Android). Usually the compiled libs aren't checked into the source control though.
Taking my paranoid glasses off, I'd guess they're checked in so contributors don't have to deal with setting up NDK to compile the app.
Edit: Also, the maintainer of the Android app is making all his changes locally and then submits all of them in one giant commit. There are pull requests, but he just includes these in his giant commits.
People do have complained about this [1][2], but he obviously doesn't (want to) understand git.
0: https://github.com/slp/Telegram-FOSS/issues
1: https://github.com/DrKLO/Telegram/pull/76
2: https://github.com/DrKLO/Telegram/issues/1007