> standard input-output RNNs have to take in part of the data with every time step

Well they can but I don't see they why they have to. And couldn't your network also take input and give output for all times?

> First, we do parameterize a theta that changes with time, using a hypernet.

Ah I see. Did you end up using it in the final model? I don't see that in the mnist example, but I could be missing it as I only skimmed the code.