Thanks for this awesome peace of research! I'm really looking forward to further
developments in the field :)
I have two small questions regarding the paper:
1. When comparing to Normalizing Flows (planar flows), in Section 4.1, how were
these fitted in the Maximum Likelihood Training section? If I understand
it correctly NF's don't have a closed form inverse, s.t. ML training should
not be possible.
2. Do you encounter any issues regarding stability during training? Other Flow
based approaches such as Glow use certain tricks to ensure that the Flow
initial reduces to an identity transform, to increase stability and ensure
reliable convergence.
1. Great question! You're correct that standard NF isn't efficiently invertible. CNF is, and we wanted a fair comparison. So for this experiment, we reversed the direction that NF transforms the data, so that it goes from the data to the latent space. Training this way means that you can't use the resulting model as a generator, but it at least let us compare likelihoods with CNF for this paper.
2. We had to set the error tolerance relatively small during training to keep the gradients stable. I don't think we used any fancy initialization tricks, but to be honest I have to ask Ricky Chen and Will Grathwohl, who ran all the FFJORD experiments.
I have two small questions regarding the paper:
1. When comparing to Normalizing Flows (planar flows), in Section 4.1, how were these fitted in the Maximum Likelihood Training section? If I understand it correctly NF's don't have a closed form inverse, s.t. ML training should not be possible.
2. Do you encounter any issues regarding stability during training? Other Flow based approaches such as Glow use certain tricks to ensure that the Flow initial reduces to an identity transform, to increase stability and ensure reliable convergence.