You'd measure when the LED is off, but you could do this many times a second

Well you can't measure they brightness of the LED that is emitting. You can ambient light but not sure that's what matters in this application.

Also don't forget about response time. You may not be able to do it that fast. Depends on the led