Add adjoint hessian called tfq.math.inner_product_hessian

[jaeyoo]

This PR adds tfq.math.inner_product_hessian() based on adjoint hessian reverse-mode calculation. It's independent of TensorFlow's Jacobian routine, so you can get the Hessian directly without tf.GradientTape.

Note: due to the large numerical error from the 2nd order finite differencing on cirq.PhasedXPowGate, it will complain if any input circuit has the gate.

Instead of getting gradient values, it accepts weight float values on programs[i] and other_programs[i][j], which can be used for any linear combination of the Hessian terms. You can pass just tf.ones() for the bare values.

[jaeyoo]

FYI. This op shows 20x~100x faster than cirq hessian calculation used in the unit test file.

[MichaelBroughton]

Hi Jae, thanks for writing all of this! I have a few high level questions/comments:
1.

It's independent of TensorFlow's Jacobian routine, so you can get the Hessian directly without tf.GradientTape.

I think we might want to have it be a part of tf.GradientTape that way users who already know to do things like:

with tf.GradientTape():
    with tf.GradientTape():
        <second order stuff>

or https://www.tensorflow.org/api_docs/python/tf/hessians , could just use these with this op and have it just work. I think this means you might need to register another custom_gradient for the gradient op of the forward pass op ?

2. It might make sense to add tests for adj_hessian_util.*** as well. In fact maybe it might be easier to review all of this PR if we first opened a PR with just adj_hessian_util.*** and tests and then moved on to this op PR. Smaller PRs tend to be better.

3. due to the large numerical error from the 2nd order finite differencing on cirq.PhasedXPowGate, it will complain if any input circuit has the gate.

4. This op shows 20x~100x faster than cirq hessian calculation used in the unit test file.

Nice!

Did you try using the analytic form of it's gradient gate instead of finite difference ?