[WIP] Attempt at adjusting SVD pullback tolerance to make its accuracy independent of the normalization

[leburgel]

A follow-up on the issue with the accuracy of the SVD pullback for tensors with a very small initial norm (see #276 and QuantumKitHub/MatrixAlgebraKit.jl#89). With the changes here, I managed to achieve the same accuracy that's achieved after #276, but without explicitly normalizing the halfinfinite and infinite environments before the SVD in the forward pass through CTMRG. This doesn't necessarily need to be merged, since I'm not exactly in a rush to remove the normalization before the SVD again now that it's cleat that this massively improved the accuracy of our gradients. Rather, it serves as a demonstration of some underlying issues that could be solved in the future (in particular in MatrixAlgebraKit.jl).

The issues I tracked down and their solutions(?):

• The most important one is that all the default tolerances for the SVD pullback should really be scaled with the norm of the input to the primal computation. In particular, this means
  • Removing the max from the scaling factor in _default_pullback_gaugetol, as was suggested in [Bug] SVD pullback does not properly take into account the norm of the initial matrix MatrixAlgebraKit.jl#89. In particular, this should definitely also be done in MatrixAlgebraKit.jl
  • Similarly scaling the upper and lower tolerance bounds that we used when clamping the tolerance based on the magnitude of the singular values (the lower of which always corresponds to _default_pullback_gaugetol)
  • Scaling the broadening coefficient with the same scaling factor based on the norm of the primal input (this is the aspect I missed in [Bug] SVD pullback does not properly take into account the norm of the initial matrix MatrixAlgebraKit.jl#89, removing the max in default_pullback_gaugetol actually did fix the issue from the MatrixAlgebraKit.jl side)
• In addition, I had to square the broadening coefficient ε as used in _lorentz_broaden to make sure the pullback remained accurate with varying initial normalization. This is more of a heuristic than anything else, but in practice the added square seems to be consistent with how things scale under scaling of the primal input.
  • This also means I had to tweak the default broadening setting and the corresponding test, since now this parameter needs to be larger to get the same relative broadening as before. Things seem to work for a new default value of 1.0e-10, but this was very much more just a guess than anything else.

TLDR: while I definitely don't think we should remove the normalization before the SVD in the CTMRG projector computation in PEPSKit.jl for now, this reflects some of the changes that should end up in MatrixAlgebraKit.jl if we want an SVD pullback whose accuracy is independent of the normalization of the primal input.

Tagging @lkdvos, @pbrehmer and @Jutho.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
src/Defaults.jl	85.71% <ø> (ø)
src/utility/svd.jl	88.80% <100.00%> (+1.29%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.