Dispose Scalars implicitly created in Tensor operators

[ds5678]

This pull request modifies the Tensor operators involving primitives to immediately dispose the Scalar object they create, rather than waiting for the garbage collector.

[ds5678]

@dotnet-policy-service agree

[ds5678]

Is there anything I need to do for this to be merged?

[hiyuh]

• FYI, let me share some my findings which are not covered by changes in this current MR, but related to missing TorchSharp.Scalar.Dispose.
  • TorchSharp torch.Tensor.add_ may internally call another torch.Tensor.add_ w/ implicit TorchSharp.Scalar conversions from hardcoded 1.

    • TorchSharp/src/TorchSharp/Tensor/Tensor.Math.cs

      Line 106 in 6ceda53

      return add_(target, 1);

    • TorchSharp/src/TorchSharp/Tensor/Tensor.Math.cs

      Line 129 in 6ceda53

      return add_(scalar, 1);

    • consider in cases of a user intentionally call torch.Tensor.add_ for a inplace op.
    • update (2025/09/05): TorchSharp torch.Tensor.add also have same symptom.

      • TorchSharp/src/TorchSharp/Tensor/Tensor.Math.cs

        Line 58 in 6ceda53

        return add(target, 1);

      • TorchSharp/src/TorchSharp/Tensor/Tensor.Math.cs

        Line 82 in 6ceda53

        return add(scalar, 1);

    • update (2025/09/05): TorchSharp torch.Tensor.addcdiv{,_} also have same symptom.

      • TorchSharp/src/TorchSharp/Tensor/Tensor.Math.cs

        Line 203 in 6ceda53

        return addcdiv(tensor1, tensor2, 1);

      • TorchSharp/src/TorchSharp/Tensor/Tensor.Math.cs

        Line 228 in 6ceda53

        return addcdiv_(tensor1, tensor2, 1);

    • update (2025/09/05): TorchSharp.Modules.BatchNorm.forward also has same symptom.

      • TorchSharp/src/TorchSharp/NN/Normalization/BatchNorm.cs

        Line 40 in 6ceda53

        num_batches_tracked.add_(1);

    • a possible workaround i can think of: carefully call torch.Tensor.add_ (or same kind method) w/ using declared TorchSharp.Scalar typed alpha (or same kind argument) explicitly.
  • TorchSharp torch.nn.functional.leaky_relu internally call torch.Tensor.leaky_relu{,_} w/ implicit TorchSharp.Scalar conversions from double typed negative_slope.

    • TorchSharp/src/TorchSharp/NN/Activation/LeakyReLU.cs

      Line 59 in 6ceda53

      return inplace ? input.leaky_relu_(negative_slope).alias() : input.leaky_relu(negative_slope);

    • other {modules,functions} that directly use specified .NET typed arguments for forward calculation also have possible same symptom.
    • a possible workaround i can think of: carefully call torch.Tensor.leaky_relu{,_} (or other same kind method) w/ using declared TorchSharp.Scalar typed negative_slope (or other same kind arguments) explicitly.
  • TorchSharp torch.arange may internally call another torch.arange w/ implicit TorchSharp.Scalar conversions from hardcoded 0 and/or 1.

    • TorchSharp/src/TorchSharp/Tensor/Factories/Tensor.Factories.cs

      Line 65 in 6ceda53

      return arange(start, stop, 1, dtype, device, requires_grad);

    • TorchSharp/src/TorchSharp/Tensor/Factories/Tensor.Factories.cs

      Line 73 in 6ceda53

      return arange(0, stop, 1, dtype, device, requires_grad);

    • a possible workaround i can think of: carefully call torch.arange w/ using declared TorchSharp.Scalar typed {start,stop,step}.
  • TorchSharp.Modules.SGD.step calls torch.Tensor.{add,add_,mul_} w/ implicit TorchSharp.Scalar conversions from double typed ones.

    • TorchSharp/src/TorchSharp/Optimizers/SGD.cs

      Line 158 in 6ceda53

      grad = grad.add(param, alpha: weight_decay);

    • TorchSharp/src/TorchSharp/Optimizers/SGD.cs

      Line 168 in 6ceda53

      buf.mul_(momentum).add_(grad, alpha: (1 - dampening));

    • TorchSharp/src/TorchSharp/Optimizers/SGD.cs

      Line 172 in 6ceda53

      grad = grad.add(buf, alpha: momentum);

    • TorchSharp/src/TorchSharp/Optimizers/SGD.cs

      Line 181 in 6ceda53

      param.add_(grad, alpha: alpha);

    • other TorchSharp optimizers also have possible same symptom.
    • a possible workaround: derive from TorchSharp.Modules.SGD (or other optimizer class) and override step.

[hiyuh]

• IMHO, relying implict conversions to type inherits IDisposable (i mean TorchSharp.Scalar) inside TorchSharp easily produces serious memory leak that won't be able to workaround by user side.
• most probably, someone would have to do global screening (and hopefully fixing) for TorchSharp.Scalar (and torch.Tensor as well if possible) w/ temporary disabling their implicit conversions, otherwise TorchSharp definetly can not be used for any applications run long time.
  • and most of deep learning based applications usually run long time; e.g. long time training, FPS speed inference iteration.
• another possible solution i can think of is making TorchSharp.DisposeScope to cover TorchSharp.Scalar as well?

[ds5678]

@hiyuh if you want to make a pull request onto my branch with your additions, I'd happily merge it so that they can be part of the review for this pull request.

[hiyuh]

@ds5678
thanks.
i'm grandually moving my task from "avoiding memory leak in my app code" to "fix leak in TorchSharp".
i'll let you know if my outcome does make something better.