[Unmanaged Memory] Memory bloating on heavily looped calls due to delayed finaliziation by GC

[Nucs]

Overview

NDArray's unmanaged buffers are released through the CLR finalizer thread, not deterministically. In hot allocation paths (training loops, batch operations, benchmark sweeps), the finalizer queue cannot keep up with allocation pressure, causing native memory to stay committed long after the managed NDArray wrappers are unreachable. This inflates process working set, triggers Windows page-trimming on actively used buffers, and produces non-linear slowdowns the more the application allocates.

Reproduction

Allocate a large number of NDArrays in sequence, drop the managed references, and observe that a plain GC.Collect() does not return the unmanaged memory. Only GC.WaitForPendingFinalizers() actually releases the buffers.

using System;
using System.Diagnostics;
using NumSharp;

var p = Process.GetCurrentProcess();

GC.Collect(); GC.WaitForPendingFinalizers(); GC.Collect();
long baseline = p.WorkingSet64;

// Allocate 800 MiB of unmanaged memory backing 100 NDArrays (8 MiB each).
NDArray[] arrs = new NDArray[100];
for (int i = 0; i < 100; i++)
    arrs[i] = new NDArray(NPTypeCode.Double, new Shape(1_000_000), fillZeros: true);

p.Refresh();
Console.WriteLine($"after alloc:                {(p.WorkingSet64 - baseline)/1024/1024,5} MiB");

// Drop every managed reference.
for (int i = 0; i < 100; i++) arrs[i] = null;
arrs = null;

// One full collection — should be enough to release everything if NDArray
// owned its native buffer deterministically.
GC.Collect();
p.Refresh();
Console.WriteLine($"after GC.Collect:           {(p.WorkingSet64 - baseline)/1024/1024,5} MiB");

// Now force the finalizer thread to drain its queue.
GC.WaitForPendingFinalizers();
GC.Collect();
p.Refresh();
Console.WriteLine($"after WaitForPendingFinal:  {(p.WorkingSet64 - baseline)/1024/1024,5} MiB");

Expected behavior

NumPy frees array buffers synchronously when the last reference drops (CPython refcount → array_dealloc → free(fa->data)). A loop like:

import numpy as np
import os, psutil
p = psutil.Process(os.getpid())
print(p.memory_info().rss // 1024 // 1024, "MiB")
for _ in range(100):
    a = np.zeros(1_000_000, dtype=np.float64)
print(p.memory_info().rss // 1024 // 1024, "MiB")

…never grows the resident set, because each a is freed during the next assignment when refcount drops to zero. No GC pass required.

NumSharp should give the same memory-safety guarantee: once an array is no longer reachable, its unmanaged buffer should be released without depending on whichever pass of the CLR finalizer thread happens to drain it.

Actual behavior

Output from the reproduction above on Windows 11 / .NET 10:

after alloc:                  765 MiB
after GC.Collect:             519 MiB    ← ~497 MiB still held
after WaitForPendingFinal:      0 MiB

After GC.Collect() alone, roughly half a gigabyte of unmanaged memory was still committed even though no managed reference points at it. Only the explicit WaitForPendingFinalizers() call drained the queue and released the buffers.

Performance impact

The same effect surfaces as a real wall-time regression under sweep workloads. The np.concatenate benchmark sweep (55 scenarios run in sequence) shows:

Scenario	NumPy	NumSharp in isolation	NumSharp after the full sweep
out_mixed_to_float64 (1M float32 + 1M int32 → 2M float64, out=)	0.50 ms	0.52 ms (≈ NumPy)	2.54 ms (5.05× NumPy)

In isolation the cross-dtype copy is competitive with NumPy. Run inside a sweep that has already churned through dozens of large allocations, the same call slows down ~5× — not because the kernel changed, but because cumulative working-set pressure from undrained finalizers causes the OS to trim active pages of the destination buffer, which then fault back in during the copy.

Root cause (mechanism)

The chain that holds an NDArray's native buffer alive looks like:

NDArray  →  UnmanagedStorage  →  IArraySlice  →  UnmanagedMemoryBlock<T>  →  Disposer  (~Disposer())
                                                                                  └── NativeMemory.Free

• NDArray is a plain class. It does not implement IDisposable and has no finalizer.
• The native free call lives in ~Disposer(), which runs on the CLR finalizer thread.
• The finalizer thread is single-threaded at THREAD_PRIORITY_NORMAL. Under sustained allocation pressure it falls behind.
• Until a Disposer's finalizer actually runs, its NativeMemory.Alloc-backed buffer stays committed even though it is unreachable from managed code.
• GC.AddMemoryPressure is already wired up (UnmanagedMemoryBlock1.cs:1012`) to nudge the GC scheduler, but it only makes collections happen sooner — it cannot run finalizers synchronously.

The user cannot release a buffer eagerly: there is no public path that drops the underlying allocation when the user knows the array is no longer needed.

Where this hurts in practice

• Training / inference loops: each iteration creates intermediate arrays (a + b, a * c, …). All of them queue for finalization. Working set climbs across epochs until either GC catches up or the OS pages.
• Batch processing: hundreds of NDArrays allocated and abandoned per batch. The finalizer queue lags behind the allocation rate.
• Benchmarks / test sweeps: sequential scenarios contaminate each other's measurements because the finalizer queue from earlier scenarios is still draining when later ones run.
• Long-running services: working set can grow to hundreds of MiB above the actual live set, increasing the chance of OS-level page trimming and the soft page faults that follow.

Acceptance criteria

A fix for this problem should:

• Let callers release an NDArray's unmanaged buffer synchronously, without depending on GC.Collect / GC.WaitForPendingFinalizers.
• Stay safe in the presence of views and aliases (releasing one reference must not leave dangling pointers in another).
• Keep the finalizer-based path as a safety net for code that doesn't opt in.
• Not break any of the ~8500 existing unit tests.
• Show measurable improvement on the concat benchmark sweep's out_mixed_to_float64 and prom_* scenarios when arrays are explicitly released.

Repro environment

• OS: Windows 11 (the working-set trimming behavior is OS-dependent; Linux exhibits the symptom differently)
• .NET: 8.0 and 10.0 — both reproduce
• NumSharp: nditer branch as of 2026-05-21
• Verified empirically: script.cs reproduction (see Reproduction section) and test/NumSharp.Benchmark concat sweep.

[Nucs]

Proposal: atomic reference counting on the unmanaged buffer, with IDisposable on NDArray

The problem in the issue boils down to a single missing piece: there is no synchronous path from "user knows this array is dead" to "native buffer is freed." Everything goes through ~Disposer() on the single CLR finalizer thread.

I propose two cooperating layers. Each is small and independently committable.

L1 — make Disposer reference-counted

Replace the Disposed boolean in UnmanagedMemoryBlock<T>.Disposer with an atomic refcount. Every logical owner (typically an NDArray) holds one ref; the buffer is freed the instant the last reference is dropped, on the calling thread.

Encoding:

• _refCount >= 0 — live, with N owners
• _refCount == -1 — released; NativeMemory.Free already called

Three operations:

public bool TryAddRef()
{
    if (_type == AllocationType.Wrap) return true;          // non-owning wrap = immortal
    long c;
    do {
        c = Interlocked.Read(ref _refCount);
        if (c < 0) return false;                            // already released
    } while (Interlocked.CompareExchange(ref _refCount, c + 1, c) != c);
    return true;
}

public void Release()
{
    if (_type == AllocationType.Wrap) return;
    long n = Interlocked.Decrement(ref _refCount);
    if (n == 0)
    {
        // Claim the free transition 0 → -1 atomically. If a racing AddRef
        // bumps it back to 1 before our CAS, we don't free — the new owner
        // is responsible.
        if (Interlocked.CompareExchange(ref _refCount, -1, 0) == 0)
            ReleaseUnmanagedResources();
    }
    else if (n < 0)
    {
        // Stray Release on already-released block. Each thread compensates
        // its own Decrement so concurrent strays self-balance one-for-one.
        Interlocked.Increment(ref _refCount);
    }
}

ReleaseUnmanagedResources is guarded by a single-shot Interlocked.Exchange(ref _freed, 1) so the existing finalizer path (~Disposer) and the new Release path can never double-free under any race. The finalizer stays in place as a safety net for code that doesn't opt in:

~Disposer()
{
    if (Volatile.Read(ref _freed) == 0)
        ReleaseUnmanagedResources();
}

L2 — make NDArray implement IDisposable

NDArray becomes IDisposable with idempotent Dispose() and a finalizer that releases the held reference if the user forgot.

public partial class NDArray : ..., IDisposable
{
    // int (not byte) — Interlocked.Exchange's narrowest type.
    private int _disposed;

    public bool IsDisposed => Volatile.Read(ref _disposed) != 0;

    public void Dispose()
    {
        if (Interlocked.Exchange(ref _disposed, 1) != 0) return;   // idempotent
        Storage?.InternalArray?.Release();
        GC.SuppressFinalize(this);
    }

    ~NDArray()
    {
        // Safety net for the "user forgot" case — runs on finalizer thread,
        // same as today, but now it cooperates with the refcount.
        if (Volatile.Read(ref _disposed) == 0)
            Storage?.InternalArray?.Release();
    }
}

Every concrete constructor calls InitializeArc() at the very end, which does a single TryAddRef on the underlying IArraySlice so this NDArray's logical ownership is tracked. Views participate by the same mechanism — slicing, transposing, reshaping all flow through an NDArray constructor, so every view ends up holding its own ref.

IArraySlice and IUnmanagedMemoryBlock each grow three thin members — TryAddRef, Release, IsReleased — that forward to the underlying Disposer.

What this changes for callers

// Existing code keeps working exactly as today — finalizer eventually frees.
var result = (a + b) * c;

// Opt-in deterministic release where it matters:
using var temp = np.zeros(big_shape);

// Or explicit, when `using` doesn't fit:
var big = np.empty(huge_shape);
DoWork(big);
big.Dispose();   // freed NOW

Safety invariants this preserves

• Disposing a parent does not invalidate live views. Each view holds its own ref; the buffer survives until every view is also disposed (or finalized).
• Dispose() is idempotent — calling it twice, or in parallel from many threads, is safe.
• Concurrent AddRef vs Release cannot produce a state where TryAddRef returned true but the block has been freed (verified empirically over 10,000 race iterations).
• Non-owning wraps (AllocationType.Wrap) bypass refcounting entirely so multiple MemoryBlock instances can share Disposer.Null without interfering.
• GCHandle-backed allocations release only the pin, never the managed array.

Acceptance criteria mapped to the issue

Issue requirement	How this proposal satisfies it
Synchronous release without GC.Collect	Release() → NativeMemory.Free runs on the calling thread when refcount hits 0
View safety	refcount transitivity — disposing a parent only decrements; buffer stays alive while any view holds a ref
Finalizer safety net preserved	~Disposer() and ~NDArray() both still exist; only the "user forgot" path uses them
No existing test regressions	All ~8500 prior tests continue to pass on net8.0 and net10.0
Measurable improvement on the sweep scenarios	The out_mixed_to_float64 scenario stays at ~0.5 ms (NumPy parity) when out= and source arrays are explicitly disposed between iterations

Behavior shown empirically

The same 800 MiB / 100 NDArray loop from the issue, with the proposed change and an explicit arr.Dispose() call per element:

after alloc:                  765 MiB
after Dispose-each, no GC:      0 MiB     ← every buffer freed atomically

10,000 alloc + Dispose cycles in a tight loop measure ~1 µs per cycle on .NET 10 / Windows 11 and leave the process working set unchanged.

Scope / non-goals

• This proposal does not add an "auto-dispose scope" / arena mechanism (np.scope() or similar). NumPy has no equivalent, and the using keyword fills the same role for callers who want determinism.
• This proposal does not change Disposer's visibility, the MemoryBlock struct's public API, or any storage-level allocation behavior.
• The existing MemoryBlock.Free() / ArraySlice.DangerousFree() keep their original "force release everything" semantics for backwards compatibility.

Migration cost

Component	LOC	Risk
Refcount in Disposer	~50	low — internal class
ARC forwarding in UnmanagedMemoryBlock<T>, ArraySlice<T>, and the two interfaces	~30	low — additive
NDArray.Dispose + finalizer + InitializeArc calls from each public ctor	~40	low — additive, no behavioral change for callers who don't call Dispose
Test coverage (ArcLifecycleTests)	~600	—

Total: a single PR, ~700 lines net additive, no public breaking changes.