ReleaseAsync on Postgres (Connection is not open)

[r-ising]

Hi,

Thank you for this great library.

We are encountering a System.InvalidOperationException: Connection is not open error in our production environment.
We see this error about 100-200 times per day under load.

Environment:

• Library Version: latest
• Npgsql Version: 9.0.4 and 10.0.1
• Framework: .NET 9 and .NET 10
• Database: PostgreSQL 18

Reproduction:

I tried to reproduce the issue locally using a PostgreSQL Docker container (with increased connection limit: 999) and the following console application. The issue seems to appear under high concurrency.

using Medallion.Threading;
using Medallion.Threading.Postgres;

var list = Enumerable.Range(1, 200).ToList();
var tasks = new List<Task>();
var error = new List<Exception>();
IDistributedLockProvider synchronizationProvider =
	new PostgresDistributedSynchronizationProvider(
		"User ID=xxxx;Password=xxxx;Database=xxxx;Host=127.0.0.1;Port=5432;Pooling=true");
for (var j = 0; j < 30; j++)
{
	foreach (var id in list)
	{
		var x = Task.Run(
			async () =>
			{
				try
				{
					var timeout = TimeSpan.FromMinutes(5);
					var key = "X" + id;
					await using var writeLock =
						await synchronizationProvider.CreateLock(key)
							.AcquireAsync(timeout);

					await Task.Delay(
						Random.Shared.Next(1, 10),
						writeLock.HandleLostToken);
				}
				catch (Exception ex)
				{
					Console.WriteLine(ex);
					Console.WriteLine(ex.StackTrace);
					Environment.Exit(1);
				}
			});
		tasks.Add(x);
	}

	await Task.WhenAll(tasks);
}

Console.WriteLine("Task completed");

Stack Trace:

System.InvalidOperationException: Connection is not open
   at Npgsql.ThrowHelper.ThrowInvalidOperationException(String message)
   at Npgsql.NpgsqlCommand.CheckAndGetConnection()
   at Npgsql.NpgsqlCommand.Prepare(Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.PrepareAsync(CancellationToken cancellationToken)
   at Medallion.Threading.Internal.Data.DatabaseCommand.PrepareIfNeededAsync(CancellationToken cancellationToken) in /_/src/DistributedLock.Core/Internal/Data/DatabaseCommand.cs:line 182
   at Medallion.Threading.Internal.Data.DatabaseCommand.ExecuteAsync[TResult](Func`3 executeAsync, Func`2 executeSync, CancellationToken cancellationToken, Boolean disallowAsyncCancellation, Boolean isConnectionMonitoringQuery) in /_/src/DistributedLock.Core/Internal/Data/DatabaseCommand.cs:line 81
   at Medallion.Threading.Postgres.PostgresAdvisoryLock.ReleaseAsync(DatabaseConnection connection, PostgresAdvisoryLockKey key, Boolean isTry) in /_/src/DistributedLock.Postgres/PostgresAdvisoryLock.cs:line 278
   at Medallion.Threading.Internal.Data.MultiplexedConnectionLock.ReleaseAsync[TLockCookie](IDbSynchronizationStrategy`1 strategy, String name, TLockCookie lockCookie) in /_/src/DistributedLock.Core/Internal/Data/MultiplexedConnectionLock.cs:line 154
   at Medallion.Threading.Internal.Data.MultiplexedConnectionLock.ReleaseAsync[TLockCookie](IDbSynchronizationStrategy`1 strategy, String name, TLockCookie lockCookie) in /_/src/DistributedLock.Core/Internal/Data/MultiplexedConnectionLock.cs:line 168
....

It looks like the connection might be closed or returned to the pool unexpectedly while the lock is trying to release or prepare a command.

Do you have any idea how to fix this or what might be causing the race condition?

Thanks

[madelson]

Thanks for reporting @r-ising .

(1) Does the issue still reproduce for you if you avoid initializing the HandleLostToken?

await Task.Delay(Random.Shared.Next(1, 10));

(2) With your setup if you run my code snippet from npgsql/npgsql#4209 does the problem occur? I wonder if that bug came back.

[r-ising]

Hi @madelson ,

thanks for your fast response.

(1)
You're right. If I remove HandleLostToken, the issue is gone.

(2)
No, I tested the code for a few minutes, but I couldn't reproduce it. That bug seems to be fixed in Npgsql.

So, should we avoid using HandleLostToken, or is there another solution?

[madelson]

Thanks @r-ising this is helpful. I will treat this as a bug in the library and see if I can figure out how it is happening.

In the meantime, it sounds like avoiding HandleLostToken is a reasonable solution. In general I would consider HandleLostToken to be a special-case API. It adds overhead by firing additional queries against the DB and requiring those queries to be canceled on every lock dispose. I would only use it when you are (a) holding a lock for a long time, (b) have experienced/expect flaky DB connections, and (c) are very sensitive to the lock semantics being violated.

UPDATE: I'm having trouble reproducing the issue using your repro code. I'm running postgres 16 locally (Windows) and I added the following test to PostgresDistributedLockTest.cs:

[Test]
public async Task Repro()
{
    var list = Enumerable.Range(1, 200).ToList();
    var tasks = new List<Task>();
    var error = new List<Exception>();
    IDistributedLockProvider synchronizationProvider =
        new PostgresDistributedSynchronizationProvider(TestingPostgresDb.DefaultConnectionString);
    for (var j = 0; j < 30; j++)
    {
        foreach (var id in list)
        {
            var x = Task.Run(
                async () =>
                {
                    var timeout = TimeSpan.FromMinutes(5);
                    var key = "X" + id;
                    await using var writeLock =
                        await synchronizationProvider.CreateLock(key)
                            .AcquireAsync(timeout);

                    await Task.Delay(
                        Random.Shared.Next(1, 10),
                        writeLock.HandleLostToken);
                });
            tasks.Add(x);
        }

        await Task.WhenAll(tasks);
    }
}

I also tried updating npgsql to 9.0.4; still no dice. Remaining differences are postgres version (16 vs 18) and .NET version (8 vs 9). **Would you be able to attempt the repro on your device but through the DistributedLock test suite as described?