backup: TestBackupRestoreWithConcurrentWrites failed #159445
Description
backup.TestBackupRestoreWithConcurrentWrites failed on master @ f30ebeb04d4bc4be12fffbd175d35a77492730d6:
Failed with:
=== RUN TestBackupRestoreWithConcurrentWrites
test_log_scope.go:171: test logs captured to: outputs.zip/logTestBackupRestoreWithConcurrentWrites174361056
test_log_scope.go:82: use -show-logs to present logs inline
test_server_shim.go:180: automatically injected an external process virtual cluster under test; see comment at top of test_server_shim.go for details.
Fatal entries found in Cockroach logs:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 incompatibility detected; lock by transaction 233a3ef6-a933-488a-a799-7926f70af3e8 with strength Exclusive incompatible with an already held lock by f226c093-597b-4ced-b219-91399896f129 with strength Exclusive
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +HINT: You have encountered an unexpected error.
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +Please check the public issue tracker to check whether this problem is
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +already tracked. If you cannot find it there, please report the error
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +with details by creating a new issue.
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +If you would rather not post publicly, please contact us directly
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +using the support form.
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 +We appreciate your feedback.
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 46569 [running]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xcfbdbe?, 0xc00b8cced0?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/allstacks/allstacks.go:38 +0x133
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/allstacks/allstacks.go:19
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0xc001eef9b0, 0x24}, {0x6d0b44a, 0x1}, {0x6d0b44a, 0x1}, {0x6c471b2, 0x6}, {0x6d0b44a, ...}}, ...})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/clog.go:294 +0xc6
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x7ace310, 0xc017e88138}, 0x2, 0x4, 0xf, 0x0?, {0x6c41938, 0x2}, {0xc009e5b488, 0x1, ...})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/channels.go:104 +0x5a5
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/channels.go:34
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.loggerKvExec.Fatalf(...)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:6577
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockAcquired(0x7ace310?, {0x7ace310, 0xc017e88138}, 0x1f?)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/concurrency/concurrency_manager.go:603 +0xcf
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadOnlyLocalEvalResult(0xc01218bb08, {0x7ace310, 0xc017e88138}, 0x0?, {0x0, {0x0, 0x0, 0x0}, {0xc012e31a20, 0x1, ...}, ...})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/replica_read.go:541 +0x278
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch(0xc01218bb08, {0x7ace310, 0xc017e88138}, 0xc017e6d2c0, 0xc0129a2340)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/replica_read.go:199 +0xff8
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc01218bb08, {0x7ace310, 0xc017e88138}, 0xc017e6d2c0, 0x703d8c8)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/replica_send.go:528 +0x388
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc01218bb08, {0x7ace310, 0xc017e880a0}, 0xc017e6d2c0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/replica_send.go:197 +0x5ae
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc00442e808, {0x7ace310?, 0xc017e88008?}, 0xc017e6d2c0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/store_send.go:188 +0x7e5
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc001b5a780, {0x7ace310, 0xc017e88008}, 0xc017e6d2c0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/stores.go:203 +0x7e
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0022c7508, {0x7ace310?, 0xc017e88008?}, {0xc008477500?}, 0xc017e6d2c0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/server/node.go:1669 +0x64e
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0022c7508, {0x7acbf18, 0xc017e66c90}, 0xc017e6d2c0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/server/node.go:1862 +0x27c
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x7acbf18?, 0xc017e66c90?}, {0x6ba7ac0?, 0xc017e6d2c0?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:11751 +0xcb
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x7acbf18, 0xc017e66c90}, {0x6ba7ac0, 0xc017e6d2c0}, 0xc017e84100, 0xc017e05578)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:60 +0x4a6
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !google.golang.org/grpc.getChainUnaryHandler.func1({0x7acbf18?, 0xc017e66c90?}, {0x6ba7ac0?, 0xc017e6d2c0?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:1177 +0x153
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x7acbf18, 0xc017e66c90}, {0x6ba7ac0, 0xc017e6d2c0}, 0xc017e84100?, 0xc017e64c80)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/rpc/context.go:181 +0x73
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !google.golang.org/grpc.getChainUnaryHandler.func1.getChainUnaryHandler.1({0x7acbf18, 0xc017e66c90}, {0x6ba7ac0, 0xc017e6d2c0})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:1177 +0xa8
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc00244e000, {{0x1}, {0x7b255a0, 0xc001989860}}, 0x0}, {0x7acbf18, 0xc017e66c90}, {0x6ba7ac0, 0xc017e6d2c0}, 0xc017e84100, ...)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/rpc/auth.go:97 +0x1f2
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !google.golang.org/grpc.getChainUnaryHandler.func1({0x7acbf18?, 0xc017e66c90?}, {0x6ba7ac0?, 0xc017e6d2c0?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:1177 +0x153
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor({0x7acbf18, 0xc017e66c90}, {0x6ba7ac0, 0xc017e6d2c0}, 0xc017e84100?, 0xc017e64c00)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/rpc/metrics.go:554 +0x123
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !google.golang.org/grpc.getChainUnaryHandler.func1.getChainUnaryHandler.1({0x7acbf18, 0xc017e66c90}, {0x6ba7ac0, 0xc017e6d2c0})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:1177 +0xa8
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4({0x7acbf18, 0xc017e66c90}, {0x6ba7ac0, 0xc017e6d2c0}, 0xc017e84100, 0xc017e64bc0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/rpc/metrics.go:472 +0x2c8
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !google.golang.org/grpc.getChainUnaryHandler.func1({0x7acbf18?, 0xc017e66c90?}, {0x6ba7ac0?, 0xc017e6d2c0?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:1177 +0x153
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x7acbf18?, 0xc017e66c90?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/rpc/context.go:138 +0x36
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0022fe630, {0x7acbf18, 0xc017e66c90}, {0x6cf1f60, 0x21}, 0xc00d473978)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/stop/stopper.go:351 +0x137
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x7acbf18?, 0xc017e66c90?}, {0x6ba7ac0?, 0xc017e6d2c0?}, 0xc017e84100?, 0xc017e05578?)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/rpc/context.go:136 +0x7d
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x7acbf18, 0xc017e66c90}, {0x6ba7ac0, 0xc017e6d2c0}, 0xc017e84100, 0x70?)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:1168 +0x7c
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x6b9d4a0, 0xc0022c7508}, {0x7acbf18, 0xc017e66c90}, 0xc017e6aa10, 0xc002556fa0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:11753 +0x143
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !google.golang.org/grpc.(*Server).processUnaryRPC(0xc0025f03c0, {0x7b26080, 0xc002d93520}, 0xc017e82a20, 0xc00262f9b0, 0xbca46c0, 0x0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:1358 +0xdd3
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !google.golang.org/grpc.(*Server).handleStream(0xc0025f03c0, {0x7b26080, 0xc002d93520}, 0xc017e82a20, 0x0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:1735 +0x9c8
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !google.golang.org/grpc.(*Server).serveStreams.func1.1()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:970 +0xaf
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 4509
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/org_golang_google_grpc/server.go:981 +0x133
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 1 [chan receive]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !testing.(*T).Run(0xc0022c4540, {0x6d151a8?, 0xc00231f980?}, 0x7035500)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/testing/testing.go:2005 +0x485
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !testing.runTests.func1(0xc0022c4540)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/testing/testing.go:2477 +0x37
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !testing.tRunner(0xc0022c4540, 0xc00231fac0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/testing/testing.go:1934 +0xea
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !testing.runTests(0xc00165ef48, {0xc0021e8540, 0x6, 0x8}, {0xc0892e0?, 0x0?, 0xc0892e0?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/testing/testing.go:2475 +0x4b4
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !testing.(*M).Run(0xc001ab0460)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/testing/testing.go:2337 +0x63a
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/backup_test.TestMain(0xc001ab0460)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/backup/main_test.go:28 +0x107
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !main.main()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! bazel-out/k8-fastbuild/bin/pkg/backup/backup_test_/testmain.go:798 +0x487
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 5 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !go.opencensus.io/stats/view.(*worker).start(0xc000f4d200)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/io_opencensus_go/stats/view/worker.go:292 +0x9f
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by go.opencensus.io/stats/view.init.0 in goroutine 1
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/io_opencensus_go/stats/view/worker.go:34 +0x8d
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 9 [chan receive]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/log_flush.go:123 +0x4e
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/log_flush.go:78 +0x1a
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 10 [chan receive]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/log_flush.go:147 +0x85
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/log_flush.go:79 +0x26
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 23 [syscall]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !os/signal.signal_recv()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/runtime/sigqueue.go:152 +0x29
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !os/signal.loop()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/os/signal/signal_unix.go:23 +0x13
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by os/signal.Notify.func1.1 in goroutine 10
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/os/signal/signal.go:152 +0x1f
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 11 [chan receive]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/goschedstats/runnable_enabled.go:144 +0xdd
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/goschedstats/runnable_enabled.go:136 +0x1a
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 24 [IO wait]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !internal/poll.runtime_pollWait(0x7fd31298e400, 0x72)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/runtime/netpoll.go:351 +0x85
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !internal/poll.(*pollDesc).wait(0xc0117aa200?, 0xc008d5c800?, 0x0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x27
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !internal/poll.(*pollDesc).waitRead(...)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/internal/poll/fd_poll_runtime.go:89
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !internal/poll.(*FD).Read(0xc0117aa200, {0xc008d5c800, 0xc00, 0xc00})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/internal/poll/fd_unix.go:165 +0x279
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !net.(*netFD).Read(0xc0117aa200, {0xc008d5c800?, 0x45f016?, 0xc2eb300?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/net/fd_posix.go:68 +0x25
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !net.(*conn).Read(0xc013d40368, {0xc008d5c800?, 0x7fd3144f81a8?, 0x7fd3607c0108?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/net/net.go:196 +0x45
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !crypto/tls.(*atLeastReader).Read(0xc0086b8000, {0xc008d5c800?, 0xc0000da008?, 0x4964a9?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/crypto/tls/conn.go:816 +0x3b
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !bytes.(*Buffer).ReadFrom(0xc00a1510a8, {0x7a7c4c0, 0xc0086b8000})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/bytes/buffer.go:217 +0x98
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !crypto/tls.(*Conn).readFromUntil(0xc00a150e08, {0x7a7afc0, 0xc013d40368}, 0x47f8f4?)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/crypto/tls/conn.go:838 +0xde
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !crypto/tls.(*Conn).readRecordOrCCS(0xc00a150e08, 0x0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/crypto/tls/conn.go:627 +0x3db
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !crypto/tls.(*Conn).readRecord(...)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/crypto/tls/conn.go:589
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !crypto/tls.(*Conn).Read(0xc00a150e08, {0xc0114a2000, 0x1000, 0xc015d43318?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/crypto/tls/conn.go:1392 +0x145
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !bufio.(*Reader).Read(0xc011630f00, {0xc002684628, 0x5, 0xc001c9a008?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/bufio/bufio.go:245 +0x197
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !io.ReadAtLeast({0x7a6dcc0, 0xc011630f00}, {0xc002684628, 0x5, 0x200}, 0x5)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/io/io.go:335 +0x8e
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !io.ReadFull(...)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/io/io.go:354
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/lib/pq.(*conn).recvMessage(0xc002684608, 0xc006229490)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_lib_pq/conn.go:1018 +0xc5
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/lib/pq.(*conn).recv1Buf(0xc002684608, 0xc006229490)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_lib_pq/conn.go:1073 +0x28
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/lib/pq.(*conn).recv1(...)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_lib_pq/conn.go:1100
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/lib/pq.(*conn).readBindResponse(0xc002684608)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_lib_pq/conn.go:1886 +0x3b
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/lib/pq.(*stmt).exec(0xc011161860, {0xc015d43768, 0x1, 0x1})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_lib_pq/conn.go:1462 +0x6c6
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/lib/pq.(*stmt).Exec(0xc011161860, {0xc015d43768, 0x1, 0x1})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_lib_pq/conn.go:1420 +0xd7
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/lib/pq.(*conn).Exec(0xc002684608, {0x6cc389e, 0x1c}, {0xc015d43768, 0x1, 0x1})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_lib_pq/conn.go:953 +0x1ac
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/lib/pq.(*conn).ExecContext(0xc002684608, {0x7acb720?, 0xc2e7440?}, {0x6cc389e, 0x1c}, {0xc00dcb7860?, 0x1, 0xc015d437c0?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_lib_pq/conn_go18.go:46 +0x1bb
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !database/sql.ctxDriverExec({0x7acb720?, 0xc2e7440?}, {0x7fd31298cda8?, 0xc002684608?}, {0x0?, 0x0?}, {0x6cc389e?, 0x0?}, {0xc00dcb7860, 0x1, ...})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/database/sql/ctxutil.go:31 +0xd7
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !database/sql.(*DB).execDC.func2()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/database/sql/sql.go:1713 +0x15c
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !database/sql.withLock({0x7a901e0, 0xc00bdd6280}, 0xc015d43aa8)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/database/sql/sql.go:3572 +0x71
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !database/sql.(*DB).execDC(0x1?, {0x7acb720, 0xc2e7440}, 0xc00bdd6280, 0x0?, {0x6cc389e, 0x1c}, {0xc00201c250, 0x1, 0x1})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/database/sql/sql.go:1708 +0x216
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !database/sql.(*DB).exec(0xc009ac6270, {0x7acb720, 0xc2e7440}, {0x6cc389e, 0x1c}, {0xc00201c250, 0x1, 0x1}, 0x0?)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/database/sql/sql.go:1693 +0xcf
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !database/sql.(*DB).ExecContext.func1(0x30?)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/database/sql/sql.go:1672 +0x4f
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !database/sql.(*DB).retry(0x5ce245?, 0xc006229c98)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/database/sql/sql.go:1576 +0x42
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !database/sql.(*DB).ExecContext(0xbbbb730?, {0x7acb720?, 0xc2e7440?}, {0x6cc389e?, 0xc0022c4700?}, {0xc00201c250?, 0x5ce1cc?, 0x20?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/database/sql/sql.go:1671 +0xc8
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).ExecWithMessage(0xc012058920, {0x7a6e000, 0xc0022c4700}, {0x0, 0x0}, {0x6cc389e, 0x1c}, {0xc00201c250, 0x1, 0x1})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/testutils/sqlutils/sql_runner.go:93 +0x123
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Exec(0xc012058920, {0x7a6e000, 0xc0022c4700}, {0x6cc389e, 0x1c}, {0xc00201c250, 0x1, 0x1})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/testutils/sqlutils/sql_runner.go:84 +0x11e
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/backup.TestBackupRestoreWithConcurrentWrites(0xc0022c4700)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/backup/backup_test.go:3371 +0x377
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !testing.tRunner(0xc0022c4700, 0x7035500)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/testing/testing.go:1934 +0xea
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by testing.(*T).Run in goroutine 1
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/testing/testing.go:1997 +0x465
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 25 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc001e7d340, {0x7acbee0, 0xc001989220})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/file_log_gc.go:20 +0x79
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/flags.go:365 +0x1355
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 26 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc001e7d420, {0x7acbee0, 0xc001989220})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/file_log_gc.go:20 +0x79
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/flags.go:365 +0x1355
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 27 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc001e7d500, {0x7acbee0, 0xc001989220})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/file_log_gc.go:20 +0x79
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/flags.go:365 +0x1355
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 28 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc001e7d5e0, {0x7acbee0, 0xc001989220})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/file_log_gc.go:20 +0x79
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/log/flags.go:365 +0x1355
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 911 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/storeliveness.(*Transport).smearingSenderLoop(0xc00372f080, {0x7ace310, 0xc003d95a40})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/storeliveness/transport.go:239 +0x152
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/storeliveness.(*Transport).runSmearingSenderGoroutine.func1({0x7ace310, 0xc003d95a40})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/storeliveness/transport.go:223 +0x8a
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/kv/kvserver/storeliveness.(*Transport).runSmearingSenderGoroutine in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/kv/kvserver/storeliveness/transport.go:221 +0x125
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 30 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/vfs/disk_health.go:682 +0xf8
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/vfs/disk_health.go:671 +0x6e
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 912 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/timeutil.(*BroadcastTicker).run(...)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/timeutil/ticker.go:58
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/timeutil.NewBroadcastTicker in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/timeutil/ticker.go:52 +0xae
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 32 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/vfs/disk_health.go:254 +0xb3
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/vfs/disk_health.go:249 +0x58
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 913 [select]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/gossip.(*infoStore).launchCallbackWorker.func1({0xc0035a77c0?, 0x7ace310?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/gossip/infostore.go:293 +0x105
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x7ace310, 0xc003d95a40})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/stop/stopper.go:440 +0x8a
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! pkg/util/stop/stopper.go:438 +0x125
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 34 [chan receive, "pebble":"gc"]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/pebble/internal/deletepacer.(*DeletePacer).mainLoop(0xc00100e508)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/internal/deletepacer/delete_pacer.go:183 +0x33b
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/pebble/internal/deletepacer.Open.func1.1({0x7acbf18?, 0xc001a58f00?})
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/internal/deletepacer/delete_pacer.go:108 +0x17
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !runtime/pprof.Do({0x7acb720?, 0xc2e7440?}, {{0xc0019eba60?, 0x14800000007?, 0x0?}}, 0xc001092fa0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! GOROOT/src/runtime/pprof/runtime.go:51 +0x8c
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/pebble/internal/deletepacer.Open.func1()
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/internal/deletepacer/delete_pacer.go:107 +0x9c
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/pebble/internal/deletepacer.Open in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/internal/deletepacer/delete_pacer.go:106 +0x229
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 35 [chan receive]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).releaseLoop(0x0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:61 +0x6f
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).Init in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:54 +0x1a7
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 36 [chan receive]:
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).releaseLoop(0x0)
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:61 +0x6f
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !created by github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).Init in goroutine 24
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 ! external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:54 +0x1a7
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !
F251213 00:59:29.074577 46569 15@kv/kvserver/concurrency/concurrency_manager.go:603 [T1,Vsystem,n1,s1,r108/1:/Tenant/10/Table/106{-/1…}] 1 !goroutine 914 [select]:
F251213 00:59:29.074577 46569
Jira issue: CRDB-57845