Forwarder eventually stops sending logs

[chuseman]

We've tested the new forwarder functionality to replace the old seqfwd app. After running correctly for a few days, we've observed several instances of this message in logs across several boxes (all running as a service on Windows):

{
  "@t":"2026-03-05T20:45:30.6719054Z",
  "@m":"Forwarding log shipper failed and exited",
  "@i":"1efb7888",
  "@l":"Fatal",
  "@x":"System.UnauthorizedAccessException: Access to the path 'C:\\ProgramData\\Seqcli\\Forwarder\\SeqCli\\Buffer\\fd5a6b94b1bf49df954745f8262a5c2d\\0000000000000001.clef' is denied.\r\n   at System.IO.FileSystem.DeleteFile(String fullPath)\r\n   at SeqCli.Forwarder.Filesystem.System.SystemStoreDirectory.TryDelete(String name) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Filesystem\\System\\SystemStoreDirectory.cs:line 96\r\n   at SeqCli.Forwarder.Storage.BufferReader.AdvanceTo(BufferPosition newReaderHead) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Storage\\BufferReader.cs:line 248\r\n   at SeqCli.Forwarder.Channel.ForwardingChannel.<>c__DisplayClass5_0.<<-ctor>b__1>d.MoveNext() in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Channel\\ForwardingChannel.cs:line 115",
  "@tr":"39d89a8ca8d77ce084232c6be16c588b",
  "@sp":"8e85c1e1a8c13011",
  "SourceContext":"SeqCli.Forwarder.Channel.ForwardingChannel",
  "RequestId":"0HNJQ5TSLESBP:00000001",
  "RequestPath":"/api/events/raw",
  "ConnectionId":"0HNJQ5TSLESBP",
  "MachineName":"xxx",
  "Application":"SeqCli Forwarder"
}

Our configuration is pretty simple - the only interesting bit is that we've set useApiKeyForwarding to true in the json:

{
  "connection": {
    "serverUrl": "https://xxx",
    "apiKey": null,
    "pooledConnectionLifetimeMilliseconds": null,
    "eventSizeLimitBytes": 262144,
    "batchSizeLimitBytes": 10485760
  },
  "output": {
    "disableColor": false,
    "forceColor": false
  },
  "forwarder": {
    "storage": {
      "targetChunkSizeBytes": 52428800,
      "maxChunks": null
    },
    "diagnostics": {
      "internalLoggingLevel": "Information",
      "internalLogServerUri": null,
      "internalLogServerApiKey": null,
      "exposeIngestionLog": false,
      "ingestionLogShowDetail": false
    },
    "api": {
      "listenUri": "http://127.0.0.1:15341"
    },
    "useApiKeyForwarding": true
  },
  "encryption": {
    "encryptor": null,
    "encryptorArgs": null,
    "decryptor": null,
    "decryptorArgs": null
  },
  "profiles": {}
}

Looks like this may happen once one of the clef buffers hits the targetChunkSizeBytes limit? Restarting the service will run through one chunk and then fail to delete it. Maybe a leaked file handle or something?

[liammclennan]

Hi @chuseman

Have you tried excluding 'C:\ProgramData\Seqcli' from your antivirus scanning?

[chuseman]

Have you tried excluding 'C:\ProgramData\Seqcli' from your antivirus scanning?

No, we don't have any exclusions setup. We run Crowdstrike and that's corporate controlled so it would take some doing to get an exclusion added.

Is the behavior here different from the old seqfwd service? That ran fine on the same servers -- we've had to roll back to it for now.

[nblumhardt]

The old Seq Forwarder (essentially) used a single file that it would lock at start-up, so there would be minimal opportunities for AV scanners etc. to jump in and cause locking problems. The new version uses a set of rolling log files, so would be more susceptible to this kind of outside interference.

There's also a chance it's a lurking bug we haven't yet spotted. You might be able to get some more info on which process is using the lock via resmon or similar.

[chuseman]

I've ran procmon to try to see what's going on. Here's how I tested:

1. Start procmon capture, filtered to fd5a6b94b1bf49df954745f8262a5c2d directory.
2. Note the fd5a6b94b1bf49df954745f8262a5c2d directory only contains the following files:
   1. 0000000000000000.bookmark
   2. 0000000000000022.clef,
   3. api.key
3. Start the seqcli service, wait for logs to report that it's running; ie:

   {"@t":"2026-03-12T20:48:49.7310130Z","@m":"Opening local buffer in \"C:\\ProgramData\\Seqcli\\Forwarder\\SeqCli\\Buffer\\fd5a6b94b1bf49df954745f8262a5c2d\"","@i":"5fca42e5","StorePath":"C:\\ProgramData\\Seqcli\\Forwarder\\SeqCli\\Buffer\\fd5a6b94b1bf49df954745f8262a5c2d","SourceContext":"SeqCli.Forwarder.Channel.ForwardingAuthenticationStrategy","MachineName":"xxx","Application":"SeqCli Forwarder"}

4. Manually write a log message with seqcli
5. Observe failure in seq-forwarder log:

   {"@t":"2026-03-12T20:53:17.0857294Z","@m":"Forwarding log shipper failed and exited","@i":"1efb7888","@l":"Fatal","@x":"System.UnauthorizedAccessException: Access to the path 'C:\\ProgramData\\Seqcli\\Forwarder\\SeqCli\\Buffer\\fd5a6b94b1bf49df954745f8262a5c2d\\0000000000000022.clef' is denied.\r\n   at System.IO.FileSystem.DeleteFile(String fullPath)\r\n   at SeqCli.Forwarder.Filesystem.System.SystemStoreDirectory.TryDelete(String name) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Filesystem\\System\\SystemStoreDirectory.cs:line 96\r\n   at SeqCli.Forwarder.Storage.BufferReader.AdvanceTo(BufferPosition newReaderHead) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Storage\\BufferReader.cs:line 248\r\n   at SeqCli.Forwarder.Channel.ForwardingChannel.<>c__DisplayClass5_0.<<-ctor>b__1>d.MoveNext() in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Channel\\ForwardingChannel.cs:line 115","SourceContext":"SeqCli.Forwarder.Channel.ForwardingChannel","MachineName":"xxx","Application":"SeqCli Forwarder"}

6. Stop procmon capture.

Here's what the procmon capture says about 0000000000000022.clef around the time of the delete (which is after 0000000000000023.clef gets created and 0000000000000000.bookmark is updated):

Time of Day	Process Name	PID	Operation	Path	Result	Detail
3:53:17.0846691 PM	seqcli.exe	24148	CreateFile	...\fd5a6b94b1bf49df954745f8262a5c2d\0000000000000022.clef	SUCCESS	Desired Access: Read Attributes, Delete, Disposition: Open, Options: Non-Directory File, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
3:53:17.0848086 PM	seqcli.exe	24148	QueryAttributeTagFile	...\fd5a6b94b1bf49df954745f8262a5c2d\0000000000000022.clef	SUCCESS	Attributes: ANCI, ReparseTag: 0x0
3:53:17.0848195 PM	seqcli.exe	24148	SetDispositionInformationFile	...\fd5a6b94b1bf49df954745f8262a5c2d\0000000000000022.clef	CANNOT DELETE	Delete: True
3:53:17.0848503 PM	seqcli.exe	24148	CloseFile	...\fd5a6b94b1bf49df954745f8262a5c2d\0000000000000022.clef	SUCCESS

procmon doesn't see any other process interested in 0000000000000022.clef other than seqcli.exe.

Do you think this rules out Crowdstrike/antivirus scanning? Or do I still need to try to push for an exclusion of this directory to test it?

[nblumhardt]

Thanks for the update, Chris, that's good info to have. We'll review it and let you know later today.

Best regarsd,
Nick

[liammclennan]

Hi @chuseman,

We are still looking into this. At this stage it looks most likely to be a bug in seqcli.

[liammclennan]

Hi @chuseman

We have not yet been able to reproduce the issue you've described. I think we will need some more detail of your setup. Can you please send a diagnostic report to support@datalust.co and we will continue the conversation there?

[liammclennan]

@chuseman skip the diagnostic report. That won't help as you are probably running seqcli on a different computer. Send a message to support@datalust.co and we can pick up the conversation there.

[nblumhardt]

One quick follow-up note; although we haven't been able to trigger the issue with them, we have found a couple of suspicious bits of code that should lead to the issue you're reporting. (Why they don't might be down to Windows versions, etc.).

I'll send a patch for dev that addresses them sometime in the next week; without us being able to repro it's not guaranteed it'll sort the issue out, but should be worth trying.

Thanks again for the report and all the investigation @chuseman!

[KodrAus]

Hi @chuseman 👋

I’m currently looking at this over in #451. Would you be happy to pull a preview build in your environment when it’s ready to confirm it’s all fixed?

[chuseman]

Sure @KodrAus I can test it out - just let me know where to grab the bits!

[nblumhardt]

Thanks @chuseman!

2025.2.02489-dev-02489 is now on the releases page. Keen to hear how you go.

[chuseman]

2025.2.02489-dev-02489 is now on the releases page. Keen to hear how you go.

Good news! This appears to fully fix the issue we were running into.

Thanks for the fix!