Use non-recursive watcher in PhysicalFilesWatcher for missing parent directories

[svick]

Fixes #116713

Problem

When PhysicalFileProvider is constructed with a root directory that does not yet exist (e.g., a configuration file path whose parent directory hasn't been created), Watch() fails because FileSystemWatcher cannot watch a non-existent directory. This commonly occurs with AddJsonFile when the config file's parent directory is missing at startup.

Solution

PhysicalFilesWatcher now gracefully handles a missing root directory by deferring FileSystemWatcher activation until the root appears. A PendingCreationWatcher monitors the nearest existing ancestor directory using a non-recursive FileSystemWatcher and cascades through intermediate directory levels as they are created. Once the root directory exists, the main recursive FileSystemWatcher is enabled and any already-existing watched entries are reported.

Callers always receive normal FSW-backed change tokens — no re-registration is needed when the root directory appears later.

Changes

File	Description
PhysicalFilesWatcher.cs	Added PendingCreationWatcher inner class that watches for a non-existent directory to be created. TryEnableFileSystemWatcher defers to EnsureRootCreationWatcher when _root doesn't exist, with a callback to retry once it appears. ReportExistingWatchedEntries fires tokens for entries created before the FSW was active. The constructor normalizes _root to always have a trailing separator. OnFileSystemEntryChange now uses DirectoryInfo for directory paths so exclusion filters work correctly.
PhysicalFileProvider.cs	Constructor no longer throws DirectoryNotFoundException for a missing root — the PhysicalFilesWatcher handles it. Updated Watch doc comments to include directories. Removed duplicate _pathSeparators field in favor of PathUtils.PathSeparators.
FileConfigurationSource.cs	ResolveFileProvider creates the PhysicalFileProvider with the file's immediate parent directory (even if missing), relying on the watcher to handle the non-existent case.
PollingFileChangeToken.cs	GetLastWriteTimeUtc now falls back to checking DirectoryInfo when FileInfo.Exists is false, so polling correctly detects directory changes.
PathUtils.cs	PathSeparators made internal for reuse across files.
PhysicalFilesWatcherTests.cs	Tests for missing root (file path and wildcard), root deleted and recreated, subdirectory create/delete/recreate cycles, directory watch tokens, hidden directory exclusion, and active polling variants.
FileConfigurationProviderTest.cs	Integration test verifying the watch token fires when a file is created inside a previously-missing directory.
PhysicalFileProviderTests.cs	Fixed TokenFiredForGlobbingPatternsPointingToSubDirectory path length issue on .NET Framework. Fixed TokensWithForwardAndBackwardSlashesAreSame and TokensFiredForNewDirectoryContentsOnRename.

[Copilot]

Pull request overview

This PR addresses excessive recursive OS file watches (e.g., inotify) triggered when watching a file whose parent directory doesn’t exist yet, by introducing a non-recursive “cascading” watcher approach and adjusting configuration/file-provider resolution to work with missing directories.

Changes:

• Add a PendingCreationWatcher path-component–by–path-component watcher for missing-parent scenarios, avoiding recursive subtree watches.
• Allow PhysicalFileProvider construction when the root directory doesn’t exist and adjust FileSystemWatcher initialization accordingly.
• Update FileConfigurationSource.ResolveFileProvider behavior and add tests covering missing-directory watch behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PhysicalFilesWatcher.cs	Adds pending-creation watcher logic and changes when the main FileSystemWatcher is enabled.
src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PhysicalFileProvider.cs	Removes root existence requirement; creates FileSystemWatcher without setting Path up-front.
src/libraries/Microsoft.Extensions.Configuration.FileExtensions/src/FileConfigurationSource.cs	Changes resolution to create a provider for the file’s immediate directory and reduce the config path to the filename.
src/libraries/Microsoft.Extensions.FileProviders.Physical/src/ArraySegmentExtensions.cs	Adds ArraySegment<T>.At and a Slice polyfill for non-NET TFMs.
src/libraries/Microsoft.Extensions.FileProviders.Physical/tests/PhysicalFilesWatcherTests.cs	Adds unit tests for pending-creation watcher behavior and main watcher enablement.
src/libraries/Microsoft.Extensions.Configuration.FileExtensions/tests/FileConfigurationProviderTest.cs	Adds an integration-style test for config reload token firing when missing directory/file is created.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

[Copilot]

After the root directory is successfully created and the main FileSystemWatcher is enabled, _rootCreationWatcher is left referenced (with a cancelled token) until PhysicalFilesWatcher.Dispose() or another missing-root cycle. Consider disposing and clearing _rootCreationWatcher once _fileWatcher.EnableRaisingEvents = true succeeds to avoid keeping the CancellationTokenSource/registration alive for the lifetime of the provider.

[Copilot]

CreateWatcher constructs new FileSystemWatcher(rootPath) when useActivePolling == false, which requires the root to exist and doesn’t exercise the new production path where PhysicalFileProvider creates new FileSystemWatcher() with an empty Path (to support missing roots). To better cover the missing-root scenario and the _fileWatcher.Path initialization in TryEnableFileSystemWatcher, consider constructing new FileSystemWatcher() here as well (and letting PhysicalFilesWatcher set the path when enabling).

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

[github-actions]

Code Review — PR #126026

❌ 1. PendingCreationWatcher._cts is never disposed on the error/early-cancel path when _root later appears

In EnsureRootCreationWatcher, when newWatcher.Token.IsCancellationRequested is true (ancestor directory doesn't exist → constructor called _cts.Cancel()), the code checks Directory.Exists(_root) and if true calls TryEnableFileSystemWatcher() recursively. But that re-entry creates a new PendingCreationWatcher (because the old one has Token.IsCancellationRequested), disposing the old one via _rootCreationWatcher?.Dispose() inside the lock. This is fine for the object—but when the ancestor truly doesn't exist, the PendingCreationWatcher just sits there with a cancelled CTS and no watcher. If the ancestor is later created externally, nothing will ever retry—the system silently gives up. The comment says "If _root appeared during setup, enable the FSW now" but no mechanism re-triggers if _root appears later.

This means: if you watch a path like /mnt/share/app/config.json and /mnt/share doesn't exist when the watcher starts, the watcher is permanently dead even if /mnt/share is mounted later. The old code didn't support missing roots at all, so this is arguably a "best effort" improvement, but worth documenting that multi-level missing ancestors above the nearest mount point are not supported.

Suggestion: At minimum, add a doc comment on EnsureRootCreationWatcher noting this limitation.

---

⚠️ 2. Race: Dispose and EnsureRootCreationWatcher — PendingCreationWatcher leaked between lock release and Token.Register

Sequence:

1. EnsureRootCreationWatcher creates newWatcher, stores it in _rootCreationWatcher under lock, releases lock.
2. Dispose acquires _rootCreationWatcherLock, calls _rootCreationWatcher.Dispose(), sets it to null.
3. EnsureRootCreationWatcher continues to newWatcher.Token.Register(...) — newWatcher is now disposed.

The ObjectDisposedException catch handles this on .NET Framework. On modern .NET, CancellationToken.Register on a disposed CTS doesn't throw—it returns a default registration and the callback is never invoked. So the watcher is disposed and the callback silently doesn't fire. This is correct behavior (the outer watcher is being disposed), but the CancellationTokenRegistration returned by Register is never stored or disposed. On modern .NET this is harmless (it's a no-op registration), but the pattern is fragile.

---

⚠️ 3. _rootWasUnavailable is not volatile and accessed outside locks

_rootWasUnavailable is written inside _fileWatcherLock in TryEnableFileSystemWatcher, but there is no memory barrier guaranteeing visibility to other threads. While all current accesses happen inside the same lock, the field is a plain bool without volatile. If a future change reads it outside the lock, it will be a subtle bug. Consider making it volatile or documenting the lock contract.

---

⚠️ 4. ReportExistingWatchedEntries can do an unbounded recursive directory scan

Directory.EnumerateFileSystemEntries(_root, "*", SearchOption.AllDirectories) walks the entire directory tree under _root. For large directory hierarchies (e.g., a root pointing at /home/user or a large project), this could cause significant latency on the thread that calls TryEnableFileSystemWatcher (which is called from GetOrAddFilePathChangeToken / GetOrAddWildcardChangeToken, i.e., from the user's Watch() call path).

The early-exit when both lookups are empty mitigates this when there's a single watch, but with wildcard patterns the scan runs to completion over the entire tree. This is called on a threadpool thread via the PendingCreationWatcher callback chain, but it still blocks that callback.

Suggestion: Consider bounding the scan or running it asynchronously.

---

⚠️ 5. TryEnableFileSystemWatcher catch block silently drops the exception when _root still exists

catch (Exception ex) when (ex is ArgumentException or IOException)
{
    if (!Directory.Exists(_root))
    {
        needsRootWatcher = true;
        _rootWasUnavailable = true;
    }
}

If _fileWatcher.Path = _root or EnableRaisingEvents = true throws an ArgumentException or IOException but _root does still exist (e.g., path too long, permissions issue), the exception is swallowed and the FSW is never enabled. No PendingCreationWatcher is created either. The watch silently becomes a no-op. Consider logging or propagating in that case.

---

⚠️ 6. Breaking behavioral change: removing DirectoryNotFoundException from PhysicalFileProvider constructor

The old constructor threw DirectoryNotFoundException when root didn't exist. User code relying on this for fast-fail validation (e.g., try { new PhysicalFileProvider(path); } catch (DirectoryNotFoundException) { ... }) will silently stop catching it. The FileConfigurationSource.ResolveFileProvider change that removes the walk-up-to-existing-ancestor logic is the companion fix, but any direct callers of new PhysicalFileProvider(nonExistentPath) will experience a behavior change.

This is intentional for the feature, and the xmldoc doesn't document the throw, so it's arguably undocumented behavior—but it's worth calling out as a known breaking change in the PR description or release notes.

---

💡 7. PendingCreationWatcher.IsTargetDirectory uses OrdinalIgnoreCase unconditionally

return target.Equals(path.AsSpan().TrimEnd(PathUtils.PathSeparators), StringComparison.OrdinalIgnoreCase);

On Linux, file system paths are case-sensitive. Using OrdinalIgnoreCase means a directory created as /root/MyDir will match a target of /root/mydir on Linux, which is incorrect. The main _filePathTokenLookup also uses OrdinalIgnoreCase, so this is consistent with the existing codebase behavior—but it means the PendingCreationWatcher inherits the same long-standing bug on case-sensitive file systems.

---

💡 8. PendingCreationWatcher.SetupWatcherNoLock — potential for unbounded loop

SetupWatcherNoLock has a while(true) that loops when the post-start race check detects the directory appeared. If a directory keeps being created and deleted rapidly (adversarial or flaky FS), this could spin. In practice the Directory.Exists check and FSW setup are heavyweight enough that this is unlikely to be a real problem, but an iteration cap would be defensive.

---

💡 9. PollingFileChangeToken: lazy _directoryInfo allocation uses a racy pattern

_directoryInfo ??= new DirectoryInfo(_fileInfo.FullName);

_directoryInfo is not volatile. Multiple threads calling HasChanged concurrently could each allocate a DirectoryInfo. This is benign (they'd all point to the same path and DirectoryInfo is cheap), but worth noting the intentional benign race.

---

Summary

#	Severity	Issue
1	❌	PendingCreationWatcher permanently gives up when no ancestor exists — no retry mechanism
2	⚠️	Dispose/EnsureRootCreationWatcher race — functionally safe but fragile pattern
3	⚠️	_rootWasUnavailable lacks volatile/documented lock contract
4	⚠️	ReportExistingWatchedEntries unbounded recursive scan on large trees
5	⚠️	Exception silently swallowed when _root exists but FSW setup fails
6	⚠️	Removing DirectoryNotFoundException from constructor is a breaking change
7	💡	OrdinalIgnoreCase path comparison incorrect on case-sensitive Linux FS (pre-existing)
8	💡	SetupWatcherNoLock unbounded loop under adversarial conditions
9	💡	Benign race on _directoryInfo ??= in PollingFileChangeToken

Generated by Code Review for issue #126026 · ◷