Implement Virtual FileSystem abstraction over ManagedCode.Storage

.github/copilot-instructions.md

@@ -0,0 +1,115 @@
+# Copilot Instructions for ManagedCode.Storage
+
+## Overview
+
+ManagedCode.Storage is a universal storage abstraction library that provides a consistent interface for working with multiple cloud blob storage providers including Azure Blob Storage, AWS S3, Google Cloud Storage, and local file system. The library aims to simplify development by providing a single API for all storage operations.
+
+## Project Structure
+
+- **ManagedCode.Storage.Core**: Core abstractions and interfaces (IStorage, BaseStorage, etc.)
+- **Storages/**: Provider-specific implementations
+  - `ManagedCode.Storage.Azure`: Azure Blob Storage implementation
+  - `ManagedCode.Storage.Aws`: AWS S3 implementation  
+  - `ManagedCode.Storage.Google`: Google Cloud Storage implementation
+  - `ManagedCode.Storage.FileSystem`: Local file system implementation
+  - `ManagedCode.Storage.Ftp`: FTP storage implementation
+  - `ManagedCode.Storage.Azure.DataLake`: Azure Data Lake implementation
+- **Tests/**: Unit and integration tests
+- **Integrations/**: Additional integrations (SignalR, Client/Server components)
+
+## Technical Context
+
+- **Target Framework**: .NET 9.0
+- **Language Version**: C# 13
+- **Architecture**: Provider pattern with unified interfaces
+- **Key Features**: Async/await support, streaming operations, metadata handling, progress reporting
+
+## Development Guidelines
+
+### Code Style & Standards
+- Use nullable reference types (enabled in project)
+- Follow async/await patterns consistently
+- Use ValueTask for performance-critical operations where appropriate
+- Implement proper cancellation token support in all async methods
+- Use ConfigureAwait(false) for library code
+- Follow dependency injection patterns
+
+### Key Interfaces & Patterns
+- `IStorage`: Main storage interface for blob operations
+- `IStorageOptions`: Configuration options for storage providers
+- `BaseStorage`: Base implementation with common functionality
+- All operations should support progress reporting via `IProgress<T>`
+- Use `BlobMetadata` for storing blob metadata
+- Support for streaming operations with `IStreamer`
+
+### Performance Considerations
+- Implement efficient streaming for large files
+- Use memory-efficient approaches for data transfer
+- Cache metadata when appropriate
+- Support parallel operations where beneficial
+- Minimize allocations in hot paths
+
+### Testing Approach
+- Unit tests for core logic
+- Integration tests for provider implementations
+- Use test fakes/mocks for external dependencies
+- Test error scenarios and edge cases
+- Validate async operation behavior
+
+### Provider Implementation Guidelines
+When implementing new storage providers:
+1. Inherit from `BaseStorage` class
+2. Implement all required interface methods
+3. Handle provider-specific errors appropriately
+4. Support all metadata operations
+5. Implement efficient streaming operations
+6. Add comprehensive tests
+7. Document provider-specific limitations or features
+
+### Error Handling
+- Use appropriate exception types for different error scenarios
+- Provide meaningful error messages
+- Handle provider-specific errors and translate to common exceptions
+- Support retry mechanisms where appropriate
+
+### Documentation
+- Document public APIs with XML comments
+- Include usage examples for complex operations
+- Document provider-specific behavior differences
+- Keep README.md updated with supported features
+
+## Common Tasks
+
+### Adding a New Storage Provider
+1. Create new project in `Storages/` folder
+2. Inherit from `BaseStorage`
+3. Implement provider-specific operations
+4. Add configuration options
+5. Create comprehensive tests
+6. Update solution file and documentation
+
+### Implementing New Features
+1. Define interface changes in Core project
+2. Update BaseStorage if needed
+3. Implement in all relevant providers
+4. Add tests for new functionality
+5. Update documentation
+
+### Performance Optimization
+- Profile critical paths
+- Optimize memory allocations
+- Improve streaming performance
+- Cache frequently accessed data
+- Use efficient data structures
+
+## Dependencies & Libraries
+- Provider-specific SDKs (Azure.Storage.Blobs, AWS SDK, Google Cloud Storage)
+- Microsoft.Extensions.* for dependency injection and configuration
+- System.Text.Json for serialization
+- Benchmarking tools for performance testing
+
+## Building & Testing
+- Use `dotnet build` to build the solution
+- Run `dotnet test` for unit tests
+- Integration tests may require cloud provider credentials
+- Use `dotnet pack` to create NuGet packages

ManagedCode.Storage.VirtualFileSystem/Core/IVfsEntry.cs

@@ -0,0 +1,137 @@
+using System;
+using System.Collections.Generic;
+using System.Threading;
+using System.Threading.Tasks;
+using ManagedCode.Storage.VirtualFileSystem.Core;
+
+namespace ManagedCode.Storage.VirtualFileSystem.Core;
+
+/// <summary>
+/// Base interface for virtual file system entries
+/// </summary>
+public interface IVfsEntry
+{
+    /// <summary>
+    /// Gets the path of this entry
+    /// </summary>
+    VfsPath Path { get; }
+
+    /// <summary>
+    /// Gets the name of this entry
+    /// </summary>
+    string Name { get; }
+
+    /// <summary>
+    /// Gets the type of this entry
+    /// </summary>
+    VfsEntryType Type { get; }
+
+    /// <summary>
+    /// Gets when this entry was created
+    /// </summary>
+    DateTimeOffset CreatedOn { get; }
+
+    /// <summary>
+    /// Gets when this entry was last modified
+    /// </summary>
+    DateTimeOffset LastModified { get; }
+
+    /// <summary>
+    /// Checks if this entry exists
+    /// </summary>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>True if the entry exists</returns>
+    ValueTask<bool> ExistsAsync(CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Refreshes the entry information from storage
+    /// </summary>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>Task representing the async operation</returns>
+    Task RefreshAsync(CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Gets the parent directory of this entry
+    /// </summary>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>The parent directory</returns>
+    ValueTask<IVirtualDirectory> GetParentAsync(CancellationToken cancellationToken = default);
+}
+
+/// <summary>
+/// Type of virtual file system entry
+/// </summary>
+public enum VfsEntryType
+{
+    /// <summary>
+    /// A file entry
+    /// </summary>
+    File,
+
+    /// <summary>
+    /// A directory entry
+    /// </summary>
+    Directory
+}
+
+/// <summary>
+/// Progress information for copy operations
+/// </summary>
+public class CopyProgress
+{
+    /// <summary>
+    /// Total number of bytes to copy
+    /// </summary>
+    public long TotalBytes { get; set; }
+
+    /// <summary>
+    /// Number of bytes copied so far
+    /// </summary>
+    public long CopiedBytes { get; set; }
+
+    /// <summary>
+    /// Total number of files to copy
+    /// </summary>
+    public int TotalFiles { get; set; }
+
+    /// <summary>
+    /// Number of files copied so far
+    /// </summary>
+    public int CopiedFiles { get; set; }
+
+    /// <summary>
+    /// Current file being copied
+    /// </summary>
+    public string? CurrentFile { get; set; }
+
+    /// <summary>
+    /// Percentage completed (0-100)
+    /// </summary>
+    public double PercentageComplete => TotalBytes > 0 ? (double)CopiedBytes / TotalBytes * 100 : 0;
+}
+
+/// <summary>
+/// Result of a delete directory operation
+/// </summary>
+public class DeleteDirectoryResult
+{
+    /// <summary>
+    /// Whether the operation was successful
+    /// </summary>
+    public bool Success { get; set; }
+
+    /// <summary>
+    /// Number of files deleted
+    /// </summary>
+    public int FilesDeleted { get; set; }
+
+    /// <summary>
+    /// Number of directories deleted
+    /// </summary>
+    public int DirectoriesDeleted { get; set; }
+
+    /// <summary>
+    /// List of errors encountered during deletion
+    /// </summary>
+    public List<string> Errors { get; set; } = new();
+}

ManagedCode.Storage.VirtualFileSystem/Core/IVirtualDirectory.cs

@@ -0,0 +1,138 @@
+using System;
+using System.Collections.Generic;
+using System.Threading;
+using System.Threading.Tasks;
+using ManagedCode.Storage.VirtualFileSystem.Options;
+
+namespace ManagedCode.Storage.VirtualFileSystem.Core;
+
+/// <summary>
+/// Represents a directory in the virtual filesystem
+/// </summary>
+public interface IVirtualDirectory : IVfsEntry
+{
+    /// <summary>
+    /// Lists files in this directory with pagination and pattern matching
+    /// </summary>
+    /// <param name="pattern">Search pattern for filtering</param>
+    /// <param name="recursive">Whether to search recursively</param>
+    /// <param name="pageSize">Page size for pagination</param>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>Async enumerable of files</returns>
+    IAsyncEnumerable<IVirtualFile> GetFilesAsync(
+        SearchPattern? pattern = null,
+        bool recursive = false,
+        int pageSize = 100,
+        CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Lists subdirectories with pagination
+    /// </summary>
+    /// <param name="pattern">Search pattern for filtering</param>
+    /// <param name="recursive">Whether to search recursively</param>
+    /// <param name="pageSize">Page size for pagination</param>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>Async enumerable of directories</returns>
+    IAsyncEnumerable<IVirtualDirectory> GetDirectoriesAsync(
+        SearchPattern? pattern = null,
+        bool recursive = false,
+        int pageSize = 100,
+        CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Lists all entries (files and directories) in this directory
+    /// </summary>
+    /// <param name="pattern">Search pattern for filtering</param>
+    /// <param name="recursive">Whether to search recursively</param>
+    /// <param name="pageSize">Page size for pagination</param>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>Async enumerable of entries</returns>
+    IAsyncEnumerable<IVfsEntry> GetEntriesAsync(
+        SearchPattern? pattern = null,
+        bool recursive = false,
+        int pageSize = 100,
+        CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Creates a file in this directory
+    /// </summary>
+    /// <param name="name">File name</param>
+    /// <param name="options">File creation options</param>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>The created file</returns>
+    ValueTask<IVirtualFile> CreateFileAsync(
+        string name,
+        CreateFileOptions? options = null,
+        CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Creates a subdirectory
+    /// </summary>
+    /// <param name="name">Directory name</param>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>The created directory</returns>
+    ValueTask<IVirtualDirectory> CreateDirectoryAsync(
+        string name,
+        CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Gets statistics for this directory
+    /// </summary>
+    /// <param name="recursive">Whether to calculate recursively</param>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>Directory statistics</returns>
+    Task<DirectoryStats> GetStatsAsync(
+        bool recursive = true,
+        CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Deletes this directory
+    /// </summary>
+    /// <param name="recursive">Whether to delete recursively</param>
+    /// <param name="cancellationToken">Cancellation token</param>
+    /// <returns>Delete operation result</returns>
+    Task<DeleteDirectoryResult> DeleteAsync(
+        bool recursive = false,
+        CancellationToken cancellationToken = default);
+}
+
+/// <summary>
+/// Statistics for a directory
+/// </summary>
+public class DirectoryStats
+{
+    /// <summary>
+    /// Number of files in the directory
+    /// </summary>
+    public int FileCount { get; init; }
+
+    /// <summary>
+    /// Number of subdirectories
+    /// </summary>
+    public int DirectoryCount { get; init; }
+
+    /// <summary>
+    /// Total size of all files in bytes
+    /// </summary>
+    public long TotalSize { get; init; }
+
+    /// <summary>
+    /// File count by extension
+    /// </summary>
+    public Dictionary<string, int> FilesByExtension { get; init; } = new();
+
+    /// <summary>
+    /// The largest file in the directory
+    /// </summary>
+    public IVirtualFile? LargestFile { get; init; }
+
+    /// <summary>
+    /// Oldest modification date
+    /// </summary>
+    public DateTimeOffset? OldestModified { get; init; }
+
+    /// <summary>
+    /// Newest modification date
+    /// </summary>
+    public DateTimeOffset? NewestModified { get; init; }
+}