Skip to content

Conversation

@xingarr
Copy link

@xingarr xingarr commented Dec 11, 2025

🗑️ Add Automatic File Cleanup for Generative Miners

Problem

Generative miners save generated images, videos, and JSON metadata files to disk temporarily, but these files were never deleted. This caused storage to fill up over time.

Solution

Added automatic cleanup functionality to the existing cleanup thread that removes old files from the output directory.

Changes Made

1. Enhanced Cleanup Thread (neurons/generator/miner.py)

  • Modified _cleanup_loop() to clean up both in-memory tasks AND disk files
  • Added _cleanup_old_files() method to delete old media files
  • Only deletes specific file types: .png, .mp4, .json, .jpg, .jpeg, .gif, .webm, .avi
  • Protects directories (like logs/) and other file types

2. Configuration (gas/config.py)

  • Added --miner.file-max-age-hours parameter (default: 1.0 hour)
  • Uses existing --miner.cleanup-interval for cleanup frequency (default: 3600 seconds)

Behavior

Default Settings:

  • Cleanup runs every 1 hour
  • Deletes files older than 1 hour
  • Logs cleanup activity: 🗑️ Deleted 12 old files from disk (older than 1.0h)

Configuration Example:

# In .env.gen_miner
MINER_CLEANUP_INTERVAL=3600         # Cleanup frequency (seconds)
MINER_FILE_MAX_AGE_HOURS=1          # Delete files older than X hours

Note: I also added some gitignore so please check it as well.

Contribution by Gittensor, learn more at https://gittensor.io

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant