Zombie extra_files after removing main dataset

[martindemko]

Possibly only on RE, but a deeper investigation is needed.

[zsalvet]

also on galaxy-cz:

# ls -ld /rbd/data/datasets/0/3/5/dataset_035172b1-f891-402b-951c-e24a1f523964*
drwxrwxr-x 2 galaxy galaxy 130 Sep  1  2024 /rbd/data/datasets/0/3/5/dataset_035172b1-f891-402b-951c-e24a1f523964_files

[martenson]

This has been identified to consume dozens of TBs of storage

[martindemko]

So I found out that if a Galaxy dataset file (*.dat) is somehow removed, the extra directory (*_files) is ignored by the cleanup scripts. I confirmed the idea and then manually created an empty .dat file for each "zombie" extra directory and ran manual cleanup with --force-retry for all datasets older than ~year:

PYTHONPATH=/srv/galaxy/venv/lib/python3.11/site-packages/ GALAXY_CONFIG_FILE=/srv/galaxy/config/galaxy.yml GALAXY_LOG_DIR=/data/galaxy/log/gxadmin/ GALAXY_ROOT=/srv/galaxy/server /srv/galaxy/server/scripts/cleanup_datasets/pgcleanup.py -o 375 -g /data/galaxy/log/cleanup_manual_retry.after_dat_create.log --force-retry purge_datasets

It has been running for several hours and has already removed thousands of these zombie extra directories. After it finishes, I will create an empty .dat file for all zombie directories (not just older than a year), and with a correctly working gxadmin galaxy cleanup cron job, the situation should stabilize, and all zombie directories should disappear eventually (considering a reasonable delay parameter for the cleanup script).

Not sure why those dataset files disappeared, though.

[zsalvet]

They did not disappear, they were purged (marked as such in Galaxy database), at least those I reported initially...

[martindemko]

@zsalvet, you are right. But that doesn't mean those files on storage will be erased. Purging a dataset in Galaxy just means the user no longer needs that file, and it's up to us, admins, to handle the cleanup. Galaxy itself offers a nice script to do such a cleanup. However, I only found out recently, thanks to your notice, that the cron job failed to run the script every time since the beginning.