preview:generate-all crashses without an error

[Cattalins]

While running "preview:generate-all", the app crashes without an error.

I usually read the output of "-vv" and check manually if the last file processed is working or not.
If the file is corrupted, I delete it through the browser (update database), and run the command again, only to crash on a different file

I tried to run the command using --workers=100, but eventually they all crash, quite on the same time
Perhaps they all try to process the same file?
(EDIT: Running 25 tasks, output log shows each worker died on a different file)

My suggestion is to change the forking mechanism to a "per media file" worker

1. The main worker lists all media files;
2. Main thread creates a child process;
3. Handle the media file to child (path argument);
4. Child generate preview;
5. Child die;
6. Return to 2 until list is empty;

--workers defaults to 1; user can input a greater value.
This logic will allow to process all files, without crashing the full queue and needing restart the process.

[Cattalins]

TOP shows task as stopped;

PHP can't handle requests until all tasks are killed
ps -e -o stat,pid | grep '^T' | sed 's/T *//g' | xargs kill -9

Excerpt from TOP

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  39027 root      20   0   19068   6144   4992 T   0.0   0.0   0:00.00 sudo -u www-data php -f /var/www/nextcloud2/occ preview:generate-all --workers=100
  39029 root      20   0   19068   2324   1152 S   0.0   0.0   0:00.00 sudo -u www-data php -f /var/www/nextcloud2/occ preview:generate-all --workers=100
  39030 www-data  20   0  177760  67280  21376 T   0.0   0.4   0:00.44 php -f /var/www/nextcloud2/occ preview:generate-all --workers=100
  39032 www-data  20   0  251980 108848  27776 T   0.0   0.6   0:43.86 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39033 www-data  20   0  254548 111036  28032 T   0.0   0.7   0:43.13 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39034 www-data  20   0  256900 113248  27648 T   0.0   0.7   0:43.94 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39035 www-data  20   0  254484 110892  27904 T   0.0   0.7   0:43.33 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39036 www-data  20   0  254284 110544  27648 T   0.0   0.7   0:43.69 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39037 www-data  20   0  258300 115872  28032 T   0.0   0.7   0:48.82 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39038 www-data  20   0  253628 110396  27904 T   0.0   0.7   0:46.07 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39039 www-data  20   0  260448 117184  28032 T   0.0   0.7   0:45.85 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39040 www-data  20   0  254420 111568  27904 T   0.0   0.7   0:46.52 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39041 www-data  20   0  257100 115248  27904 T   0.0   0.7   0:48.78 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39042 www-data  20   0  253756 110236  27904 T   0.0   0.7   0:44.43 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39043 www-data  20   0  256736 113984  27904 T   0.0   0.7   0:47.86 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39044 www-data  20   0  256116 112256  28032 T   0.0   0.7   0:44.33 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39045 www-data  20   0  255436 112580  28032 T   0.0   0.7   0:46.04 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39046 www-data  20   0  256624 113644  27904 T   0.0   0.7   0:45.27 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39047 www-data  20   0  254300 111364  27904 T   0.0   0.7   0:45.75 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39048 www-data  20   0  253788 110576  27904 T   0.0   0.7   0:45.12 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39049 www-data  20   0  255460 111592  27648 T   0.0   0.7   0:44.75 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39050 www-data  20   0  259612 117184  28032 T   0.0   0.7   0:45.20 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39051 www-data  20   0  253988 110948  27904 T   0.0   0.7   0:46.60 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39052 www-data  20   0  253392 109660  27904 T   0.0   0.7   0:44.71 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39053 www-data  20   0  254720 111168  27776 T   0.0   0.7   0:44.27 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39054 www-data  20   0  256104 112144  27776 T   0.0   0.7   0:44.94 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39055 www-data  20   0  253536 110316  27904 T   0.0   0.7   0:46.53 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39056 www-data  20   0  253140 109424  27904 T   0.0   0.7   0:44.08 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39057 www-data  20   0  254300 111352  28032 T   0.0   0.7   0:45.94 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39058 www-data  20   0  253960 110400  27904 T   0.0   0.7   0:44.62 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39059 www-data  20   0  254012 110932  27904 T   0.0   0.7   0:43.81 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39060 www-data  20   0  255272 112324  27776 T   0.0   0.7   0:45.68 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39061 www-data  20   0  253052 110384  27904 T   0.0   0.7   0:45.75 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39062 www-data  20   0  254924 111492  27776 T   0.0   0.7   0:44.54 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39063 www-data  20   0  256352 113952  28160 T   0.0   0.7   0:47.98 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39064 www-data  20   0  254372 111608  28032 T   0.0   0.7   0:45.83 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39065 www-data  20   0  255752 112840  28032 T   0.0   0.7   0:48.02 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39066 www-data  20   0  253404 109792  27904 T   0.0   0.7   0:43.05 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39067 www-data  20   0  253764 110344  28032 T   0.0   0.7   0:42.76 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39068 www-data  20   0  253796 110180  27648 T   0.0   0.7   0:46.14 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39069 www-data  20   0  254940 111596  27776 T   0.0   0.7   0:44.51 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39070 www-data  20   0  256048 112692  27776 T   0.0   0.7   0:43.19 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39071 www-data  20   0  254460 111660  27904 T   0.0   0.7   0:45.69 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39072 www-data  20   0  255044 112024  27648 T   0.0   0.7   0:46.30 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39073 www-data  20   0  253296 109672  27904 T   0.0   0.7   0:43.62 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39074 www-data  20   0  253000 110348  27776 T   0.0   0.7   0:45.44 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39075 www-data  20   0  260612 118056  27904 T   0.0   0.7   0:46.63 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100
  39076 www-data  20   0  257792 114484  27648 T   0.0   0.7   0:46.90 /usr/bin/php8.4 /var/www/nextcloud2/occ preview:generate-all --workers=100

[st3iny]

Hi, thanks for the report. Is there no error logged on stdout and in the nextcloud log files? If so, this is very hard to debug.

You could try to generate a preview using the following command and check if there is an error: occ preview:generate -vvv <file-id-or-path>

To be honest, I didn't test it locally with that many workers. The worker count is meant to be equal to your number of CPU cores (or double perhaps).

[st3iny]

Another question: How did you determine that the worker(s) crashed? Was the exit code non-zero?

[Cattalins]

I tried many workers just to see if the rendering would pass through the problematic files, without crashing the main thread

I don't know how to check the output
What I did was to redirect the output to a file, which made rendering go faster
What I noticed was the processes piling up in TOP, with states T and P.
If too many piled up, the PHP/Apache would not respond to requests

Then on the output file, I could see where the process stopped
(Single or multi thread)

While running a single thread, it just stopped/crashed
No error logged (same as multi thread)
Then I opened the nextcloud on the browser, and browsed to the file
In some cases the file was corrupted, and I had to delete it
In other cases, the browser generated the preview

Then running the script again, would go through all files and skip those
Eventually it completed the job

But my suggestion to avoid this manual labor, is to render the previews on a forked thread
This way, if the rendering thread crashed for no reason, the main thread could fork a new one and render the next file

Since you already handle forking in your program, changing this logic would not be a big deal

How do you handle the forking?
Each child receives a folder to scan, or a list of files?

[Cattalins]

Thinking it over, you don't have to change the logic at all

• Main Thread (Workers 2)
• • Child 1 (do its job as usual)
• • • Right before opening the file, fork again
• • • Forked closes file
• • • Forked die
• • Child 1, collect dead (success or error)
• • Child 1, move to next file
• • • Right befor openin the file, fork again
• • • Forked closes file
• • • Forked die
• • Child 1, collect dead (success or error)
• Main Thread collect dead

This will prevent the main thread to die or hang unexpectedly