HyperQueue crashes with "Deserialization error: Journal load error: SizeLimit"

[svatosFZU]

Hi,
during the weekend, my HyperQueue stopped working and could not be restarted. Looking at the debug log, it ends with:

[2026-01-31T18:36:22.865Z DEBUG hyperqueue::server::restore] Replaying: WorkerConnected 0
Deserialization error: Journal load error: SizeLimit.
It appears that the journal file is corrupted.

I did not know there is a limit on journal size. Does it mean I need to prune it regularly? I dont have a problem with it, I just did not know it.
Also, is this a new thing? In #785, my journal size was twice as big and the HyperQueue was working fine.
Zipped journal (the original size is 4.2GB) and last debug log is at
https://www.fzu.cz/~svatosm/hqJournal.zip
https://www.fzu.cz/~svatosm/hq-debug-output.log

[spirali]

The SizeLimit is a misleading error message. It does not refer to the total size of the file but a size limit on a parsed message. Because the size of every message is checked before the message is written, the only reason why you see this message now is because the file is corrupted and some an invalid number is read as a message size.

After a quick scan, there are a lot of zeros in the middle of the file at offset 0x00010BA818B5.
It is probably why we see lots of messages like:

Replaying: WorkerConnected 0
Replaying: WorkerConnected 0
Replaying: WorkerConnected 0
Replaying: WorkerConnected 0

Worker never has zero ID, and they are unique.

I will try to improve diagnostic for this error message and report back.
But my best guess we observe a problem on the network filesystem / disk when a part of the file was replaced with zeros.

[spirali]

I can confirm that there is about 1280 zero bytes inside the file that causes the problem.

Because of this, messages EventPayload::WorkerConnected read with worker argument 0. (It is the first element in EventPayload enum so it has tag 0).

HQ should write only at the end of the log (or truncate it when there are some partial message written). So it is not clear to me how this could happen. On the other hand, the size of the zero block and its position does not have a page size, so it does not seems to be a drive error. It also seems that block of zeroes start at the boundary of the messages (but does ends on such a boundary). The block of zeros is also near the end of the file but not at the end of file.

Would it be possible that two HQ servers tried to restore from the same journal file? Then there could be some race condition between truncating and writing new data and somehow produce these zeros.

[svatosFZU]

Hi,
I don't see how that could happen. I just deleted the journal and since then, it is working again. Coincidentally, my second machine had a similar problem one day later:

[2026-02-01T10:17:28.257Z DEBUG hyperqueue::server::restore] Replaying: WorkerConnected 0
Deserialization error: Journal load error: InvalidTagEncoding(84).
It appears that the journal file is corrupted.

https://www.fzu.cz/~svatosm/svatosm-hqcrash.zip

[spirali]

It contains the same problem. There are two blocks of zeros (8561B and 8134B); they are both quite near the end of file, there is about 22MB space between them.

This is quite confusing.

Was the server restarted from this journal before this crash or both were reloaded for a first time?

[svatosFZU]

Well, both instances had jobIDs around 200k. So, yes, both were restarted many times (even if it would be just for log rotation). But it does not look like the server was restarted right before this problem appeared.

[svatosFZU]

Hi,
another round of corruption happened last week:

[2026-02-11T10:17:08.739Z DEBUG hyperqueue::server::restore] Replaying: WorkerConnected 0
Deserialization error: Journal load error: InvalidTagEncoding(253).
It appears that the journal file is corrupted.

https://www.fzu.cz/~svatosm/hq-debug-output.log20260211T111709.gz
https://www.fzu.cz/~svatosm/hqJournal-2026-02-16.tar.gz

[spirali]

It seems that some network file systems have weaker guarantees on order of page writing. #1068 adds explicit file synchronization (as the original flush() operation does not do that). I am not sure that it will handle all cases, since the current algorithm for journal reload assumes that data may be missing from the end, but not that zeros occur at the end of the file.

@Kobzol Maybe we need to think about integration about some external file synchronization library, like sqlite (or rather some pure rust solution if it already exists).