fix: avoid using uninitialized memory

[buriedpot]

Fix crash in RUM index scan when VACUUM removes all items from a posting tree leaf page

During a RUM index scan, if a concurrent VACUUM operation completes and removes all entries from a posting tree leaf page, the scan entry’s nlist field may become zero. In this state, the curItem pointer can end up referencing uninitialized or invalid memory, leading to a backend crash.

This commit resolves the issue by ensuring that curItem is properly handled (or not dereferenced) when nlist == 0, thus preventing undefined behavior during index scans under concurrent vacuuming.

Reproduction Steps

1. Prepare test data

CREATE EXTENSION rum;
ALTER SYSTEM SET enable_seqscan = off;
ALTER SYSTEM SET enable_indexscan = off;
ALTER SYSTEM SET enable_bitmapscan = on;
SELECT pg_reload_conf();

DROP TABLE IF EXISTS t;
CREATE TABLE t (id int, body tsvector);
ALTER TABLE t SET (autovacuum_enabled = FALSE);

INSERT INTO t
SELECT i, to_tsvector('rare word')
FROM generate_series(1, 100000) AS i;

CREATE INDEX ON t USING rum (body rum_tsvector_ops);

DELETE FROM t;  -- Leaves index entries pointing to dead tuples

2. Trigger the race condition using GDB

• Backend 1 (VACUUM):
  Attach GDB and set a breakpoint at rumVacuumPostingTreeLeaves.
  Run:

  VACUUM t;

  Let it pause at the breakpoint.

• Backend 2 (Query):
  Attach another GDB instance and set a breakpoint at startScanEntry.
  Run:

  SELECT * FROM t WHERE body @@ 'rare'::tsquery;

  Let it pause at the breakpoint.

• Resume execution:
  First, continue Backend 1 to completion (so VACUUM fully removes all items from the leaf page).
  Then, continue Backend 2.

  → The query backend will crash due to accessing invalid memory via entry->curItem when entry->nlist == 0.

This fix ensures safe handling of empty posting lists during scans, eliminating the crash under concurrent VACUUM and query workloads. The related issue is #168 and #99.

[buriedpot]

I strongly recommend prioritizing the review and applying of this fix.

[buriedpot]

You can observe that a concurrent VACUUM may set a page’s maxoff to zero in rumvacuum.c:

			if (newMaxOff > 0)
				memcpy(RumDataPageGetData(newPage), cleaned, newSize);

			pfree(cleaned);
			RumPageGetOpaque(newPage)->maxoff = newMaxOff;
			updateItemIndexes(newPage, attnum, &gvs->rumstate);

When newMaxOff == 0, the page ends up with no valid items.
Later, during a RUM index scan, the query backend reads this maxoff as entry->nlist. If nlist == 0, the subsequent logic in rumget.c still attempts to initialize entry->curItem:

			entry->nlist = maxoff;

			ptr = RumDataPageGetData(page);

			for (i = FirstOffsetNumber; i <= maxoff; i = OffsetNumberNext(i))
			{
				ptr = rumDataPageLeafRead(ptr, entry->attnum, &item, true,
										  rumstate);
				entry->list[i - FirstOffsetNumber] = item;
			}

			LockBuffer(entry->buffer, RUM_UNLOCK);
			entry->isFinished = setListPositionScanEntry(rumstate, entry);
			if (!entry->isFinished)
				entry->curItem = entry->list[entry->offset];

Since the loop body is never executed when maxoff == 0, entry->list remains uninitialized. Yet, if setListPositionScanEntry() returns false (which can happen even with an empty list under certain scan states), entry->curItem will point to garbage memory — leading to a crash.

[arseny114]

Hi! Thanks for the report.
Sorry I didn't respond sooner; I was trying to reproduce the problem before commenting. Unfortunately, the PR was closed in the meantime.

For some reason, I couldn't reproduce it the first time, but after a while I tried again and the problem was reproduced.

Could you please reopen the pull request? I'll review and merge it soon.

[buriedpot]

Hi! Thanks for the report. Sorry I didn't respond sooner; I was trying to reproduce the problem before commenting. Unfortunately, the PR was closed in the meantime.

For some reason, I couldn't reproduce it the first time, but after a while I tried again and the problem was reproduced.

Could you please reopen the pull request? I'll review and merge it soon.

OK. I have a comprehensive fix that has been verified. Please allow me to update this PR—just a moment.

[buriedpot]

Hi! Thanks for the report. Sorry I didn't respond sooner; I was trying to reproduce the problem before commenting. Unfortunately, the PR was closed in the meantime.

For some reason, I couldn't reproduce it the first time, but after a while I tried again and the problem was reproduced.

Could you please reopen the pull request? I'll review and merge it soon.

Hi, I have updated the commit.

[arseny114]

Hi, I have updated the commit.

Thanks!

[samrose]

Is there anything we can do to help get this fix merged and released? Thank you!

[arseny114]

Is there anything we can do to help get this fix merged and released? Thank you!

Hi!
We are currently reviewing the fix and plan to complete it soon. Thanks for your patience!

[Strijdhagen]

Just a note that I'm having this issue in production atm. Apreciate all your hard work on the fix!