release-25.4: sql: fix partial index data loss / phantom rows during update

[fqazi]

Backport 1/1 commits from #166123 on behalf of @fqazi.

---

sql: fix partial index data loss / phantom rows during update

This commit fixes a bug on tables with multiple column families where a
concurrent update that does not overlap with a partial index's column
family could cause the partial index to write a NULL instead of the
actual data, or incorrectly add phantom rows to a temporary index
during a schema change backfill.

This bug was previously masked on default (single-column-family) tables
because an update to any column causes the optimizer to conservatively
fetch all columns in that family. However, with multiple column families,
two normalization rules in the optimizer caused issues:

1. PruneMutationFetchCols: If an update does not change any column
   associated with an index, the optimizer avoids fetching those
   columns. This causes the execution layer to see NULLs for the
   unfetched columns.
2. SimplifyPartialIndexProjections: If an update does not change
   any column associated with a partial index, the optimizer
   simplifies the partial index predicate evaluation to FALSE.
   This causes the execution layer to skip writes to the index.

During a schema change backfill, the execution layer's updater must
unconditionally write complete index entries to temporary (mutating)
indexes for any concurrent update, even if the index's columns are
unchanged. This ensures the backfill merger has a complete snapshot
to correctly reconcile the final index. If columns are pruned (Rule 1)
or writes are simplified away (Rule 2), the temporary index receives
incomplete entries (NULLs) or misses the update entirely.

Furthermore, missing columns can lead to phantom rows if the partial
index predicate evaluates to TRUE when given a NULL value (e.g.,
WHERE val IS NULL).

This change ensures the optimizer always fetches the required
columns and avoids simplifying predicate evaluation if the index
is a mutation index, correctly propagating the full row state to the
execution layer.

Fixes: #166122

Release note (bug fix): Fixed a bug where concurrent updates to a table
using multiple column families during a partial index creation could
result in data loss, incorrect NULL values, or validation failures in
the resulting index.

---

Release justification: addresses a bug that can lead to partial indexes that are corrupt or fail to construct in the face of concurrent updates.

[blathers-crl]

Thanks for opening a backport.

Before merging, please confirm that it falls into one of the following categories (select one):

• Non-production code changes OR fixes for serious issues. Non-production includes test-only changes, build system changes, etc. Serious issues are defined in the policy as correctness, stability, or security issues, data corruption/loss, significant performance regressions, breaking working and widely used functionality, or an inability to detect and debug production issues.
• Other approved changes. These changes must be gated behind a disabled-by-default feature flag unless there is a strong justification not to. Reference the approved ENGREQ ticket in the PR body (e.g., "Fixes ENGREQ-123").

Add a brief release justification to the PR description explaining your selection.

Also, confirm that the change does not break backward compatibility and complies with all aspects of the backport policy.

All backports must be reviewed by the TL and EM for the owning area.

[cockroach-teamcity]

This change is 

[blathers-crl]

✅ PR #166323 is compliant with backport policy

Confidence: high
Critical bug criteria met: [Data corruption/loss Bugs that can cause the DB to return incorrect results or result in suboptimal performance]
Backward compatible: true
Explanation: This PR qualifies as compliant under the CockroachDB backport policy for a critical bug fix. The bug addressed in this PR involves data corruption/loss and incorrect results during concurrent updates with multiple column families in the presence of a partial index, which is considered critical under the policy criteria. This is backed by the PR's description which includes a thorough explanation of how the bug leads to these issues and how the patch resolves them. Additionally, the PR provides a 'Release justification' that clearly states the necessity of the fix for maintaining the integrity of partial indexes, fulfilling the requirement for an exception in the backport policy. All files changed are related to the SQL logic and optimizer, which are core components of the database functionality, confirming the production impact of the changes. There are no changes that suggest the removal of version gates or backward incompatibility, maintaining backward compatibility.

✅ ENGREQ Check Passed: No ENGREQ required (non-production code or serious issues).

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[michae2]

@michae2 reviewed 4 files and all commit messages, and made 1 comment.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on yuzefovich).