feat: multi-threaded slave (MTS) parallel DML apply

[jackiesre721]

Implements LOGICAL_CLOCK-based parallel binlog event application, mirroring MySQL 5.7 MTS scheduling. With --num-workers=N, gh-ost applies DML events to the ghost table using N concurrent workers, significantly increasing throughput for high-write tables.

Key components:

• commitBarrier: dependency tracking via last_committed/sequence_number
• mtsScheduleState: new-group detection and epoch reset handling
• dmlCoordinator: transaction grouping and dependency-aware dispatch
• dmlWorker: per-worker goroutine with independent DB connection
• Deadlock-aware retry: immediate retry on errno 1213, 1s sleep on others
• Monotonic coordinate update: prevents checkpoint regression when workers complete out of order

Backward compatible: --num-workers=1 (default) uses the original single-threaded path with zero behavioral changes.

A Pull Request should be associated with an Issue.

We wish to have discussions in Issues. A single issue may be targeted by multiple PRs.
If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues,
and potentially we'll be able to point development in a particular direction.

Related issue: https://github.com/github/gh-ost/issues/0123456789

Further notes in https://github.com/github/gh-ost/blob/master/.github/CONTRIBUTING.md
Thank you! We are open to PRs, but please understand if for technical reasons we are unable to accept each and any PR

Description

This PR [briefly explain what it does]

In case this PR introduced Go code changes:

• contributed code is using same conventions as original code
• script/cibuild returns with no formatting errors, build errors or unit test errors.

[meiji163]

Thanks for the PR @jackiesre721. When we attempted MTS before we ran into data consistency issues. If possible could you run some integration tests and tests under high query load to increase confidence in correctness?

If I have time I will run some tests as well

[jackiesre721]

Hi @meiji163, great questions. Here is a detailed summary of the testing we have done and the specific consistency issues we identified and fixed.

PR #1454 Root Cause Analysis

PR #1454 previously had data consistency issues. We traced the root cause to two bugs:

1. Naive gap-free LWM stalls on sparse sequence numbers

gh-ost only streams binlog rows for the single migrated table, but MySQL assigns sequence_number across ALL transactions on ALL tables. Consecutive transactions on our table might be numbered 5, 9, 14, ... with gaps for other tables' transactions we never see. A naive gap-free LWM (advance only lwm+1) would stall permanently at the first gap because those missing sequence numbers belong to other tables and are never committed in our barrier.

Fix: Replaced gap-free LWM with dispatched-subsequence tracking. The coordinator records every transaction it dispatches via addDelegatedJob(seq) in binlog order. The LWM advances over the committed prefix of that dispatched subsequence — skipping gaps from unobserved cross-table transactions. This mirrors MySQL 8.0's GAQ scheduling restricted to the transactions gh-ost actually applies.

2. Cross-table dependency false blocking

PR #1454 tracked whether a parent transaction was "seen on stream" via an explicit boolean. This had an observe/dispatch ordering hazard.

Fix: Cross-table dependencies are now detected directly from the dispatched set: if lastCommitted was never dispatched on this stream, it is a cross-table parent and we skip waiting. Because the coordinator processes transactions in binlog order and a child's lastCommitted is always < its own sequence_number, any parent on this stream has already been dispatched by the time we check.

3. sync.Cond deadlock under high concurrency

sync.Cond.Wait() does NOT respond to Go context cancellation. Under 16-thread sysbench load with 4 workers, concurrent deadlock retries caused all workers to block in waitForDependency indefinitely (Applied stalled at 82).

Fix: Added waitWhileLocked helper that spawns a short-lived goroutine per Wait() call. The goroutine calls Broadcast() on ctx.Done(), ensuring no waiter blocks past context cancellation.

Tests Run Locally

All tests were run on MySQL 9.6.0 with GTID enabled, binlog_format=ROW.

Unit Tests (28 tests, all passing)

TestCommitBarrier_CommitZeroIsNoop
TestCommitBarrier_WaitForDependencyAlreadySatisfied
TestCommitBarrier_WaitForDependencyBlocksUntilLWMAdvances
TestCommitBarrier_WaitForDependencyZeroIsNoop
TestCommitBarrier_WaitForDependencyCrossTableIsNoop
TestCommitBarrier_WaitForDependencyUndispatchedGapIsNoop
TestCommitBarrier_WaitForAllWorkers
TestCommitBarrier_WaitForAllWorkersNoLostWakeup
TestCommitBarrier_ConcurrentCommitsAdvanceLWM
TestCommitBarrier_Reset
TestCommitBarrier_PR1454_OutOfOrderDoesNotShortCircuit
TestCommitBarrier_PR1454_WaitBlocksWhenLWMBelowLastCommitted
TestCommitBarrier_PR1454_BinlogRotationResetsLWM
TestCommitBarrier_PR1454_ErrorDoesNotMarkCommitted

These specifically cover the PR #1454 scenarios: out-of-order commits, LWM blocking, binlog rotation resets, and error handling.

Sysbench Write-Load Integration Tests

8-thread sysbench, 4 workers (50k rows, 60s):

• ~1,764 TPS, 105,882 transactions, 635,509 DML queries
• Triple verification passed: row count + CHECKSUM TABLE + md5sum

16-thread sysbench, 4 workers (50k rows, 60s):

• ~242 TPS, 14,569 transactions, 1 binlog rotation detected during migration
• Triple verification passed: row count + CHECKSUM TABLE + md5sum

Both tests start sysbench write load BEFORE gh-ost migration to ensure concurrent writes throughout the entire migration lifecycle.

Multi-Iteration Consistency Tests

Also ran the mts-consistency-test.sh script which runs 5 iterations each with 2 and 4 workers, verifying data consistency after each run. All 10 iterations passed.

CI Integration

The localtests/mts-sysbench/ test is now integrated into the existing replica-tests.yml CI workflow. It runs as a standard localtest across all MySQL versions in the matrix (5.7, 8.0, 8.4, Percona). The test:

• Starts sysbench write load on the master
• Runs gh-ost migration with --gtid --num-workers=4 on the replica
• Verifies MTS mode was activated (checks log for "Starting MTS mode with 4 workers")
• Triple-verifies data consistency (row count + CHECKSUM TABLE + md5sum)
• Reports binlog rotation detection

This should give confidence that MTS parallel apply maintains data consistency under concurrent write load across supported MySQL versions.

[meiji163]

Thanks for all the details @jackiesre721. It looks like the CI tests are getting a nil pointer dereference somewhere

[jackiesre721]

Hi @meiji163, thanks for flagging the CI failures. We identified and fixed the root cause of the docker-tests (mysql:8.4.3) failure:

Problem: Under MTS mode with parallel DML workers, the row-copy path uses SELECT ... FOR SHARE NOWAIT which fails immediately with errno 3572 when concurrent workers hold row locks on the ghost table. The internal retry budget (20 attempts) plus the outer retryOperation (3 attempts) was insufficient under heavy write load on MySQL 8.4, causing the migration to FATAL.

Fix: Extracted the noWait decision into rowCopyUsesNoWait() (applier.go) which returns false when NumWorkers > 1, making row-copy use LOCK IN SHARE MODE (blocking wait) instead of NOWAIT. This is the correct trade-off: DML apply is latency-sensitive and should not be preempted by row-copy retries, so row-copy should wait for DML locks to release rather than failing fast. Single-threaded mode on MySQL 8.x retains the original NOWAIT behaviour.

Re: the nil pointer dereference you mentioned — that was caused by context cancellation propagating through onChangelogHeartbeatEvent → enqueueApplyEvent during the FATAL teardown. With the NOWAIT fix in place, the migration no longer hits that crash path because it does not FATAL mid-flight.

The latest push (641f2119) includes this fix. Could you re-run the CI checks?