Skip to content

perf(dev-env): tune local database performance defaults#2874

Open
WRasada wants to merge 2 commits into
trunkfrom
perf/dev-env-database-import-tuning
Open

perf(dev-env): tune local database performance defaults#2874
WRasada wants to merge 2 commits into
trunkfrom
perf/dev-env-database-import-tuning

Conversation

@WRasada

@WRasada WRasada commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Problem

The dev-env database containers run with stock settings that are hostile to bulk imports:

  • Binary logging is on (MySQL 8 default): every imported byte is journaled into binlogs nothing consumes locally — ~30GB of pure write amplification for a 30GB dump. myloader's own SET SESSION SQL_LOG_BIN = 0 opt-out fails (the wordpress user lacks SYSTEM_VARIABLES_ADMIN), so this always happens.
  • 128M buffer pool / ~100M redo log (defaults): large imports stall constantly on Redo log writer is waiting for a new redo log file / log_checkpointer still lagging behind warnings.
  • fsync per commit (innodb_flush_log_at_trx_commit=1): durability guarantees that are meaningless for a disposable local database, paid for on every one of myloader's transactions.

Fix

assets/dev-env.lando.template.yml.ejs, both database branches. Note these are server-wide
mysqld startup flags — they benefit all local database use (the buffer pool in particular
helps everyday WordPress queries on large sites), with the most dramatic gains on bulk imports:

Setting MySQL 8.4 MariaDB
Binary logging --skip-log-bin already off by default
Buffer pool --innodb-buffer-pool-size=1G same
Redo capacity --innodb-redo-log-capacity=1G --innodb-log-file-size=1G
Commit flushing --innodb-flush-log-at-trx-commit=2 same

DEV_ENVIRONMENT_VERSION is bumped 2.3.32.3.4 (src/lib/constants/dev-environment.ts) so existing environments re-render the template and rebuild automatically on their next vip dev-env start — no manual dev-env update needed.

Testing

  • Verified the rendered .lando.yml and live values after the auto-update path (maybeUpdateVersion()updateEnvironment() → lando rebuild): log_bin=0, pool 1G, redo 1G, flush=2
  • 201k-table / 30GB import: schema creation phase completed ~30% faster than with stock settings, with zero redo-log stall warnings (previously continuous)
  • Upgrade path validated end-to-end: created an environment on the previous defaults (dev-env version 2.3.3, binlogging active, 128M pool), published a post, then started it with this branch — it auto-updated to 2.3.4 non-interactively, rebuilt, came up cleanly with the new flags applied over the existing data directory, the post survived, and the site loaded with no database errors (manually verified in the browser as well)
  • New e2e spec __tests__/devenv-e2e/014-database-defaults.spec.js covers both scenarios permanently: (1) a fresh environment runs with non-stock database defaults, and (2) an environment stamped with the previous version auto-updates on start, applies the new defaults to its existing data directory, and keeps its data. Assertions use floors (e.g. pool > stock 128M) rather than pinned values, so future tuning of the sizes does not break the test — only losing an override does

Trade-off note

innodb_flush_log_at_trx_commit=2 means up to ~1s of writes can be lost if the host crashes (not just mysqld). For a local dev database that is recreated from imports, this is the intended trade.

Changelog Description

Changed

  • Dev-env: Improved local database performance defaults — these apply to all database use, with the largest gains on big SQL imports (binary logging disabled on MySQL, InnoDB buffer pool raised to 1G to match the VIP production default, larger redo log, relaxed per-commit flushing). Existing environments pick the settings up automatically on their next start

Related

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Disable binary logging (MySQL), raise the buffer pool to 1G (matching
the VIP production default), raise redo log capacity, and relax
per-commit flushing. These are server-wide settings that benefit all
local database use; the largest gains are on bulk SQL imports. Bump
DEV_ENVIRONMENT_VERSION so existing environments pick up the new flags
on next start.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@WRasada WRasada force-pushed the perf/dev-env-database-import-tuning branch from fdf9ff7 to 14cf32f Compare June 5, 2026 04:38
Comment on lines -111 to +118
command: docker-entrypoint.sh mysqld --sql-mode=ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION --max_allowed_packet=67M --mysql-native-password=ON
command: docker-entrypoint.sh mysqld --sql-mode=ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION --max_allowed_packet=67M --mysql-native-password=ON --skip-log-bin --innodb-buffer-pool-size=1G --innodb-redo-log-capacity=1G --innodb-flush-log-at-trx-commit=2

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change friendly to existing environments? That is, if I create a dev env with an old VIP CLI, then run this VIP CLI and restart the environment, will it start?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. Something I did not test, and I'll run some tests and validate

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright. This validated fine and found no issues with existing environments.

I created a new environment on trunk (old 2.3.3 dev-env defaults), created an example post, switched to this branch and restarted the environment. It auto-updated to 2.3.4 and rebuilt, the site loaded fine with no database connection errors, the post survived, and the new flags were verified live in the container.

I also added an e2e test (014-database-defaults.spec.js).

Asserts the running database is not on stock defaults (floors, not
pinned values, so future tuning does not break the test) and that an
environment stamped with the previous version auto-updates on start,
applies the new flags over its existing data directory, and keeps data.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Jun 5, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants