diff --git a/.gitignore b/.gitignore index e3f8d79..5a5d4d5 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ .env +.DS_Store -ref \ No newline at end of file +ref diff --git a/skills/tidb-query-tuning/AGENTS.md b/skills/tidb-query-tuning/AGENTS.md new file mode 100644 index 0000000..478bd99 --- /dev/null +++ b/skills/tidb-query-tuning/AGENTS.md @@ -0,0 +1,92 @@ +# TiDB Query Tuning Agent Notes + +Use this skill directory for two different knowledge sources: + +- curated topic references under `references/` +- GitHub issue-derived field experience generated into markdown files + +## When to mine issue experience + +Use GitHub issue mining when: + +- the local references do not cover a customer-facing symptom well enough +- you need a recent field precedent instead of a general tuning rule +- you want fix PRs, merge timestamps, and open issue reminders +- you want to build or refresh a local corpus of customer-driven planner or stats bugs + +Do not start from issue mining if a stable reference under `references/` already answers the question. Use the issue corpus to complement the topic docs, not replace them. + +## Default workflow + +1. Start with the local references in `references/`. +2. Search `references/optimizer-oncall-experiences-redacted/` for a symptom match. +3. Search `references/tidb-customer-planner-issues/` if you need linked PRs, merge times, or still-open customer gaps. +4. If the local corpora are still missing the pattern, mine GitHub issues with the script in `scripts/`. +5. Review the generated files, then fold reusable learnings back into the relevant reference docs when appropriate. + +## Issue mining script + +Use: + +`scripts/generate_tidb_issue_experiences.py` + +The script: + +- searches GitHub issues with a provided query +- follows issue timeline cross-references and explicit fix comments +- collects linked PR metadata and changed files +- writes one markdown file per issue +- writes an index `README.md` into the output directory + +The current checked-in issue corpus lives under: + +- `references/tidb-customer-planner-issues/` + +## Recommended query patterns + +For customer-driven planner issues: + +```text +repo:pingcap/tidb is:issue label:"report/customer" label:"sig/planner" created:>=2024-01-01 +``` + +For stats-heavy issue mining: + +```text +repo:pingcap/tidb is:issue label:"report/customer" (label:"sig/planner" OR label:"sig/execution") stats created:>=2024-01-01 +``` + +Adjust the query rather than hardcoding different behaviors into the script. + +## Usage example + +```bash +python3 skills/tidb-query-tuning/scripts/generate_tidb_issue_experiences.py \ + --query 'repo:pingcap/tidb is:issue label:"report/customer" label:"sig/planner" created:>=2024-01-01' \ + --out-dir outputs/tidb-customer-planner-issues +``` + +## What to keep from generated issue files + +Keep and reuse: + +- customer-facing symptom descriptions +- investigation clues +- linked fix PRs +- merge timestamps +- affected modules +- open issues that should remain on the reminder list + +Do not treat every generated issue as a mature tuning rule. Generated files are raw field precedents. Promote them into `references/` only after the pattern is stable and reusable. + +## Tooling assumptions + +- `gh` CLI must be installed and authenticated +- network access to GitHub must be available +- the output directory should be treated as generated content + +## Editing guidance + +- Keep curated docs in `references/` concise and topic-oriented. +- Keep generated issue corpora outside the hand-written topic docs unless they are intentionally promoted. +- If you regenerate a corpus, prefer writing into a fresh output directory or knowingly replacing the previous generated set. diff --git a/skills/tidb-query-tuning/SKILL.md b/skills/tidb-query-tuning/SKILL.md new file mode 100644 index 0000000..851d454 --- /dev/null +++ b/skills/tidb-query-tuning/SKILL.md @@ -0,0 +1,66 @@ +# TiDB Query Tuning + +**name:** tidb-query-tuning + +**description:** Diagnose and optimize slow TiDB queries using optimizer hints, session variables, join strategy selection, subquery optimization, and index tuning. Use when a query is slow, produces a bad plan, or needs performance guidance on TiDB. + +--- + +## Workflow + +1. **Capture the current plan:** + - Run `EXPLAIN ANALYZE ` to get actual execution stats. + - Compare `estRows` vs `actRows` — large divergence means stale or missing statistics. + - Note the most expensive operators (wall time, memory, rows processed). + +2. **Check statistics health:** + - `SHOW STATS_HEALTHY WHERE Db_name = '' AND Table_name = '';` + - If health < 80 or `actRows` diverges significantly from `estRows`, run `ANALYZE TABLE
;` and re-check the plan. + - For specific indexes: `ANALYZE TABLE
INDEX ;` + +3. **Identify the bottleneck pattern:** + - **Bad join order or strategy** → see `references/join-strategies.md` + - **Subquery not handled well** → see `references/subquery-optimization.md` + - **Wrong or missing index** → see `references/index-selection.md` + - **Optimizer choosing a suboptimal plan despite good stats** → see `references/optimizer-hints.md` and `references/session-variables.md` + - **Stats are stale or auto analyze cannot keep up** → see `references/stats-health-and-auto-analyze.md` + - **Plans change after restart or sync stats loading times out** → see `references/stats-loading-and-startup.md` + - **Need to tune analyze version, column coverage, or memory-heavy stats collection** → see `references/stats-version-and-analyze-configuration.md` + - **Need a matching field incident, workaround, or fixed-version precedent** → search `references/optimizer-oncall-experiences-redacted/` + - **Need recent customer issue precedents with linked PRs and merge timestamps** → search `references/tidb-customer-planner-issues/` + +4. **Apply the fix:** + - Prefer the least invasive change: refresh stats → add index → hint → session variable. + - Use hints when the fix is query-specific. + - Use session variables when the fix applies to a workload pattern. + +5. **Verify the improvement:** + - Re-run `EXPLAIN ANALYZE` with the fix applied. + - Confirm `actRows` and execution time improved. + - If the fix is a hint, document it in a SQL comment so future readers understand why. + +## High-signal rules + +- **Always check stats first.** Most bad plans in TiDB come from stale or missing statistics, not optimizer bugs. +- **Treat stats maintenance as capacity planning.** If `AUTO ANALYZE` cannot keep up with stats decay, plan quality will drift even when SQL does not change. +- **`EXPLAIN ANALYZE` is the ground truth.** `EXPLAIN` alone shows estimates; `ANALYZE` shows what actually happened. +- **Search known field cases before inventing a new workaround.** The oncall corpus under `references/optimizer-oncall-experiences-redacted/` is useful for symptom matching, investigation signals, and fix-version lookup. +- **Search recent GitHub issue precedents when fix lineage matters.** The corpus under `references/tidb-customer-planner-issues/` is useful when you need linked PRs, merge times, and still-open customer gaps. +- **Correlated subqueries:** TiDB decorrelates by default. When the subquery is well-indexed and the outer query is selective, `NO_DECORRELATE()` often wins. See `references/subquery-optimization.md`. +- **Join strategies matter:** TiDB supports hash join, index join, merge join, and shuffle joins. The right choice depends on table sizes, index availability, and data distribution. See `references/join-strategies.md`. +- **Hints are per-query; variables are per-session/global.** Use hints for surgical fixes, variables for workload-wide tuning. +- **TiFlash acceleration:** For analytical queries on large tables, push computation to TiFlash replicas using `READ_FROM_STORAGE(TIFLASH[
])`. See `references/session-variables.md`. + +## References + +- `references/optimizer-hints.md` — Optimizer hints: syntax, catalog, and when to use each. +- `references/session-variables.md` — Session/global variables that affect plan choice. +- `references/join-strategies.md` — Join algorithms, when TiDB picks each, and how to override. +- `references/subquery-optimization.md` — Decorrelation, semi-join, EXISTS/IN patterns and NO_DECORRELATE. +- `references/index-selection.md` — Index hints, invisible indexes, index advisor, composite index guidance. +- `references/explain-patterns.md` — Reading EXPLAIN ANALYZE output to identify bottlenecks. +- `references/stats-health-and-auto-analyze.md` — Statistics health, auto analyze backlog diagnosis, and safe concurrency tuning. +- `references/stats-loading-and-startup.md` — Init stats, sync load, restart-time plan instability, and version-based mitigation. +- `references/stats-version-and-analyze-configuration.md` — Stats versioning, analyze coverage, and memory-safe stats collection settings. +- `references/optimizer-oncall-experiences-redacted/` — Redacted optimizer oncall case corpus with user symptoms, investigation signals, workarounds, and fixed versions. +- `references/tidb-customer-planner-issues/README.md` — Generated GitHub issue corpus with one file per customer-driven planner issue, including linked PRs and merge timestamps. diff --git a/skills/tidb-query-tuning/references/explain-patterns.md b/skills/tidb-query-tuning/references/explain-patterns.md new file mode 100644 index 0000000..112e642 --- /dev/null +++ b/skills/tidb-query-tuning/references/explain-patterns.md @@ -0,0 +1,142 @@ +# Reading EXPLAIN ANALYZE Output + +`EXPLAIN ANALYZE` is the primary tool for understanding how TiDB actually executes a query. It runs the query and reports actual row counts, execution time, and memory usage for each operator. + +## Syntax + +```sql +-- Basic (text format): +EXPLAIN ANALYZE SELECT ...; + +-- Structured JSON format (better for programmatic analysis): +EXPLAIN FORMAT = "tidb_json" SELECT ...; + +-- Estimate only (does NOT execute the query): +EXPLAIN SELECT ...; +``` + +**Use `EXPLAIN ANALYZE` for tuning.** Plain `EXPLAIN` shows estimates only, which may be wrong. + +## Key columns in EXPLAIN ANALYZE output + +| Column | Meaning | +|--------|---------| +| `id` | Operator name and position in the tree | +| `estRows` | Estimated row count from optimizer statistics | +| `actRows` | Actual row count observed during execution | +| `task` | Where the operator runs: `root` (TiDB), `cop[tikv]` (TiKV coprocessor), `cop[tiflash]` (TiFlash) | +| `access object` | Table, index, or partition being accessed | +| `execution info` | Wall time, loops, memory, disk usage, concurrency | +| `operator info` | Filter conditions, join keys, sort keys | + +## What to look for + +### 1. estRows vs actRows divergence + +Large differences indicate stale or inaccurate statistics. + +``` +estRows: 100 actRows: 500000 ← Stats are wrong! +``` + +**Action:** Run `ANALYZE TABLE
` and re-check. + +### 2. Expensive operators (by wall time) + +Look at `execution info` for `time:` values. The operator with the longest time is the bottleneck. + +``` +execution info: time:2.5s, loops:1, ... ← This is the bottleneck +``` + +### 3. Full table scans on large tables + +``` +TableFullScan table:orders actRows:10000000 +``` + +**Action:** Add an index or use a hint to force an index scan. + +### 4. Hash join with large build side + +``` +HashJoin actRows:50000 +├── Build actRows:5000000 ← Build side is huge +└── Probe actRows:50000 +``` + +**Action:** Consider `INL_JOIN` if the build side has an index, or `LEADING` to swap build/probe sides. + +### 5. Unnecessary Sort operators + +``` +Sort actRows:1000000 +└── TableFullScan +``` + +If there's an index that provides the sort order, use `ORDER_INDEX` to eliminate the Sort. + +### 6. Apply operator (correlated subquery) + +``` +Apply actRows:1000 +├── Outer actRows:1000 +└── Inner actRows:1000 (executed 1000 times) +``` + +This means correlated execution. If `actRows` on the outer side is small, this is fine. If large, consider letting TiDB decorrelate (remove `NO_DECORRELATE`) or adding an index on the inner side. + +## Operator reference + +### Scan operators + +| Operator | Meaning | +|----------|---------| +| `TableFullScan` | Full table scan — reads every row | +| `TableRangeScan` | Scans a range of the primary key | +| `IndexRangeScan` | Scans a range of an index | +| `IndexFullScan` | Scans the entire index | +| `IndexLookUp` | Two-phase: index scan → table lookup for remaining columns | + +### Join operators + +| Operator | Meaning | +|----------|---------| +| `HashJoin` | Hash join (look for Build and Probe children) | +| `IndexJoin` | Index nested loop join | +| `MergeJoin` | Sort-merge join | +| `Apply` | Correlated subquery execution (per-row) | + +### Aggregation operators + +| Operator | Meaning | +|----------|---------| +| `HashAgg` | Hash-based aggregation | +| `StreamAgg` | Stream aggregation (requires sorted input) | + +### Other operators + +| Operator | Meaning | +|----------|---------| +| `Sort` | Sorts rows (expensive for large datasets) | +| `TopN` | Sort + limit combined (more efficient than separate Sort + Limit) | +| `Selection` | Filters rows (WHERE conditions not pushed to scan) | +| `Projection` | Computes output columns | +| `Limit` | Returns only N rows | + +## Diagnostic workflow + +``` +1. Run EXPLAIN ANALYZE +2. Find the operator with the highest wall time +3. Check estRows vs actRows for that operator and its children + ├── Big divergence → ANALYZE TABLE, then re-run + └── Stats are fine → Operator choice is the problem +4. Identify the pattern: + ├── Full scan → Add index or USE_INDEX hint + ├── Wrong join strategy → HASH_JOIN / INL_JOIN hint + ├── Wrong join order → LEADING hint + ├── Expensive correlated subquery → Check NO_DECORRELATE guidance + └── Unnecessary Sort → ORDER_INDEX hint or add sorted index +5. Apply fix and re-run EXPLAIN ANALYZE to verify +``` diff --git a/skills/tidb-query-tuning/references/index-selection.md b/skills/tidb-query-tuning/references/index-selection.md new file mode 100644 index 0000000..9f0b2b7 --- /dev/null +++ b/skills/tidb-query-tuning/references/index-selection.md @@ -0,0 +1,137 @@ +# Index Selection + +Good index design is the single highest-impact tuning lever. TiDB's optimizer uses cost-based index selection, but it can choose poorly when stats are stale or cardinality is misestimated. + +## Index hints + +### USE_INDEX + +Force the optimizer to use a specific index: + +```sql +SELECT /*+ USE_INDEX(t, idx_status_created) */ * +FROM orders t +WHERE t.status = 'pending' AND t.created_at > '2024-01-01'; +``` + +### IGNORE_INDEX + +Prevent use of a specific index (useful for testing whether a different index or table scan is faster): + +```sql +SELECT /*+ IGNORE_INDEX(t, idx_old) */ * +FROM orders t +WHERE t.status = 'pending'; +``` + +### USE_INDEX_MERGE + +Combine multiple indexes when a query filters on columns covered by different indexes: + +```sql +-- Uses both idx_status and idx_region, merges results +SELECT /*+ USE_INDEX_MERGE(t, idx_status, idx_region) */ * +FROM orders t +WHERE t.status = 'pending' OR t.region = 'us-east'; +``` + +Index merge is particularly useful for OR conditions that span different indexes. + +### ORDER_INDEX / NO_ORDER_INDEX + +Control whether the optimizer uses an index to provide sort order: + +```sql +-- Force ordered scan: avoid a Sort operator by reading from index in order +SELECT /*+ ORDER_INDEX(t, idx_created) */ * +FROM orders t +ORDER BY t.created_at DESC +LIMIT 20; + +-- Prevent ordered scan: if the optimizer incorrectly chooses a slow ordered scan +SELECT /*+ NO_ORDER_INDEX(t, idx_created) */ * +FROM orders t +WHERE t.status = 'pending' +ORDER BY t.created_at DESC +LIMIT 20; +``` + +## Composite index design + +### Leftmost prefix rule + +TiDB (like MySQL) can only use a composite index from the leftmost column onward. An index on `(a, b, c)` supports: +- Queries filtering on `a` +- Queries filtering on `a, b` +- Queries filtering on `a, b, c` +- But NOT queries filtering only on `b` or `c` + +### Column ordering guidelines + +1. **Equality columns first:** Columns in `=` conditions go leftmost. +2. **Range column last:** The column used in `>`, `<`, `BETWEEN`, or `LIKE 'prefix%'` goes after equality columns. +3. **High-selectivity columns first** (among equality columns): Reduces rows scanned early. + +```sql +-- Query: WHERE status = 'active' AND region = 'us' AND created_at > '2024-01-01' +-- Best index: (status, region, created_at) +-- status and region are equality; created_at is range +``` + +### Covering indexes + +If a query only needs columns in the index (plus the primary key), TiDB can serve it from the index alone without a table lookup: + +```sql +-- If index is (status, created_at) and PK is id: +-- This query is "covered" — no table lookup needed +SELECT id, status, created_at FROM orders WHERE status = 'pending'; +``` + +## Invisible indexes + +Test the impact of dropping an index without actually dropping it: + +```sql +-- Make index invisible (optimizer won't use it, but it's still maintained) +ALTER TABLE orders ALTER INDEX idx_old INVISIBLE; + +-- Test queries... if performance is fine, drop it +DROP INDEX idx_old ON orders; + +-- If queries regressed, make it visible again +ALTER TABLE orders ALTER INDEX idx_old VISIBLE; +``` + +## Diagnosing index selection issues + +### Step 1: Check which index is used + +```sql +EXPLAIN SELECT * FROM orders WHERE status = 'pending' AND region = 'us'; +``` + +Look for `IndexRangeScan`, `IndexFullScan`, or `TableFullScan` in the plan. + +### Step 2: Check index statistics + +```sql +SHOW STATS_HEALTHY WHERE Table_name = 'orders'; +SHOW INDEX FROM orders; +``` + +### Step 3: Compare index choices + +```sql +-- Force each candidate index and compare EXPLAIN ANALYZE results +EXPLAIN ANALYZE SELECT /*+ USE_INDEX(orders, idx_status) */ * FROM orders WHERE ...; +EXPLAIN ANALYZE SELECT /*+ USE_INDEX(orders, idx_region) */ * FROM orders WHERE ...; +EXPLAIN ANALYZE SELECT /*+ USE_INDEX(orders, idx_status_region) */ * FROM orders WHERE ...; +``` + +## Common pitfalls + +- **Too many indexes:** Each index adds write overhead and storage. Only create indexes that serve real query patterns. +- **Redundant indexes:** Index `(a, b)` makes a standalone index on `(a)` redundant. Audit with `SHOW INDEX`. +- **Low-selectivity leading column:** An index on `(gender, user_id)` is almost useless if the first column has only 2-3 distinct values and queries don't always filter on it. +- **Stale stats on new indexes:** After creating an index, run `ANALYZE TABLE` to collect stats for it. Otherwise the optimizer may not pick it. diff --git a/skills/tidb-query-tuning/references/join-strategies.md b/skills/tidb-query-tuning/references/join-strategies.md new file mode 100644 index 0000000..6173ade --- /dev/null +++ b/skills/tidb-query-tuning/references/join-strategies.md @@ -0,0 +1,114 @@ +# Join Strategies + +TiDB supports multiple join algorithms. The optimizer picks one based on cost estimation, but stale stats or cardinality misestimation can lead to a wrong choice. + +## Available join algorithms + +### Hash Join (`HASH_JOIN`) + +- **How it works:** Builds a hash table on the smaller (build) side, probes with the larger (probe) side. +- **Best for:** Large equi-joins where neither side has a useful index on the join key. +- **Cost profile:** O(build + probe) in time, O(build side) in memory. +- **Watch out for:** Memory pressure when the build side is large. Check `MEMORY_QUOTA` or `tidb_mem_quota_query`. + +### Index Nested Loop Join (`INL_JOIN`) + +- **How it works:** For each row on the outer side, probes the inner side using an index lookup. +- **Best for:** Inner table has a good index on the join key; outer side is small to moderate. +- **Cost profile:** O(outer rows × index lookup cost). Very fast when outer side is small. +- **Watch out for:** Outer side cardinality misestimation. If outer side is actually large, this becomes expensive. + +### Index Hash Join (`INL_HASH_JOIN`) + +- **How it works:** Like INL_JOIN but uses hash matching on the inner side instead of pure index order. +- **Best for:** Similar to INL_JOIN but when inner side needs hash-based matching. + +### Merge Join (`MERGE_JOIN`) + +- **How it works:** Both sides sorted on join key, then merged in a single pass. +- **Best for:** Both sides already sorted (e.g., from index scans on the join key) or when data is naturally ordered. +- **Cost profile:** O(N + M) after sorting. Sorting cost matters if data isn't pre-sorted. + +### Shuffle Hash Join (`SHUFFLE_JOIN`) — MPP only + +- **How it works:** Redistributes (shuffles) both sides by join key across TiFlash nodes, then hash joins locally. +- **Best for:** Large-to-large joins in TiFlash where neither side is small enough to broadcast. + +### Broadcast Join (`BROADCAST_JOIN`) — MPP only + +- **How it works:** Broadcasts the smaller side to all TiFlash nodes, then joins locally. +- **Best for:** One side is small, the other is large. Avoids shuffling the large side. + +## Decision guide + +``` +Both tables small (< 10K rows)? +├── Doesn't matter much. Any strategy works. +│ +Is there a good index on the join key of one side? +├── YES → Is the other side small? +│ ├── YES → INL_JOIN (indexed side as inner) +│ └── NO → Still consider INL_JOIN if indexed side is large but +│ outer side is moderate. Otherwise HASH_JOIN. +└── NO → HASH_JOIN (smaller side as build). + └── Are both sides pre-sorted on join key? + └── YES → Consider MERGE_JOIN. + +Using TiFlash? +├── One side small → BROADCAST_JOIN +└── Both sides large → SHUFFLE_JOIN +``` + +## Forcing join strategy + +```sql +-- Force hash join between orders and customers +SELECT /*+ HASH_JOIN(o, c) */ * +FROM orders o JOIN customers c ON o.cust_id = c.id; + +-- Force index join with customers as inner (probed) side +SELECT /*+ INL_JOIN(c) */ * +FROM orders o JOIN customers c ON o.cust_id = c.id; + +-- Force join order: join orders and items first, then customers +SELECT /*+ LEADING(o, i, c) */ * +FROM orders o +JOIN items i ON i.order_id = o.id +JOIN customers c ON c.id = o.cust_id; +``` + +## Common misoptimization patterns + +### Pattern: Hash join with huge build side + +**Symptom:** `EXPLAIN ANALYZE` shows `HashJoin` with a large `Build` side consuming excessive memory or spilling to disk. + +**Fix:** If the inner table has an index on the join key, try `INL_JOIN`: + +```sql +SELECT /*+ INL_JOIN(big_table) */ ... +FROM small_table s JOIN big_table b ON s.id = b.ref_id; +``` + +### Pattern: Index join on a non-selective outer side + +**Symptom:** `EXPLAIN ANALYZE` shows `IndexJoin` with a large `actRows` on the outer (driving) side, causing millions of index probes. + +**Fix:** Switch to `HASH_JOIN`: + +```sql +SELECT /*+ HASH_JOIN(t1, t2) */ ... +FROM t1 JOIN t2 ON t1.id = t2.ref_id +WHERE t1.status = 'active'; +``` + +### Pattern: Wrong join order in multi-way join + +**Symptom:** Intermediate join produces a huge result that is then joined with a small table. + +**Fix:** Use `LEADING` to force the small table earlier: + +```sql +SELECT /*+ LEADING(small, medium, large) */ ... +FROM large l JOIN medium m ON ... JOIN small s ON ...; +``` diff --git a/skills/tidb-query-tuning/references/optimizer-cost-factors.md b/skills/tidb-query-tuning/references/optimizer-cost-factors.md new file mode 100644 index 0000000..41449ea --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-cost-factors.md @@ -0,0 +1,181 @@ +# Optimizer Cost Factors + +TiDB provides a set of `tidb_opt_*_cost_factor` variables that act as multipliers on the cost model. Each variable controls the cost of a specific physical operator. By adjusting these multipliers you can **encourage** or **discourage** the optimizer from choosing particular operators without using hard-coded hints. + +These cost factors were delivered to allow the EXPLAIN EXPLORE feature of TiDB to iterate through alternative cost factors and explore other viable query plans that may not have been chosen as the winning plan initially. + +## How cost factors work + +- Every cost factor **defaults to 1.0** (neutral — no adjustment). +- The factor is a **multiplier** on the operator's calculated cost: + - **Lower than 1.0** (e.g., 0.5) → makes the operator **cheaper** → **encourages** the optimizer to choose it. + - **Higher than 1.0** (e.g., 10 or 100) → makes the operator **more expensive** → **discourages** the optimizer from choosing it. +- The optimizer still uses cost-based selection — the factor biases the cost, it doesn't force or disable the operator entirely. + +## Setting cost factors + +Cost factors can be set at three scopes: + +### Global (all new connections) + +```sql +SET GLOBAL tidb_opt_hash_join_cost_factor = 10; +``` + +Use for cluster-wide tuning when a workload systematically benefits from avoiding (or favoring) a particular operator. + +### Session (current connection only) + +```sql +SET @@session.tidb_opt_table_full_scan_cost_factor = 100; +``` + +Use for a specific workload or batch job within a single connection. + +### Per-query via SET_VAR hint + +```sql +SELECT /*+ SET_VAR(tidb_opt_hash_join_cost_factor=100) */ * +FROM orders o JOIN customers c ON o.cust_id = c.id; +``` + +Use for surgical, per-query tuning — the most precise approach. The variable is set only for the duration of that single statement. + +## Complete cost factor catalog + +### Scan operators + +| Variable | Operator affected | Use case | +|----------|-------------------|----------| +| `tidb_opt_table_full_scan_cost_factor` | TableFullScan | Increase to discourage full table scans; the optimizer will prefer index scans when available | +| `tidb_opt_table_range_scan_cost_factor` | TableRangeScan | Adjust cost of range scans on the primary key / clustered index | +| `tidb_opt_table_rowid_scan_cost_factor` | TableRowIDScan | Adjust cost of scanning by row ID (typically after an index lookup) | +| `tidb_opt_index_scan_cost_factor` | IndexScan (single index range scan) | Lower to encourage index scans; increase to push toward full scans or other indexes | +| `tidb_opt_table_tiflash_scan_cost_factor` | TiFlash table scan | Lower to encourage TiFlash reads for analytical queries; increase to prefer TiKV | + +### Reader operators (TiDB ↔ storage layer) + +| Variable | Operator affected | Use case | +|----------|-------------------|----------| +| `tidb_opt_table_reader_cost_factor` | TableReader (reads full rows from TiKV) | Adjust the cost of reading full table data from TiKV | +| `tidb_opt_index_reader_cost_factor` | IndexReader (reads from index only, no table lookup) | Adjust the cost of covering index reads | +| `tidb_opt_index_lookup_cost_factor` | IndexLookUp (index scan + table lookup) | Increase to discourage index lookups when the table lookup is expensive; lower to encourage them | +| `tidb_opt_index_merge_cost_factor` | IndexMerge (combines multiple indexes) | Lower to encourage index merge for OR conditions; increase if index merge is causing overhead | + +### Join operators + +| Variable | Operator affected | Use case | +|----------|-------------------|----------| +| `tidb_opt_hash_join_cost_factor` | HashJoin | Increase to discourage hash joins (e.g., when memory is constrained); lower to encourage them | +| `tidb_opt_index_join_cost_factor` | IndexJoin (index nested loop join) | Lower to encourage index joins when the inner side has a good index; increase if index join probes are expensive | +| `tidb_opt_merge_join_cost_factor` | MergeJoin | Lower to encourage merge joins when data is pre-sorted; increase to prefer hash or index joins | + +### Aggregation operators + +| Variable | Operator affected | Use case | +|----------|-------------------|----------| +| `tidb_opt_hash_agg_cost_factor` | HashAgg | Increase to discourage hash aggregation; lower to encourage it | +| `tidb_opt_stream_agg_cost_factor` | StreamAgg | Lower to encourage stream aggregation when data is sorted on the GROUP BY key | + +### Other operators + +| Variable | Operator affected | Use case | +|----------|-------------------|----------| +| `tidb_opt_sort_cost_factor` | Sort | Increase to discourage explicit sort operators (optimizer may prefer index-ordered reads instead) | +| `tidb_opt_topn_cost_factor` | TopN (sort + limit combined) | Adjust cost of TopN operations | +| `tidb_opt_limit_cost_factor` | Limit | Adjust cost of Limit operators | + +## Practical examples + +### Discourage full table scans, prefer index access + +```sql +-- Session-wide: make full table scans 100x more expensive +SET @@session.tidb_opt_table_full_scan_cost_factor = 100; + +-- Per-query alternative: +SELECT /*+ SET_VAR(tidb_opt_table_full_scan_cost_factor=100) */ * +FROM orders WHERE status = 'pending'; +``` + +### Discourage hash join, encourage index join + +When the inner table has a good index on the join key but the optimizer keeps choosing hash join: + +```sql +SELECT /*+ SET_VAR(tidb_opt_hash_join_cost_factor=100) SET_VAR(tidb_opt_index_join_cost_factor=0.5) */ * +FROM orders o JOIN customers c ON o.cust_id = c.id +WHERE o.created_at > '2024-01-01'; +``` + +### Encourage TiFlash for analytical queries + +```sql +SELECT /*+ SET_VAR(tidb_opt_table_tiflash_scan_cost_factor=0.1) */ + region, SUM(amount) +FROM orders +GROUP BY region; +``` + +### Discourage sort operators (prefer index-ordered reads) + +```sql +SET @@session.tidb_opt_sort_cost_factor = 50; + +SELECT * FROM orders ORDER BY created_at DESC LIMIT 20; +-- Optimizer will now strongly prefer an index on created_at over a full scan + sort +``` + +### Combine multiple factors to reshape a complex query plan + +```sql +-- Discourage full scans and hash joins; encourage index lookups +SELECT /*+ + SET_VAR(tidb_opt_table_full_scan_cost_factor=100) + SET_VAR(tidb_opt_hash_join_cost_factor=50) + SET_VAR(tidb_opt_index_lookup_cost_factor=0.5) +*/ * +FROM orders o +JOIN items i ON i.order_id = o.id +JOIN products p ON p.id = i.product_id +WHERE o.status = 'active'; +``` + +## Cost factors vs hints + +| Approach | Precision | Effect | +|----------|-----------|--------| +| **Hints** (e.g., `HASH_JOIN(t1, t2)`) | Forces a specific operator | Hard override — optimizer has no choice | +| **Cost factors** | Biases the cost model | Soft influence — optimizer still picks the cheapest plan, but costs are weighted differently | + +**When to prefer cost factors over hints:** +- You want to **nudge** the optimizer without completely overriding it. +- The query has many joins or operators, and hinting each one is impractical. +- You want a **session-wide or global** policy (e.g., "always discourage full scans") rather than per-query tuning. +- You have a pure OLTP application and want to discourage OLAP style operations (table full scans, hash joins etc). +- You're unsure which specific operator is the problem — adjusting cost factors lets the optimizer re-evaluate the whole plan. + +**When to prefer hints:** +- You know exactly which operator and which tables should use a specific strategy. +- You need to force a certain query join order or index choice +- Cost factor adjustments still don't produce the desired plan. + +**What if the cost factor doesn't discourage the operator?:** +- It's possible that the desired operator isn't valid for this plan - for example, discouraging hash join won't impact a query that doesn't involve a join. +- There are logical rewrites in the optimizer that may disable some plan choices. An example is for correlated subqueries, if you're not seeing a desired index join it may be because the optimizer decorrelated the subquery, and the solution is either the NO_DECORRELATE() hint or for subqueries in the select list you can use the session/global variable tidb_opt_enable_no_decorrelate_in_select. + +## Diagnostic workflow + +1. Run `EXPLAIN ANALYZE` on the slow query. +2. Identify the problematic operator (e.g., an unnecessary TableFullScan or HashJoin with a huge build side). +3. Increase the cost factor for the unwanted operator and/or decrease the factor for the preferred alternative. +4. Re-run `EXPLAIN ANALYZE` and compare execution time and plan shape. +5. If the plan improves, decide the right scope (per-query SET_VAR, session, or global). + +## Cautions + +- **Don't set extreme values globally without testing.** A global `tidb_opt_table_full_scan_cost_factor = 1000` will affect every query in the cluster, including those where a full scan is genuinely the best plan. +- **Prefer per-query SET_VAR for targeted tuning.** It's the safest scope — no side effects on other queries. +- **Cost factors don't disable operators.** Even at very high values, the optimizer may still choose the operator if there is no alternative. Use hints to truly force a choice. +- **Re-evaluate after stats refresh.** A bad plan caused by stale stats may resolve itself after `ANALYZE TABLE`, making the cost factor adjustment unnecessary. +- **Document non-default settings.** If you set cost factors at session or global scope, document the reason so future maintainers understand the intent. diff --git a/skills/tidb-query-tuning/references/optimizer-hints.md b/skills/tidb-query-tuning/references/optimizer-hints.md new file mode 100644 index 0000000..f1f13df --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-hints.md @@ -0,0 +1,85 @@ +# Optimizer Hints + +TiDB supports query-level optimizer hints using the `/*+ HINT() */` syntax placed immediately after `SELECT`, `UPDATE`, or `DELETE`. + +## Syntax + +```sql +SELECT /*+ HINT_NAME(arguments) */ ... +``` + +Multiple hints can be combined: + +```sql +SELECT /*+ HASH_JOIN(t1, t2) NO_DECORRELATE() */ ... +``` + +## Hint Catalog + +### Join strategy hints + +| Hint | Effect | When to use | +|------|--------|-------------| +| `HASH_JOIN(t1, t2)` | Force hash join between named tables | Large table joins without useful indexes; both sides are large | +| `INL_JOIN(t)` | Force index nested loop join, `t` is the inner (probed) side | Inner table has a good index on the join key and is large; outer side is small | +| `INL_HASH_JOIN(t)` | Index join with hash join on the inner side | Similar to INL_JOIN but inner side needs hash matching | +| `MERGE_JOIN(t1, t2)` | Force sort-merge join | Both sides are already sorted on join key or can be cheaply sorted | +| `SHUFFLE_JOIN(t1, t2)` | Force shuffle hash join (MPP) | TiFlash queries where data needs redistributing across nodes | +| `BROADCAST_JOIN(t1, t2)` | Force broadcast join (MPP) | TiFlash queries where one side is small enough to broadcast | + +### Join order hints + +| Hint | Effect | When to use | +|------|--------|-------------| +| `LEADING(t1, t2, t3)` | Force join order; tables joined left to right | Optimizer picks a bad join order due to cardinality misestimation | +| `STRAIGHT_JOIN()` | Join tables in the order written in FROM clause | Quick override when you know the right order | + +### Subquery hints + +| Hint | Effect | When to use | +|------|--------|-------------| +| `NO_DECORRELATE()` | Keep correlated subquery as-is; do not flatten into a join | Subquery is well-indexed, outer query is selective. See `subquery-optimization.md` | +| `SEMI_JOIN_REWRITE()` | Rewrite semi-join to inner join with deduplication | When semi-join execution is slow and inner side has few duplicates | + +### Index hints + +| Hint | Effect | When to use | +|------|--------|-------------| +| `USE_INDEX(t, idx)` | Force use of a specific index | Optimizer picks wrong index or full table scan | +| `IGNORE_INDEX(t, idx)` | Prevent use of a specific index | A specific index produces a bad plan | +| `USE_INDEX_MERGE(t, idx1, idx2)` | Force index merge (combine multiple indexes) | Query filters on columns from different indexes | +| `ORDER_INDEX(t, idx)` | Force ordered scan on index | Need sorted results and the index provides the order | +| `NO_ORDER_INDEX(t, idx)` | Prevent ordered scan on index | Optimizer incorrectly prefers an ordered scan | + +### Read strategy hints + +| Hint | Effect | When to use | +|------|--------|-------------| +| `READ_FROM_STORAGE(TIKV[t])` | Force reading from TiKV (row store) | OLTP point lookups; avoid accidental TiFlash reads | +| `READ_FROM_STORAGE(TIFLASH[t])` | Force reading from TiFlash (columnar) | Analytical scans on large tables with TiFlash replicas | +| `USE_TOJA(TRUE)` | Enable outer join to anti/semi-join transformation | Specific outer join patterns that can be simplified | +| `USE_TOJA(FALSE)` | Disable the transformation | Transformation produces a worse plan | + +### Aggregation hints + +| Hint | Effect | When to use | +|------|--------|-------------| +| `HASH_AGG()` | Force hash aggregation | Large GROUP BY with many distinct values | +| `STREAM_AGG()` | Force stream aggregation | Data is already sorted on GROUP BY columns | +| `MPP_1PHASE_AGG()` | Force single-phase MPP aggregation | TiFlash; low cardinality GROUP BY | +| `MPP_2PHASE_AGG()` | Force two-phase MPP aggregation | TiFlash; high cardinality GROUP BY | + +### Other hints + +| Hint | Effect | When to use | +|------|--------|-------------| +| `MEMORY_QUOTA(N)` | Set per-query memory limit (e.g., `MEMORY_QUOTA(1 GB)`) | Prevent a single query from using too much memory | +| `MAX_EXECUTION_TIME(N)` | Set per-query timeout in milliseconds | Safety net for queries that might run too long | +| `RESOURCE_GROUP(name)` | Assign query to a resource group | Workload isolation | + +## General guidance + +- **Start without hints.** Fix stats and indexes first. Only add hints when the optimizer consistently picks a wrong plan. +- **Be specific.** Specify table names in hints to avoid ambiguity, especially in multi-join queries. +- **Document why.** Add a SQL comment explaining the reason for the hint so future maintainers don't remove it blindly. +- **Re-evaluate after upgrades.** Optimizer improvements in new TiDB versions may make existing hints unnecessary or counterproductive. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/analyze-jobs-run-too-long-or-too-hot.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/analyze-jobs-run-too-long-or-too-hot.md new file mode 100644 index 0000000..ce71aef --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/analyze-jobs-run-too-long-or-too-hot.md @@ -0,0 +1,20 @@ +# Analyze Jobs Run Too Long or Too Hot + +## User Symptom +- Auto analyze keeps timing out, gets killed, or causes visible CPU pressure on TiDB or TiKV. +- Partitioned tables and very large tables are especially problematic. + +## Investigation Signals +- Analyze stacks show lock-heavy functions such as `mergeglobaltopn` or `queryLockedTables`. +- Large partition counts or many concurrent analyze jobs amplify the problem. +- Some tables have not completed a successful analyze for a very long time. +- TiKV CPU is already high before any attempt to raise scan concurrency. + +## Workaround +- Reduce analyze concurrency, especially for partitioned tables. +- Lower sample rate for the specific table if the business can tolerate it. +- Increase `tidb_max_auto_analyze_time` only when the job is fundamentally correct but too large for the current limit. +- Be careful with `tidb_sysproc_scan_concurrency`; it can worsen hot TiKV nodes. + +## Fixed Version +- Some sub-cases were improved in later releases such as `v7.5`, but this remains a mixed operational and product area rather than one single bug. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/binding-cache-quota-blocks-baselines.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/binding-cache-quota-blocks-baselines.md new file mode 100644 index 0000000..55444bd --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/binding-cache-quota-blocks-baselines.md @@ -0,0 +1,18 @@ +# Binding Cache Quota Blocks Baselines + +## User Symptom +- A binding exists, but the query sometimes behaves as if the binding does not exist. +- Regressed plans appear after upgrade or restart even though the expected bindings were already created. + +## Investigation Signals +- Search TiDB logs for `category=sql-bind`. +- A representative log pattern is the error saying available bindings exceed `tidb_mem_quota_binding_cache`. +- Binding load is not synchronous like stats load. A manual reload may still be required. + +## Workaround +- Increase `tidb_mem_quota_binding_cache`. +- Run `ADMIN RELOAD BINDINGS` after increasing the quota. +- Treat binding reload as a separate recovery step after restart or migration. + +## Fixed Version +- No product fix was recorded in the handbook. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/binding-digest-truncated-by-stmt-summary-limit.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/binding-digest-truncated-by-stmt-summary-limit.md new file mode 100644 index 0000000..f3eae40 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/binding-digest-truncated-by-stmt-summary-limit.md @@ -0,0 +1,16 @@ +# Binding Digest Truncated by Statement Summary Limit + +## User Symptom +- A binding exists but does not take effect for a long SQL statement. + +## Investigation Signals +- `tidb_stmt_summary_max_sql_length` is too small. +- The SQL text used for digest calculation is truncated, so the binding digest no longer matches the real statement shape. + +## Workaround +- Increase `tidb_stmt_summary_max_sql_length`. +- Keep `tidb_use_plan_baseline` enabled while checking whether the binding itself is valid. +- Re-verify the digest after the length limit is raised. + +## Fixed Version +- No product fix was recorded. This is a configuration-side pitfall. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/bindings-for-single-and-multi-value-in-lists-do-not-match.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/bindings-for-single-and-multi-value-in-lists-do-not-match.md new file mode 100644 index 0000000..8ba68ca --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/bindings-for-single-and-multi-value-in-lists-do-not-match.md @@ -0,0 +1,19 @@ +# Bindings for Single- and Multi-Value IN Lists Do Not Match + +## User Symptom +- A binding created for `IN (?)` does not work when the same application later sends `IN (?, ?, ?)` or another multi-value variant. + +## Investigation Signals +- The normalized digest for `IN (?)` is different from the normalized digest for `IN (...)`. +- The binding exists and is valid, but only one of the two SQL shapes can hit it. + +## Workaround +- Create two bindings: + one for `IN (?)` and another for `IN (...)`. +- Verify both normalized forms before closing the case. + +## Fixed Version +- `v7.4` + +## Public References +- https://github.com/pingcap/tidb/pull/44601 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/cost-model-v1-picks-the-wrong-engine.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/cost-model-v1-picks-the-wrong-engine.md new file mode 100644 index 0000000..6ac65bb --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/cost-model-v1-picks-the-wrong-engine.md @@ -0,0 +1,17 @@ +# Cost Model V1 Picks the Wrong Engine + +## User Symptom +- A query that used to run on TiFlash suddenly runs on TiKV and becomes much slower. +- TiKV CPU rises sharply because the new path triggers heavy scan or lookup work. + +## Investigation Signals +- `EXPLAIN ANALYZE` shows `estRows` close to `actRows`, so the main problem is probably not stale stats. +- `EXPLAIN FORMAT='verbose'` or reproduced plans show the engine choice is driven by cost model behavior. +- Slow query history or clinic confirms the optimizer, not a manual binding, selected the worse engine. + +## Workaround +- Prefer cost model v2 on supported releases. +- If switching cost model globally is too risky, bind or hint the intended engine for the problematic SQL and monitor the fallout. + +## Fixed Version +- The handbook treats cost model v2 as the practical fix path on supported versions rather than a one-off patch. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/ddl-owner-memory-leak-from-internal-session-pool.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/ddl-owner-memory-leak-from-internal-session-pool.md new file mode 100644 index 0000000..5b8de2d --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/ddl-owner-memory-leak-from-internal-session-pool.md @@ -0,0 +1,20 @@ +# DDL Owner Memory Leak From the Internal Session Pool + +## User Symptom +- One TiDB node, usually the owner node, has much higher memory than its peers. + +## Investigation Signals +- The memory skew is concentrated on the owner node. +- TTL framework, distributed framework, and auto analyze are all active around the same time. +- The public issue attributes the leak to the internal session pool and session bookkeeping. + +## Workaround +- The handbook does not record a clean runtime workaround. +- If operationally possible, reduce background framework pressure until the node can be upgraded. + +## Fixed Version +- `v8.4.0` + +## Public References +- https://github.com/pingcap/tidb/issues/56934 +- https://github.com/pingcap/tidb/pull/56935 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/empty-range-seeks-cause-tikv-cpu-spikes.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/empty-range-seeks-cause-tikv-cpu-spikes.md new file mode 100644 index 0000000..780cd8d --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/empty-range-seeks-cause-tikv-cpu-spikes.md @@ -0,0 +1,20 @@ +# Empty Range Seeks Cause TiKV CPU Spikes + +## User Symptom +- TiKV CPU is high while the query returns very few rows. +- End-to-end latency is bad even though the result set is tiny. + +## Investigation Signals +- `total_keys` is much larger than `total_process_keys`. +- Seek-heavy metrics are abnormal. +- The pattern often appears with long `IN` lists or index-join inner-side range amplification. +- `tot_wait` and task counts can also show heavy queueing pressure in TiKV. + +## Workaround +- Change the plan with hints or bindings. +- Reduce the size of the `IN` list or split the request. +- Tune `tidb_opt_range_max_size` if range construction is the trigger. +- In some cases, a table scan is less harmful than exploding the number of cop tasks. + +## Fixed Version +- No single fix version was recorded. This is a recurring task-amplification pattern. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/force-init-stats-leaves-pseudo-stats-after-restart.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/force-init-stats-leaves-pseudo-stats-after-restart.md new file mode 100644 index 0000000..943dccc --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/force-init-stats-leaves-pseudo-stats-after-restart.md @@ -0,0 +1,21 @@ +# Force-Init Stats Leaves Pseudo Stats After Restart + +## User Symptom +- After restart, queries keep using `stats:pseudo` for a long time and CPU rises because the wrong index is chosen. + +## Investigation Signals +- The node starts with `force-init-stats=true` and `lite-init-stats=false`. +- `EXPLAIN` or `EXPLAIN ANALYZE` shows `stats:pseudo` on the chosen access path. +- There is no obvious sync-load error in logs, yet statistics still do not arrive for tens of minutes. +- Manual analyze immediately changes the plan and reduces the pressure. + +## Workaround +- Run manual `ANALYZE` for the affected table or partition. +- Avoid routing full traffic to a freshly restarted node before stats warm-up is complete. + +## Fixed Version +- The handbook points to follow-up fixes for missing table IDs during init and for sync-load starvation, but does not record a single release number. + +## Public References +- https://github.com/pingcap/tidb/pull/58280 +- https://github.com/pingcap/tidb/pull/58302 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/gc-stw-causes-simple-query-latency-jitter.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/gc-stw-causes-simple-query-latency-jitter.md new file mode 100644 index 0000000..894b7e6 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/gc-stw-causes-simple-query-latency-jitter.md @@ -0,0 +1,18 @@ +# GC STW Causes Simple Query Latency Jitter + +## User Symptom +- Very simple queries such as `select ... limit 1` show noticeable latency jitter. +- The slowdown does not stay in one obvious execution stage. + +## Investigation Signals +- Slow samples appear across parser, planner, and executor stages instead of one operator hotspot. +- Grafana shows high `GC STW Duration`. +- The pattern is more visible on TiDB nodes with very large schema or stats memory pressure. + +## Workaround +- Increase `tidb_server_memory_limit_gc_trigger`. +- Increase `tidb_gogc_tuner_threshold`. +- Give TiDB more memory if the node is structurally memory-tight. + +## Fixed Version +- No universal fix was recorded in the handbook. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/global-bindings-override-hints.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/global-bindings-override-hints.md new file mode 100644 index 0000000..54980e1 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/global-bindings-override-hints.md @@ -0,0 +1,18 @@ +# Global Bindings Override Hints + +## User Symptom +- A query keeps using the bound plan even after adding `FORCE INDEX`, `IGNORE_INDEX`, or other hints. +- Dropping a session binding does not change the plan. + +## Investigation Signals +- The unexpected plan persists only in environments where a global binding exists. +- Plan replayer or isolated reproduction may not show the global binding, which makes frontline debugging confusing. + +## Workaround +- For session-level validation, disable baselines in that session: + `set @@session.tidb_enable_plan_baselines = 0;` +- Check whether a global binding matches the query before trusting hint tests. +- Remove or adjust the global binding if the binding itself is stale. + +## Fixed Version +- No code fix was identified in the handbook. This is mostly expected behavior plus an operational pitfall. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/huge-in-list-falls-back-to-index-full-scan.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/huge-in-list-falls-back-to-index-full-scan.md new file mode 100644 index 0000000..55b7e34 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/huge-in-list-falls-back-to-index-full-scan.md @@ -0,0 +1,17 @@ +# Huge IN List Falls Back to Index Full Scan + +## User Symptom +- A query with a very large `IN` list unexpectedly falls back to `IndexFullScan` and becomes very slow. + +## Investigation Signals +- The `IN` list is extremely large. +- Decoding the full plan may be hard because the SQL and plan are both huge. +- The likely trigger is the optimizer range-construction limit controlled by `tidb_opt_range_max_size`. + +## Workaround +- Do not send giant `IN` lists directly. +- Split the request into smaller batches or stage the keys in a table and join on it. +- Raise `tidb_opt_range_max_size` only if you explicitly accept the optimizer memory tradeoff. + +## Fixed Version +- No general fix was recorded in the handbook. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/new-tidb-instance-starts-without-stats.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/new-tidb-instance-starts-without-stats.md new file mode 100644 index 0000000..0b16bcc --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/new-tidb-instance-starts-without-stats.md @@ -0,0 +1,20 @@ +# New TiDB Instance Starts Without Stats + +## User Symptom +- A newly scaled-out TiDB instance is healthy enough to serve traffic, but it has no usable stats and may choose bad plans. + +## Investigation Signals +- `stats_meta` or related `stats_xxx` data is missing on the new node. +- TiDB logs show `Out of Memory Quota`, `SortAndSpillDiskAction`, or `Incorrect time value` during stats initialization. +- The issue often appears during `initStatsBuckets`. + +## Workaround +- Temporarily increase `tidb_mem_quota_query` during instance startup so stats initialization can finish. +- If invalid historical time values block stats loading, use a compatible SQL mode such as `ALLOW_INVALID_DATES` long enough to load the old stats. +- After the instance is warm, restore the normal operational settings. + +## Fixed Version +- Invalid time-value loading was fixed in `v6.5.11`. + +## Public References +- https://github.com/pingcap/tidb/issues/56480 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/order-by-limit-picks-the-ordering-index.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/order-by-limit-picks-the-ordering-index.md new file mode 100644 index 0000000..40030b1 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/order-by-limit-picks-the-ordering-index.md @@ -0,0 +1,18 @@ +# ORDER BY LIMIT Picks the Ordering Index + +## User Symptom +- A query with `WHERE` and `ORDER BY ... LIMIT` chooses the ordering index and becomes much slower. +- Index lookup count becomes very large, or the path degenerates into an index full scan. + +## Investigation Signals +- The estimated row count on the ordering index is much smaller than the actual row count. +- The query shape has a real tradeoff between a filtering index and an ordering index. +- In some incidents, a temporary selectivity collapse to zero also pushed the plan toward the ordering index. + +## Workaround +- On supported versions, tune `tidb_opt_ordering_index_selectivity_threshold` and related ordering-index knobs. +- On older releases, prefer SPM or explicit hints. +- If stale stats are involved, run manual analyze first before changing heuristics. + +## Fixed Version +- This is mostly mitigated by configuration and SPM rather than a single universal fix. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/parallel-join-query-oom-on-tidb.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/parallel-join-query-oom-on-tidb.md new file mode 100644 index 0000000..83cff85 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/parallel-join-query-oom-on-tidb.md @@ -0,0 +1,19 @@ +# Parallel Join Query OOM on TiDB + +## User Symptom +- Running one complex query at moderate or high concurrency causes TiDB OOM or very high heap usage. +- Single-query latency may still look acceptable, but concurrency collapses the node. + +## Investigation Signals +- The plan crosses TiFlash and TiKV and builds a large hash join on TiDB. +- `EXPLAIN ANALYZE` memory numbers are not reliable for this pattern. +- Grafana TiDB heap memory and TiFlash logs are more trustworthy than the plan output. + +## Workaround +- Reduce rows as early as possible in the query shape. +- Move selective inner joins earlier. +- Rewrite the query so `TopN` can push down before the long chain of outer joins. +- If high concurrency is required, scale out TiDB or increase memory. + +## Fixed Version +- No general product fix. This is mostly a query-shape and resource-sizing problem. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/analyze-retry-stuck-when-global-time-zone-differs.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/analyze-retry-stuck-when-global-time-zone-differs.md new file mode 100644 index 0000000..5f9d6ed --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/analyze-retry-stuck-when-global-time-zone-differs.md @@ -0,0 +1,19 @@ +# Analyze Retry Stuck When Global Time Zone Differs + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- A failed analyze job does not retry for an unexpectedly long time. + +## Likely Oncall Signals +- Auto analyze for the same table appears stalled after one failure. +- Global `time_zone` differs from `SYSTEM`. +- No obvious new retry job is scheduled even though the table remains unhealthy. + +## Missing Oncall Signals +- The handbook records the issue and public bug, but does not provide concrete log keywords or metric panels that confirm this path quickly. + +## Public References +- https://github.com/pingcap/tidb/issues/65815 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/disable-outer-join-reorder-after-upgrade.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/disable-outer-join-reorder-after-upgrade.md new file mode 100644 index 0000000..c30fe98 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/disable-outer-join-reorder-after-upgrade.md @@ -0,0 +1,25 @@ +# Disable Outer Join Reorder After Upgrade + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- A query changes plan after upgrade and loses the previous join order or index join opportunity. + +## When To Use +- The main visible difference is join order. +- The regression appeared only after upgrade. +- Session-level validation with `set @@tidb_enable_outer_join_reorder = off` recovers the expected plan. + +## When Not To Use +- The incident includes wrong results, executor build errors, or signs of a deeper rewrite bug. +- The old plan was never actually good; the upgrade only made an old schema or index problem visible. +- You have not compared the before/after join tree carefully. + +## Risks +- Turning the flag off is a workload-wide optimizer behavior change if applied globally. +- It may restore one SQL while regressing others that benefit from reorder. + +## Missing Evidence +- The handbook records multiple recoveries with this flag, but the safe rollout boundary is still workload-dependent rather than sharply defined. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/dp-join-reorder-can-trigger-cant-find-column.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/dp-join-reorder-can-trigger-cant-find-column.md new file mode 100644 index 0000000..8fae437 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/dp-join-reorder-can-trigger-cant-find-column.md @@ -0,0 +1,19 @@ +# DP Join Reorder Can Trigger `can't find column` + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- A join query fails with a `can't find column` style error when DP join reorder is enabled. + +## Likely Oncall Signals +- The failure is sensitive to join reorder settings such as `tidb_opt_join_reorder_threshold`. +- Disabling the relevant reorder behavior makes the query succeed. +- The problem appears during optimization or physical-plan generation rather than execution after a long runtime. + +## Missing Oncall Signals +- The handbook cites the public issue but does not preserve a representative stack, exact error text variant, or minimal SQL shape. + +## Public References +- https://github.com/pingcap/tidb/issues/63353 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/enable-advanced-join-hint-only-in-query-scope.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/enable-advanced-join-hint-only-in-query-scope.md new file mode 100644 index 0000000..fa6dd8e --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/enable-advanced-join-hint-only-in-query-scope.md @@ -0,0 +1,23 @@ +# Enable Advanced Join Hint Only in Query Scope + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- `leading()` appears ineffective when it is used together with a join-method hint such as `inl_join`. + +## When To Use +- The cluster originated from an older release line where `tidb_opt_advanced_join_hint` remains off. +- The query needs both join order control and join method control. +- A local `set_var(tidb_opt_advanced_join_hint=1)` hint restores the intended behavior. + +## When Not To Use +- You are considering enabling the variable globally without workload review. +- The query already behaves correctly with simpler hints. + +## Risks +- Global enablement is a real behavior change and can alter historical hint semantics. + +## Missing Evidence +- The handbook clearly describes the mechanism but not a broader rollout playbook, so this remains a local workaround pattern rather than a mature SOP. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/exchange-partition-can-duplicate-rowid-and-skip-updates.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/exchange-partition-can-duplicate-rowid-and-skip-updates.md new file mode 100644 index 0000000..4da7f83 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/exchange-partition-can-duplicate-rowid-and-skip-updates.md @@ -0,0 +1,22 @@ +# Exchange Partition Can Duplicate `_tidb_rowid` and Skip Updates + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- After `EXCHANGE PARTITION`, an `UPDATE` on a nonclustered primary-key table silently misses some rows. + +## Likely Oncall Signals +- The affected table recently participated in `EXCHANGE PARTITION`. +- The table uses a nonclustered primary key. +- The missed rows are consistent with duplicated internal row IDs after the exchange. + +## Missing Oncall Signals +- The handbook gives the root cause and fix PR, but does not provide a quick SQL-side integrity check to confirm the duplicate-rowid condition during incident handling. + +## Fixed Version +- The handbook says the problem affects releases since `v6.5` and cites the fix PR. + +## Public References +- https://github.com/pingcap/tidb/pull/65084 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/force-engine-with-hints-or-spm.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/force-engine-with-hints-or-spm.md new file mode 100644 index 0000000..32236f9 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/force-engine-with-hints-or-spm.md @@ -0,0 +1,25 @@ +# Force Engine With Hints or SPM + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- One SQL is consistently routed to the wrong engine or wrong access path and causes localized latency or storage pressure. + +## When To Use +- The problem is concentrated in a small number of SQLs. +- Engine-specific metrics clearly show one path is materially better than the other. +- You need a contained mitigation without changing global cost behavior. + +## When Not To Use +- The root cause is still unknown. +- A stale global binding may already be masking the real behavior. +- The same SQL shape appears in many variants, making one-off hinting fragile. + +## Risks +- Hints and bindings can fossilize a plan that becomes wrong after later data or schema changes. +- Forcing an engine can hide a product bug that still needs follow-up. + +## Missing Evidence +- The handbook has several successful examples, but usually does not spell out the rollback criteria for removing the hint or binding later. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/manual-analyze-as-emergency-plan-stabilizer.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/manual-analyze-as-emergency-plan-stabilizer.md new file mode 100644 index 0000000..2c881ea --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/manual-analyze-as-emergency-plan-stabilizer.md @@ -0,0 +1,26 @@ +# Manual Analyze as Emergency Plan Stabilizer + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- A query suddenly switches to a much heavier plan after recent data churn. +- TiKV CPU rises and `EXPLAIN ANALYZE` shows a large `estRows` vs `actRows` gap. + +## When To Use +- `show stats_meta` is clearly stale or `modify_count` is high. +- The bad plan appears after bulk insert, delete, or obvious data distribution change. +- Manual `ANALYZE` on the table or partition is operationally affordable. + +## When Not To Use +- `estRows` is already close to `actRows`, which suggests the main problem is not stale stats. +- The incident is primarily an engine-choice, binding, or plan-cache problem. +- The cluster is already resource-starved and an immediate analyze is likely to worsen the outage. + +## Risks +- Analyze itself can be expensive, time out, or even trigger more pressure. +- On very large or skewed tables, a manual analyze may take much longer than expected. + +## Missing Evidence +- The handbook contains many successful manual-analyze recoveries, but most do not spell out the hard threshold for when analyze is the right first action versus when it is only incidental. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/point-get-plan-cache-may-return-zero-or-one-row.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/point-get-plan-cache-may-return-zero-or-one-row.md new file mode 100644 index 0000000..be33712 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/point-get-plan-cache-may-return-zero-or-one-row.md @@ -0,0 +1,20 @@ +# Point Get Plan Cache May Return Zero or One Row + +## Status +- Partial +- Confidence: Low + +## User Symptom +- The same point-get query under plan cache sometimes returns the expected single row and sometimes does not. + +## Likely Oncall Signals +- The query is a point-get shape and uses plan cache. +- The application side insists the same key should always return one row. +- Metadata lock is already enabled, so the old obvious explanation may not apply. + +## Missing Oncall Signals +- The handbook explicitly says the issue is still unrevealed. +- There is no stable reproduction recipe or decisive monitoring/log signature yet. + +## Public References +- https://github.com/pingcap/tidb/issues/64351 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/rc-plus-forced-mpp-on-batch-point-get-can-error.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/rc-plus-forced-mpp-on-batch-point-get-can-error.md new file mode 100644 index 0000000..199a0e9 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/rc-plus-forced-mpp-on-batch-point-get-can-error.md @@ -0,0 +1,19 @@ +# RC Plus Forced MPP on Batch Point Get Can Error + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- An ordinary `SELECT` unexpectedly errors under `READ COMMITTED` when MPP is strongly forced. + +## Likely Oncall Signals +- Transaction isolation is `RC`. +- The query shape needs batch point get or a similar point-access path. +- A TiFlash-forcing hint or binding is present. + +## Missing Oncall Signals +- The handbook records the public issue and cause, but not the exact end-user error text or the fastest way to distinguish it from a normal `SELECT FOR UPDATE` restriction. + +## Public References +- https://github.com/pingcap/tidb/issues/65059 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/read-from-storage-with-virtual-column-cannot-build-plan.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/read-from-storage-with-virtual-column-cannot-build-plan.md new file mode 100644 index 0000000..30d7d9d --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/read-from-storage-with-virtual-column-cannot-build-plan.md @@ -0,0 +1,22 @@ +# `read_from_storage` With a Virtual Column Cannot Build a Physical Plan + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- A query using `read_from_storage(...)` fails with a `can't find physical plan` style error. + +## Likely Oncall Signals +- The query references a virtual column. +- The failure only appears when the storage-engine hint is added. +- Affected versions include `v7.5.6` according to the handbook. + +## Missing Oncall Signals +- The handbook does not preserve a representative failing SQL shape or the exact planner warning text. + +## Fixed Version +- `v8.1.2` + +## Public References +- https://github.com/pingcap/tidb/pull/48041 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/reduce-analyze-concurrency-or-sample-rate.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/reduce-analyze-concurrency-or-sample-rate.md new file mode 100644 index 0000000..08fe3a6 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/reduce-analyze-concurrency-or-sample-rate.md @@ -0,0 +1,24 @@ +# Reduce Analyze Concurrency or Sample Rate + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- Analyze jobs are extremely slow, repeatedly killed, or create visible CPU pressure. + +## When To Use +- Goroutine or profile evidence points to analyze execution itself as the hot path. +- Partition count is high, or the table is very large. +- The cluster is already sensitive to background system load. + +## When Not To Use +- The real incident is stale stats on a hot table and lowering concurrency would only delay recovery. +- The current problem is not analyze runtime but analyze correctness. +- There is no evidence that analyze is the main pressure source. + +## Risks +- Lower concurrency or sample rate improves stability at the cost of slower stats convergence and potentially lower stats quality. + +## Missing Evidence +- The handbook suggests this repeatedly, but does not provide a stable rule for how far to reduce concurrency or sample rate under different table sizes. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/regexp-or-rlike-on-information-schema-can-be-wrong.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/regexp-or-rlike-on-information-schema-can-be-wrong.md new file mode 100644 index 0000000..5041d8d --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/regexp-or-rlike-on-information-schema-can-be-wrong.md @@ -0,0 +1,19 @@ +# `REGEXP` or `RLIKE` on `information_schema` Can Be Wrong + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- A query against certain `information_schema` tables returns wrong results when filtered with `REGEXP` or `RLIKE`. + +## Likely Oncall Signals +- The incorrect result only appears on system tables in `information_schema`. +- Equivalent filters behave more like `LIKE` than true regex matching. +- The bug is very old, so it may surface during compatibility validation rather than after a recent regression. + +## Missing Oncall Signals +- The handbook does not record which exact system tables are known to be affected or a fastest differential query to confirm the bug. + +## Public References +- https://github.com/pingcap/tidb/issues/64249 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/switch-cost-model-v2-to-correct-engine-choice.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/switch-cost-model-v2-to-correct-engine-choice.md new file mode 100644 index 0000000..ca3bb04 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/partial/switch-cost-model-v2-to-correct-engine-choice.md @@ -0,0 +1,25 @@ +# Switch Cost Model V2 to Correct Engine Choice + +## Status +- Partial +- Confidence: Medium + +## User Symptom +- The optimizer chooses TiKV instead of TiFlash, or chooses another clearly worse engine path, and overall latency or TiKV pressure rises. + +## When To Use +- `EXPLAIN ANALYZE` shows `estRows` close to `actRows`, so stale stats are not the main suspect. +- The plan is chosen by the optimizer rather than by binding or manual hint. +- Evidence points to a cost-model v1 misjudgment instead of a missing index. + +## When Not To Use +- The cluster cannot tolerate broad plan churn. +- The problem is isolated to one or two SQLs that can be safely controlled by binding or hint. +- There are still unresolved stats-quality problems. + +## Risks +- Switching cost model is a workload-wide behavior change. +- After the switch, some unrelated SQLs may change plans and need follow-up bindings. + +## Missing Evidence +- The handbook strongly suggests this workaround in several cases, but does not define a crisp decision tree for when the global switch is preferable to local binding. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/plan-cache-memory-growth-from-varying-parameter-shapes.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/plan-cache-memory-growth-from-varying-parameter-shapes.md new file mode 100644 index 0000000..c3d85a8 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/plan-cache-memory-growth-from-varying-parameter-shapes.md @@ -0,0 +1,18 @@ +# Plan Cache Memory Growth From Varying Parameter Shapes + +## User Symptom +- TiDB memory keeps growing while the business workload appears stable. +- In worse cases the node eventually OOMs. + +## Investigation Signals +- Grafana shows plan-cache memory growing together with TiDB heap memory. +- The application uses prepared statements, but the number of `IN` values or placeholder shapes varies widely. +- Prepare statement count or session count also jumps during the same window. + +## Workaround +- Add `/*+ ignore_plan_cache() */` for this unstable query pattern. +- Avoid prepared statements for highly variable `IN` list shapes. +- Audit session lifecycle because plan cache is session-local on each node. + +## Fixed Version +- No universal fix was recorded for this workload pattern. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/plan-cache-skipped-by-short-circuit-predicates.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/plan-cache-skipped-by-short-circuit-predicates.md new file mode 100644 index 0000000..6e03e5e --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/plan-cache-skipped-by-short-circuit-predicates.md @@ -0,0 +1,20 @@ +# Plan Cache Skipped by Short-Circuit Predicates + +## User Symptom +- Prepared statements do not hit the plan cache even though plan cache is enabled. + +## Investigation Signals +- The query contains short-circuit predicate patterns such as: + `c = ? OR ISNULL(?) OR LENGTH(TRIM(?)) < 1` +- Predicate simplification can skip plan cache admission for this shape. + +## Workaround +- Rewrite the predicate to avoid the short-circuit pattern when practical. +- If stable caching is more important than SQL shape preservation, simplify the application-side query builder. +- For known unstable patterns, use `/*+ ignore_plan_cache() */` and treat them as uncached queries. + +## Fixed Version +- The handbook points to a public issue for this pattern but does not record a fixed release. + +## Public References +- https://github.com/pingcap/tidb/issues/63754 diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/stale-stats-after-bulk-data-change.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/stale-stats-after-bulk-data-change.md new file mode 100644 index 0000000..21c37a9 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/stale-stats-after-bulk-data-change.md @@ -0,0 +1,18 @@ +# Stale Stats After Bulk Data Change + +## User Symptom +- A query suddenly switches to a bad index or a much heavier plan. +- TiKV CPU rises after rapid inserts, deletes, or major data distribution changes. + +## Investigation Signals +- `EXPLAIN ANALYZE` shows `estRows` far away from `actRows`. +- `show stats_meta` shows stale `update_time`, large `modify_count`, or a delayed analyze cycle. +- The plan becomes normal again right after manual analyze or after scheduled analyze finally runs. + +## Workaround +- Run manual `ANALYZE` on the affected table or partition. +- Check whether auto analyze thresholds or modification ratios delayed refresh too long. +- If the workload changed together with schema or index design, verify the new predicates still match the intended index. + +## Fixed Version +- No single fix version. This is a recurring operational pattern around stale or delayed statistics. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/storage-iops-saturation-from-read-heavy-query.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/storage-iops-saturation-from-read-heavy-query.md new file mode 100644 index 0000000..8a54bc9 --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/storage-iops-saturation-from-read-heavy-query.md @@ -0,0 +1,18 @@ +# Storage IOPS Saturation From a Read-Heavy Query + +## User Symptom +- Query latency jumps sharply and the storage device reaches its IOPS ceiling. +- TiKV CPU is not proportionally high, so the usual CPU-first workflow misses the culprit. + +## Investigation Signals +- Volume or EBS IOPS becomes saturated. +- `rocksdb` read metrics such as read size or `block_read_bytes` are abnormal. +- TopSQL is often not enough; the decisive clue usually comes from slow query logs and storage metrics together. + +## Workaround +- Temporarily raise the storage IOPS limit if the platform allows it. +- Manually inspect the slow query log to find the read-heavy query. +- Reduce query frequency or change the query shape after the culprit is found. + +## Fixed Version +- No product fix was recorded. The immediate mitigation is workload reduction or capacity adjustment. diff --git a/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/tiflash-cop-overload-without-mpp.md b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/tiflash-cop-overload-without-mpp.md new file mode 100644 index 0000000..5ec4fea --- /dev/null +++ b/skills/tidb-query-tuning/references/optimizer-oncall-experiences-redacted/tiflash-cop-overload-without-mpp.md @@ -0,0 +1,18 @@ +# TiFlash Cop Overload Without MPP + +## User Symptom +- TiFlash CPU or cop latency spikes during fixed time windows. +- High-frequency read traffic becomes unstable even though query latency may not look catastrophic at first. + +## Investigation Signals +- The plan uses `cop[tiflash]` instead of `mpp[tiflash]` or a TiKV path. +- `tidb_enable_tiflash_cop` is enabled. +- TiFlash monitoring shows cop pressure rather than a balanced MPP workload. + +## Workaround +- Prefer MPP for analytical traffic when possible. +- Consider turning off `tidb_enable_tiflash_cop` for workloads that bombard TiFlash with high-frequency cop requests. +- Avoid routing point-like or high-frequency OLTP traffic to TiFlash. + +## Fixed Version +- No general product fix was identified. This is mainly a workload-to-engine mismatch. diff --git a/skills/tidb-query-tuning/references/session-variables.md b/skills/tidb-query-tuning/references/session-variables.md new file mode 100644 index 0000000..da868cb --- /dev/null +++ b/skills/tidb-query-tuning/references/session-variables.md @@ -0,0 +1,104 @@ +# Session Variables for Query Tuning + +TiDB provides session and global variables that influence optimizer behavior. These are useful for workload-wide tuning (as opposed to per-query hints). + +## Syntax + +```sql +-- Session scope (current connection only): +SET tidb_variable_name = value; + +-- Global scope (all new connections): +SET GLOBAL tidb_variable_name = value; + +-- Check current value: +SELECT @@tidb_variable_name; +``` + +## Key optimizer variables + +### Join and execution strategy + +| Variable | Default | Effect | +|----------|---------|--------| +| `tidb_opt_prefer_range_scan` | OFF | When ON, prefer range scans over full table scans even if cost estimate favors the scan. Useful when stats underestimate filter selectivity. | +| `tidb_opt_enable_hash_join` | ON | Master switch for hash join. Set OFF to disable hash joins entirely (rarely needed). | +| `tidb_index_join_batch_size` | 25000 | Batch size for index join outer side. Increasing may help throughput; decreasing reduces memory. | +| `tidb_hash_join_concurrency` | 5 | Number of concurrent hash join workers. Increase for CPU-rich environments with large hash joins. | + +### Subquery and rewrite behavior + +| Variable | Default | Effect | +|----------|---------|--------| +| `tidb_opt_insubq_to_join_and_agg` | ON | Controls whether IN subqueries are rewritten to joins with aggregation. Disable if the rewrite produces worse plans. | +| `tidb_opt_derive_topn` | ON | Derives TopN through outer joins and projections. Usually beneficial. | +| `tidb_opt_enable_late_materialization` | ON | Delays column reads until after filtering. Helps when many rows are filtered out early. | + +### Cost model tuning + +| Variable | Default | Effect | +|----------|---------|--------| +| `tidb_cost_model_version` | 2 | Cost model version. Version 2 (TiDB 6.2+) is more accurate. Use version 2 unless debugging specific regressions. | +| `tidb_opt_seek_factor` | 20 | Cost factor for random I/O seeks. Higher values make the optimizer prefer sequential scans over index lookups. | + +### Statistics behavior + +| Variable | Default | Effect | +|----------|---------|--------| +| `tidb_auto_analyze_ratio` | 0.5 | Threshold of modified rows before auto-ANALYZE triggers. Lower values keep stats fresher but increase analyze frequency. | +| `tidb_stats_load_sync_wait` | 100 (ms) | Wait time for synchronous stats loading during plan optimization. If stats loading is slow, increase this. | +| `tidb_enable_pseudo_for_outdated_stats` | OFF | When ON, uses pseudo statistics for tables with outdated stats. Keep OFF (default) to get warnings about stale stats instead of silently bad plans. | + +### Memory and resource limits + +| Variable | Default | Effect | +|----------|---------|--------| +| `tidb_mem_quota_query` | 1 GB | Memory limit per query. Queries exceeding this are cancelled or spill to disk (depending on the operator). | +| `tidb_max_chunk_size` | 1024 | Max rows per chunk during execution. Larger values improve throughput but use more memory. | +| `tidb_distsql_scan_concurrency` | 15 | Concurrency for distributed scan (TiKV coprocessor requests). Increase for scan-heavy analytical queries. | + +### TiFlash and MPP + +| Variable | Default | Effect | +|----------|---------|--------| +| `tidb_enforce_mpp` | OFF | When ON, forces all supported queries to use MPP execution on TiFlash. Use when you know TiFlash is the right engine. | +| `tidb_allow_mpp` | ON | Master switch for MPP. Must be ON for any TiFlash MPP execution. | +| `tidb_isolation_read_engines` | tikv,tiflash,tidb | Controls which storage engines are used. Set to `tiflash` to force TiFlash reads; `tikv` to force TiKV reads. | + +## Common tuning scenarios + +### Force all reads from TiFlash for an analytical session + +```sql +SET tidb_isolation_read_engines = 'tiflash'; +SET tidb_enforce_mpp = ON; +``` + +### Increase concurrency for a batch job + +```sql +SET tidb_distsql_scan_concurrency = 30; +SET tidb_hash_join_concurrency = 8; +SET tidb_index_join_batch_size = 50000; +``` + +### Debug a plan regression after stats change + +```sql +-- Check stats health first +SHOW STATS_HEALTHY WHERE Db_name = 'mydb'; + +-- If stats are stale, refresh +ANALYZE TABLE mydb.mytable; + +-- If plan is still bad, check cost model +SELECT @@tidb_cost_model_version; +SET tidb_cost_model_version = 2; +``` + +## General guidance + +- Prefer per-query hints over session variables when the fix is query-specific. +- Session variables affect all queries in the connection — use with care in connection-pooled environments. +- Global variable changes take effect for new connections only, not existing ones. +- Document any non-default variable settings in application configuration so they survive restarts. diff --git a/skills/tidb-query-tuning/references/stats-health-and-auto-analyze.md b/skills/tidb-query-tuning/references/stats-health-and-auto-analyze.md new file mode 100644 index 0000000..dde767b --- /dev/null +++ b/skills/tidb-query-tuning/references/stats-health-and-auto-analyze.md @@ -0,0 +1,210 @@ +# Statistics Health and Auto Analyze + +Statistics health determines whether the optimizer sees the real data distribution. When statistics go stale, TiDB can pick the wrong join order, the wrong access path, or the wrong execution engine even if the SQL itself did not change. + +This guide focuses on keeping `AUTO ANALYZE` ahead of statistics decay without overloading TiDB or TiKV. + +## What "healthy stats" means operationally + +`AUTO ANALYZE` is healthy when it can refresh statistics at least as fast as tables become stale. + +Treat the following as the primary signals: + +- `Statistics -> Stats Healthy Distribution` +- `Statistics -> Auto Analyze Duration` +- `Statistics -> Auto Analyze Query Per Minute` +- TiDB CPU and memory usage +- TiKV read pressure during analyze windows + +In practice: + +- Most relevant tables should stay in the high-health buckets (`[80, 100]`). +- The unhealthy bucket should not show a persistent backlog. +- `AUTO ANALYZE` throughput should not spend long periods at zero when there are stale tables waiting. + +## Diagnostic workflow + +### 1. Check whether stats are actually decaying faster than TiDB can repair them + +Start with: + +```sql +SHOW STATS_HEALTHY; +SHOW ANALYZE STATUS; +``` + +Then compare the dashboard trends: + +- If the unhealthy range keeps growing for days, `AUTO ANALYZE` is not keeping up. +- If `Auto Analyze Query Per Minute` frequently drops to zero while stale tables still exist, the trigger threshold may be too conservative. +- If `Auto Analyze Duration` P95 is high, analyze work is too slow and needs capacity or concurrency tuning. + +### 2. Decide whether the bottleneck is triggering or execution + +Use this split: + +- **Too few analyze tasks start**: lower `tidb_auto_analyze_ratio` +- **Tasks start but finish too slowly**: tune analyze concurrency, scan concurrency, or partition/global-stats behavior +- **Tasks are slow only for partitioned tables**: inspect partition merge behavior +- **Analyze harms business traffic**: isolate the stats owner and add TiKV rate limiting before increasing concurrency + +## Primary tuning levers + +### `tidb_auto_analyze_ratio` + +Controls when a table becomes eligible for `AUTO ANALYZE`. + +- Lower value: refresh stats earlier, more frequent analyze +- Higher value: fewer analyze tasks, higher risk of stale stats + +Use a lower ratio only when: + +- Analyze duration is already acceptable +- TiDB and TiKV still have headroom +- The unhealthy bucket is rising because analyze is triggered too late, not because tasks are too slow + +### `tidb_auto_analyze_concurrency` (v8.5+) + +Controls how many analyze jobs can run at the same time. + +Use it when: + +- `AUTO ANALYZE` is clearly throughput-limited +- TiDB CPU is not saturated +- TiKV read pressure is controlled + +Do not increase it blindly. If possible, dedicate one or more TiDB nodes to stats work first. + +### `tidb_auto_build_stats_concurrency` + +Controls parallel stats building. This is especially useful for partitioned tables because multiple partitions can be analyzed in parallel. + +Increase it when: + +- Partition tables dominate the backlog +- TiDB CPU is still underutilized + +### `tidb_build_sampling_stats_concurrency` + +Controls concurrent column sampling. + +Increase it when: + +- Wide tables are slow to analyze +- Column sampling, not partition fan-out, is the bottleneck + +### Effective concurrency rule + +Treat these parameters as multiplicative, not independent: + +- `tidb_auto_build_stats_concurrency` +- `tidb_build_sampling_stats_concurrency` +- `tidb_auto_analyze_concurrency` (v8.5+) + +Keep the effective concurrency within what the TiDB CPU can sustain. Overdriving them creates contention and can make analyze slower, not faster. + +## Partitioned tables + +Partitioned tables often dominate stats maintenance cost. + +### `tidb_auto_analyze_partition_batch_size` + +Increase it when TiDB repeatedly performs many small partition analyze batches and global stats merge becomes the long pole. + +### `tidb_enable_async_merge_global_stats` + +Enable this on versions that support it when global stats merge is slow or memory-heavy. + +Use it when: + +- `SHOW ANALYZE STATUS` shows long-running `merge global stats` +- Partitioned-table analyze causes memory spikes + +### `tidb_merge_partition_stats_concurrency` + +Increase it when the bottleneck is merging global partition stats rather than scanning base data. + +## Scan concurrency and TiKV protection + +### `tidb_analyze_distsql_scan_concurrency` + +This controls the scan concurrency used by analyze. + +Increase it when: + +- Analyze is scan-bound +- TiKV still has spare read bandwidth + +### `tidb_sysproc_scan_concurrency` + +Tune this carefully in large clusters, especially after scale-out. + +General rule: + +- Do not set it higher than the practical TiKV fan-out your cluster can absorb +- Add TiKV background read rate limiting before raising it + +### TiKV guardrail + +Before increasing analyze scan concurrency, rate-limit background reads in TiKV. Otherwise analyze can steal too much read bandwidth from business traffic. + +## Stats-owner isolation + +### `tidb_enable_stats_owner` + +Use dedicated TiDB nodes for stats ownership when available. + +This is valuable when: + +- Analyze work competes with latency-sensitive SQL +- You want to raise analyze concurrency safely +- You need predictable operational isolation + +Do not disable stats ownership on every TiDB node. That is effectively equivalent to disabling cluster-wide auto analyze execution. + +## Common failure patterns + +### Backlog grows, but query-per-minute is often zero + +Likely cause: + +- `tidb_auto_analyze_ratio` is too high + +Action: + +1. Lower `tidb_auto_analyze_ratio` gradually +2. Watch the unhealthy bucket trend +3. Confirm TiDB and TiKV still have headroom + +### Backlog grows and duration P95 is high + +Likely cause: + +- Analyze is under-provisioned or blocked on partition/global-stats work + +Action: + +1. Inspect `SHOW ANALYZE STATUS` +2. Tune partition/global-stats settings first for partition-heavy workloads +3. Increase analyze/build/sampling concurrency gradually +4. Verify TiKV read pressure remains acceptable + +### Analyze hurts online traffic + +Likely cause: + +- Too much scan concurrency or too many parallel jobs without isolation + +Action: + +1. Add TiKV background read rate limiting +2. Move stats ownership to dedicated TiDB nodes if possible +3. Reduce analyze concurrency if TiKV or business latency regresses + +## Recommended operating discipline + +- Change one variable at a time. +- Use dashboard trends, not single snapshots, to judge improvement. +- For partition-heavy clusters, inspect global-stats merge explicitly. +- Prefer dedicated stats-owner TiDB nodes before aggressive concurrency increases. +- Re-check query plans after major stats tuning changes because plan choices can legitimately change. diff --git a/skills/tidb-query-tuning/references/stats-loading-and-startup.md b/skills/tidb-query-tuning/references/stats-loading-and-startup.md new file mode 100644 index 0000000..1c433a0 --- /dev/null +++ b/skills/tidb-query-tuning/references/stats-loading-and-startup.md @@ -0,0 +1,139 @@ +# Statistics Loading and Startup Behavior + +Two stats-loading paths matter during TiDB startup and query compilation: + +- **Init stats**: what TiDB loads during startup before or while serving traffic +- **Sync load**: what TiDB loads on demand during optimization when the needed histogram, TopN, or other stats objects are not fully in memory + +If either path is slow or incomplete, the same SQL can get different plans before and after startup settles. + +## Typical symptoms + +- Query plans change shortly after TiDB restart +- The first wave of queries after restart is slower than steady-state traffic +- `EXPLAIN` shows unstable estimates during warm-up +- `Statistics -> Sync Load QPS` shows many timeouts + +## Init stats evolution by version + +### Before v7.1 + +TiDB loaded all statistics eagerly into memory during startup. On clusters with many tables, this could make startup very slow. + +### v7.1: lite-init-stats + +TiDB started by loading only stats metadata first, then relied on asynchronous loading for the heavier stats objects. + +Trade-off: + +- Faster startup +- Higher risk of plan instability during the warm-up period + +### v6.5.7 and v7.1: force-init-stats + +These versions added a mode that waits for stats initialization to complete before serving traffic. + +Use this when: + +- Startup latency is less important than predictable first-query plans +- Your workload is sensitive to plan changes immediately after restart + +### v7.1.2 and v8.1: concurrently-init-stats + +These versions improved startup by initializing stats concurrently, which reduces cold-start pain on clusters with many tables. + +## Known fixes for startup-time plan instability + +If plans jump after restart and the root cause is stats loading behavior, prefer upgrading to a release that includes the fix: + +- v6.5.10 +- v7.1.6 +- v7.5.2 +- v8.1.0 + +## Sync load + +Sync load was introduced so the optimizer can wait briefly for complete column statistics instead of planning against partial metadata only. + +### `tidb_stats_load_sync_wait` + +This is the maximum time the optimizer waits for sync-loaded stats. + +- Higher value: fewer stats-load timeouts, more stable planning, but longer compile latency +- Lower value: faster compile path, but higher risk of planning with incomplete stats + +Use it when: + +- `Sync Load QPS` shows meaningful timeout volume +- Compilation latency spikes correlate with missing stats objects + +## How to diagnose sync load issues + +### 1. Check the dashboard + +Look at: + +- `Statistics -> Sync Load QPS` + +If you see many timeouts, the optimizer is frequently planning before full stats are ready. + +### 2. Check the TiDB version + +If the cluster is older than the fixed releases for known loading issues, upgrading is usually the cleanest fix. + +### 3. Decide whether the problem is timeout budget or worker capacity + +Use this split: + +- **Timeouts are occasional and scattered**: increase `tidb_stats_load_sync_wait` +- **Timeouts are frequent and continuous**: increase stats-load worker capacity if the version requires manual tuning + +### `stats-load-concurrency` + +On versions where this is manually tuned, raise it when sync load is backlogged. + +### v8.2+ adaptive behavior + +In newer releases, `stats-load-concurrency = 0` enables adaptive concurrency based on CPU cores. + +Implication: + +- If you are already on v8.2+ and still see many sync-load timeouts, first tune `tidb_stats_load_sync_wait` +- If that is not enough, the machine may simply need more CPU or a different deployment shape + +## Operational guidance + +### Prefer upgrade over large startup workarounds + +If you are seeing restart-time plan drift on an affected older release, upgrade before building operational complexity around it. + +### Use force-init-stats only when the first queries matter more than startup latency + +Examples: + +- Stateful services that execute latency-sensitive SQL immediately after failover +- Planned maintenance windows where predictable warm-up is more important than the shortest restart time + +### Watch startup regressions after changing stats policy + +If you change: + +- init-stats behavior +- sync-load wait budget +- stats-load concurrency + +then validate: + +- first-query latency after restart +- plan stability for the top critical queries +- TiDB CPU during warm-up + +## Fast decision matrix + +| Symptom | Most likely first move | +|--------|-------------------------| +| Plans change after restart on older releases | Upgrade to a fixed release | +| Sync Load timeouts are occasional | Increase `tidb_stats_load_sync_wait` | +| Sync Load timeouts are sustained | Increase stats-load worker capacity or machine CPU | +| Startup is too slow with many tables | Prefer concurrent init on newer releases | +| First queries after restart must be stable | Consider force-init-stats mode | diff --git a/skills/tidb-query-tuning/references/stats-version-and-analyze-configuration.md b/skills/tidb-query-tuning/references/stats-version-and-analyze-configuration.md new file mode 100644 index 0000000..4a97308 --- /dev/null +++ b/skills/tidb-query-tuning/references/stats-version-and-analyze-configuration.md @@ -0,0 +1,140 @@ +# Stats Version and Analyze Configuration + +Statistics collection quality is not only about whether `ANALYZE` runs. The stats version, column coverage, partition behavior, and column-type exclusions all shape plan quality and resource usage. + +## Prefer stats version 2 + +TiDB historically had two stats systems: + +- stats v1 +- stats v2 + +Stats v1 is no longer maintained. Since v6.5, the default has moved to stats v2 and new deployments should stay there unless debugging a specific regression. + +### Why stats v2 is preferred + +- Better estimation on large tables and skewed data +- Better handling of out-of-range predicates +- Faster analyze because table and index sampling is more efficient +- Lower collection overhead because not every older stats structure is required + +## `tidb_analyze_version` + +Use: + +```sql +SELECT @@tidb_analyze_version; +SET GLOBAL tidb_analyze_version = 2; +``` + +Operational rule: + +- Keep the cluster on a consistent stats version as much as possible +- Mixed-version stats across tables and indexes make diagnosis harder and can create temporary plan instability during migrations + +## When migrating old stats to v2 + +The safest pattern is: + +1. Switch the analyze version +2. Re-analyze the tables that matter +3. Validate plan quality on critical SQL + +Avoid partial migrations of only a few hot tables unless you have a clear reason to do so. + +## Column coverage policy + +### `tidb_analyze_column_options` + +This variable controls how much column-level stats `ANALYZE` collects. + +Main choices: + +- `PREDICATE`: collect only predicate columns +- `ALL`: collect all columns + +Use `ALL` when: + +- The workload is analytical +- Query shapes vary a lot +- Non-predicate columns still influence plan quality or resource usage indirectly + +Use `PREDICATE` when: + +- The workload is strongly OLTP-oriented +- You need to cut analyze overhead +- Plan regressions are not caused by missing column coverage + +For OLAP-heavy workloads, `PREDICATE` can reduce stats completeness enough to hurt plan quality. + +## OOM-oriented configuration + +### Partitioned-table global stats + +Partitioned-table analyze can cause large memory spikes when global stats merge is expensive. + +Preferred mitigation: + +- Enable `tidb_enable_async_merge_global_stats` on supporting versions + +### `tidb_enable_historical_stats` + +Disable it unless you specifically need historical stats snapshots and have validated the memory cost. + +Why: + +- Historical stats can add meaningful TiDB memory pressure +- Older implementations were especially risky on tables with many stats objects or partitions + +### `tidb_analyze_skip_column_types` + +Use this to skip stats collection for low-value, memory-heavy column types such as: + +- `json` +- `blob` +- `mediumblob` +- `longblob` +- `text` +- `mediumtext` +- `longtext` + +This is appropriate when: + +- These columns are large +- They are not useful for predicate estimation +- Analyze or memory usage is unstable because of them + +## Sampling and build concurrency + +### `tidb_build_sampling_stats_concurrency` + +Use it to accelerate column sampling for wide tables. + +### `tidb_auto_build_stats_concurrency` + +Use it to accelerate stats building, especially on partition-heavy workloads. + +Do not tune them in isolation. Their combined effect can exceed what the host CPU can handle. + +## Version-sensitive guidance + +- v6.5+: stats v2 should be the default path +- v7.5+: async global-stats merge is a key tool for partitioned-table analyze +- v8.5+: concurrent auto analyze makes throughput tuning more practical, but only if TiDB and TiKV still have capacity + +## Practical checklist + +Use this order: + +1. Confirm the cluster is on stats v2 +2. Decide whether `ALL` or `PREDICATE` fits the workload +3. Disable or limit memory-heavy stats features unless they are truly needed +4. Exclude large low-value column types from analyze +5. Tune sampling/build concurrency only after the policy choices above are correct + +## Common anti-patterns + +- Leaving the cluster on old stats behavior after upgrade because "nothing broke yet" +- Using `PREDICATE` for OLAP workloads and then debugging bad plans as if they were optimizer bugs +- Increasing analyze concurrency before excluding memory-heavy column types +- Turning on historical stats without a concrete operational need diff --git a/skills/tidb-query-tuning/references/subquery-optimization.md b/skills/tidb-query-tuning/references/subquery-optimization.md new file mode 100644 index 0000000..274b097 --- /dev/null +++ b/skills/tidb-query-tuning/references/subquery-optimization.md @@ -0,0 +1,277 @@ +# Subquery Optimization + +TiDB's optimizer decorrelates correlated subqueries by default, transforming them into joins. This is usually beneficial but not always. Especially in OLTP workloads. Understanding when to override this behavior is key to tuning subquery performance. + +## How TiDB handles subqueries + +1. **Decorrelation (default):** TiDB rewrites correlated subqueries (EXISTS, IN, scalar) into semi-joins, anti-semi-joins, or left outer joins. This allows the join to be executed with hash join or index join strategies. + +2. **Correlated execution:** The subquery executes once per outer row. This is the fallback when decorrelation is disabled or not possible. + +## NO_DECORRELATE + +### Hint syntax + +Place `NO_DECORRELATE()` inside the subquery's SELECT: + +```sql +SELECT * +FROM orders o +WHERE EXISTS ( + SELECT /*+ NO_DECORRELATE() */ 1 + FROM returns r + WHERE r.order_id = o.id +); +``` + +The hint applies to EXISTS, IN, and scalar correlated subqueries. TiDB will issue a warning if used on a subquery with no correlated columns (since there is nothing to decorrelate). + +**Important:** When `NO_DECORRELATE()` is used on an EXISTS subquery, TiDB automatically injects a `LIMIT 1` into the subquery for early exit — it only needs to find one matching row per outer row. + +### Session variable: `tidb_opt_enable_no_decorrelate_in_select` + +| Property | Value | +|----------|-------| +| Scope | GLOBAL, SESSION | +| Default | OFF | +| Effect | Automatically applies NO_DECORRELATE behavior to correlated subqueries **in the SELECT list** (scalar subqueries) | + +```sql +-- Enable for the session: +SET @@session.tidb_opt_enable_no_decorrelate_in_select = ON; + +-- Enable globally: +SET GLOBAL tidb_opt_enable_no_decorrelate_in_select = ON; +``` + +**What this variable does:** Correlated subqueries in the SELECT list (scalar subqueries) are automatically kept as correlated execution (Apply) instead of being decorrelated into a left outer join. This also enables outer join elimination rules that can remove unnecessary Apply operators when the subquery result is not needed. + +**When to use:** +- Your workload has many scalar subqueries in the SELECT list that are well-indexed on the correlation columns. +- The outer queries are generally selective. +- You want a session-wide or global policy rather than adding hints to every query. + +**When NOT to use:** +- The SELECT-list subqueries reference large tables without indexes on the correlation columns — correlated execution will be slow. +- The outer queries return many rows — each row triggers a subquery probe. + +**Note:** This variable only affects subqueries in the SELECT list. It does NOT affect EXISTS/IN subqueries in the WHERE clause — use the `NO_DECORRELATE()` hint directly for those. + +### When to use NO_DECORRELATE + +Use `NO_DECORRELATE()` (via hint or variable) when **all** of these are true: + +- The **outer query is selective** — it produces few rows (after WHERE filtering), so the subquery executes only a small number of times. +- The **subquery has a good index** on the correlation column(s) — each correlated probe is a fast index lookup. +- The **inner table is large** — decorrelation would build a large hash table or produce a large semi-join build side. + +In this scenario, correlated execution does N fast index lookups (where N is small), while decorrelation builds an expensive hash table over the entire inner table. + +### When NOT to use NO_DECORRELATE + +- The **outer query returns many rows** — correlated execution means many subquery probes, which is slow. +- The **subquery lacks indexes** on the correlation predicate — each probe is a full scan. +- The **inner table is small** — decorrelation into a hash join is cheap and often faster. + +### Example: well-indexed EXISTS with selective outer query + +```sql +-- Outer query filters orders to a small set; subquery probes returns by indexed order_id. +-- Decorrelation would hash-join the entire returns table. Correlated execution is cheaper. +SELECT * +FROM orders o +WHERE o.status = 'disputed' + AND o.created_at > '2024-01-01' + AND EXISTS ( + SELECT /*+ NO_DECORRELATE() */ 1 + FROM returns r + WHERE r.order_id = o.id + AND r.reason = 'defective' + ); +``` + +**Prerequisite:** An index on `returns(order_id, reason)` or at minimum `returns(order_id)`. + +### Example: scalar subquery in SELECT list + +```sql +-- With variable: automatically keeps the correlated scalar subquery as Apply +SET @@session.tidb_opt_enable_no_decorrelate_in_select = ON; + +SELECT + o.id, + o.total, + (SELECT MAX(r.created_at) FROM returns r WHERE r.order_id = o.id) AS last_return_date +FROM orders o +WHERE o.status = 'disputed'; + +-- Or with hint (works regardless of variable): +SELECT + o.id, + o.total, + (SELECT /*+ NO_DECORRELATE() */ MAX(r.created_at) FROM returns r WHERE r.order_id = o.id) AS last_return_date +FROM orders o +WHERE o.status = 'disputed'; +``` + +### Diagnostic: comparing with and without + +```sql +-- Without hint (default decorrelation): +EXPLAIN ANALYZE +SELECT * FROM orders o +WHERE o.status = 'disputed' + AND EXISTS (SELECT 1 FROM returns r WHERE r.order_id = o.id); + +-- With hint (correlated execution): +EXPLAIN ANALYZE +SELECT * FROM orders o +WHERE o.status = 'disputed' + AND EXISTS (SELECT /*+ NO_DECORRELATE() */ 1 FROM returns r WHERE r.order_id = o.id); +``` + +Compare total execution time and the operator tree. Look for: +- **With decorrelation:** `HashJoin` + `TableFullScan` on returns → expensive if returns is large. +- **With NO_DECORRELATE:** `Apply` + `IndexLookUp` on returns → cheap if outer rows are few. Note the automatic `Limit` operator inside the Apply for EXISTS subqueries. + +## SEMI_JOIN_REWRITE + +### What it does + +Rewrites a semi-join into an inner join with aggregation (GROUP BY on the join keys). This transformation is: + +``` +-- Before (semi-join): +SELECT * FROM t WHERE EXISTS (SELECT 1 FROM s WHERE s.a = t.a); + +-- After (inner join + dedup): +SELECT * FROM t JOIN (SELECT a FROM s GROUP BY a) s ON t.a = s.a; +``` + +The aggregation deduplicates the inner side on the join keys, converting the semi-join into a regular inner join. This unlocks additional join strategies and optimizations that may not be available for semi-joins. + +### Hint syntax + +Place `SEMI_JOIN_REWRITE()` inside the EXISTS subquery: + +```sql +SELECT * +FROM orders o +WHERE EXISTS ( + SELECT /*+ SEMI_JOIN_REWRITE() */ 1 + FROM returns r + WHERE r.order_id = o.id +); +``` + +### Session variable: `tidb_opt_enable_semi_join_rewrite` + +| Property | Value | +|----------|-------| +| Scope | GLOBAL, SESSION | +| Default | OFF | +| Effect | Automatically applies SEMI_JOIN_REWRITE for all eligible semi-joins | + +```sql +SET @@session.tidb_opt_enable_semi_join_rewrite = ON; +``` + +### When to use SEMI_JOIN_REWRITE + +- The semi-join execution is slow (e.g., hash semi-join with a large build side). +- The inner side has **few duplicates on the join key** — the GROUP BY dedup cost is low. +- The rewritten inner join can use a better join strategy (e.g., index join) that wasn't available for the semi-join. + +### When NOT to use SEMI_JOIN_REWRITE + +- The inner side has **many duplicates on the join key** — the GROUP BY aggregation becomes expensive. +- The semi-join is already fast (hash semi-join with a small build side). +- The subquery is correlated (Apply) — SEMI_JOIN_REWRITE only works on **non-correlated** semi-joins (i.e., after decorrelation). + +### Limitations + +- **Only works for SemiJoin**, not LeftOuterSemiJoin. TiDB will warn: `SEMI_JOIN_REWRITE() is inapplicable for LeftOuterSemiJoin.` +- **Cannot have left conditions or other conditions** — only equal conditions on the join keys are supported. +- **Conflicts with NO_DECORRELATE:** If both hints are specified on the same subquery, TiDB disables both and warns: `NO_DECORRELATE() and SEMI_JOIN_REWRITE() are in conflict. Both will be ineffective.` This makes sense because NO_DECORRELATE keeps the subquery correlated (Apply), while SEMI_JOIN_REWRITE requires a decorrelated semi-join to rewrite. + +### Example + +```sql +-- The inner side (returns) has unique order_id values, so GROUP BY is cheap. +-- Rewriting to inner join allows the optimizer to use index join on orders. +SELECT * +FROM orders o +WHERE EXISTS ( + SELECT /*+ SEMI_JOIN_REWRITE() */ 1 + FROM returns r + WHERE r.order_id = o.id +); + +-- Equivalent rewritten plan: +-- SELECT * FROM orders o JOIN (SELECT order_id FROM returns GROUP BY order_id) r ON o.id = r.order_id; +``` + +## Semi-join patterns + +When TiDB decorrelates an EXISTS subquery, it typically creates a **semi-join**. The execution strategy for the semi-join matters: + +| Strategy | When it works well | +|----------|-------------------| +| Hash semi-join | Inner side fits in memory; no useful index on join key | +| Index semi-join (INL_JOIN) | Inner side has index on join key; outer side is moderate | +| Merge semi-join | Both sides sorted on join key | + +You can combine hints: + +```sql +-- Force index join for the semi-join after decorrelation +SELECT /*+ INL_JOIN(r) */ * +FROM orders o +WHERE EXISTS (SELECT 1 FROM returns r WHERE r.order_id = o.id); +``` + +## NOT EXISTS / NOT IN (anti-semi-join) + +TiDB converts `NOT EXISTS` and `NOT IN` into anti-semi-joins. The same principles apply: +- Check whether decorrelation helps or hurts. +- `NO_DECORRELATE()` works for `NOT EXISTS` subqueries too. + +**Caution with NOT IN:** `NOT IN` has NULL-handling semantics that can prevent optimization. Prefer `NOT EXISTS` when possible, as it avoids NULL edge cases and gives the optimizer more freedom. + +## Decision flowchart + +``` +Is the subquery correlated? +├── NO → TiDB handles it as an uncorrelated subquery (executed once). TiDB does not currently correlate an uncorrelated subquery. If performance is sub-optimal, a rewrite or other tuning options may be needed. +└── YES → Continue below. + +Where is the subquery? +├── SELECT list (scalar subquery) +│ └── Is the outer query selective AND subquery well-indexed? +│ ├── YES → SET tidb_opt_enable_no_decorrelate_in_select = ON +│ │ (or use NO_DECORRELATE() hint on the subquery) +│ └── NO → Let TiDB decorrelate (default). +│ +└── WHERE clause (EXISTS / IN / NOT EXISTS / NOT IN) + └── Is the outer query selective (few rows after filtering)? + ├── YES → Does the subquery have a good index on correlation columns? + │ ├── YES → Try NO_DECORRELATE(). Verify with EXPLAIN ANALYZE. + │ └── NO → Let TiDB decorrelate (default). Consider adding an index. + └── NO → Let TiDB decorrelate (default). + └── Is the semi-join strategy optimal? + ├── YES → Done. + ├── NO, and inner side has few duplicates on join key + │ → Try SEMI_JOIN_REWRITE() to convert to inner join + dedup. + └── NO → Try INL_JOIN / HASH_JOIN hints on the semi-join tables. +``` + +## Summary: hints vs variables + +| Mechanism | Scope | Applies to | Default | +|-----------|-------|------------|---------| +| `/*+ NO_DECORRELATE() */` | Per-subquery (hint) | EXISTS, IN, scalar subqueries in WHERE or SELECT | — | +| `tidb_opt_enable_no_decorrelate_in_select` | Session / Global | Scalar subqueries in the SELECT list only | OFF | +| `/*+ SEMI_JOIN_REWRITE() */` | Per-subquery (hint) | EXISTS subqueries (non-correlated semi-joins only) | — | +| `tidb_opt_enable_semi_join_rewrite` | Session / Global | All eligible semi-joins | OFF | + +**Key rule:** `NO_DECORRELATE` and `SEMI_JOIN_REWRITE` are mutually exclusive on the same subquery. NO_DECORRELATE keeps the subquery correlated; SEMI_JOIN_REWRITE requires it to be decorrelated first. Specifying both cancels both. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/README.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/README.md new file mode 100644 index 0000000..d7fcdfa --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/README.md @@ -0,0 +1,182 @@ +# TiDB Customer Planner Issues + +- Source Query: `repo:pingcap/tidb is:issue label:"report/customer" label:"sig/planner" created:>=2024-01-01` +- Total Issues: 175 + +| Issue | Status | Type | Linked PR Count | File | +| --- | --- | --- | ---: | --- | +| #66668 | open | type/enhancement | 1 | [issue-66668-planner-can-t-use-primary-as-an-index-in-indexmerge-for-predicate-id-and-a-or-b.md](./issue-66668-planner-can-t-use-primary-as-an-index-in-indexmerge-for-predicate-id-and-a-or-b.md) | +| #66658 | open | type/enhancement | 0 | [issue-66658-planner-decrease-concurrency-batch-size-automatically-for-streaming-plans-with-l.md](./issue-66658-planner-decrease-concurrency-batch-size-automatically-for-streaming-plans-with-l.md) | +| #66623 | closed | type/bug | 3 | [issue-66623-planner-same-plans-with-different-in-list-should-have-the-same-plan-digests.md](./issue-66623-planner-same-plans-with-different-in-list-should-have-the-same-plan-digests.md) | +| #66320 | open | type/bug | 1 | [issue-66320-planner-correlate-a-non-correlated-in-subquery.md](./issue-66320-planner-correlate-a-non-correlated-in-subquery.md) | +| #66203 | open | type/enhancement | 0 | [issue-66203-planner-decrease-the-default-value-64mb-of-tidb-opt-range-max-size.md](./issue-66203-planner-decrease-the-default-value-64mb-of-tidb-opt-range-max-size.md) | +| #65924 | open | type/bug | 2 | [issue-65924-the-estrows-of-equal-condition-is-overestimated-when-the-modify-count-is-zero.md](./issue-65924-the-estrows-of-equal-condition-is-overestimated-when-the-modify-count-is-zero.md) | +| #65916 | open | type/bug | 0 | [issue-65916-explain-for-connection-fails-with-non-prepared-plan-cache-parameterization.md](./issue-65916-explain-for-connection-fails-with-non-prepared-plan-cache-parameterization.md) | +| #65915 | closed | type/bug | 1 | [issue-65915-analyze-statement-freezes-if-saveanalyzeresulttostorage-meets-error-when-analyze.md](./issue-65915-analyze-statement-freezes-if-saveanalyzeresulttostorage-meets-error-when-analyze.md) | +| #65878 | open | type/enhancement | 1 | [issue-65878-planner-plan-cache-support-queries-with-int-col-str-val.md](./issue-65878-planner-plan-cache-support-queries-with-int-col-str-val.md) | +| #65876 | closed | type/enhancement | 1 | [issue-65876-planner-non-prep-plan-cache-support-is-null.md](./issue-65876-planner-non-prep-plan-cache-support-is-null.md) | +| #65867 | closed | type/bug | 1 | [issue-65867-tidb-runtime-error-invalid-memory-address-or-nil-pointer-dereference.md](./issue-65867-tidb-runtime-error-invalid-memory-address-or-nil-pointer-dereference.md) | +| #65838 | open | type/bug | 0 | [issue-65838-inl-join-hint-shows-warning-when-tidb-opt-advanced-join-hint-off-but-silently-fa.md](./issue-65838-inl-join-hint-shows-warning-when-tidb-opt-advanced-join-hint-off-but-silently-fa.md) | +| #65822 | open | type/enhancement | 0 | [issue-65822-support-building-indexmerge-path-and-pushing-down-limit-for-in-expression-in-nes.md](./issue-65822-support-building-indexmerge-path-and-pushing-down-limit-for-in-expression-in-nes.md) | +| #65818 | open | type/bug | 1 | [issue-65818-analyze-cannot-be-cancelled-promptly.md](./issue-65818-analyze-cannot-be-cancelled-promptly.md) | +| #65712 | open | type/enhancement | 1 | [issue-65712-better-plan-for-limit-n-order-by-on-indexmerge-when-not-all-partial-paths-satisf.md](./issue-65712-better-plan-for-limit-n-order-by-on-indexmerge-when-not-all-partial-paths-satisf.md) | +| #65639 | open | type/bug | 0 | [issue-65639-sql-binding-fails-silently-when-sql-is-too-long-binding-created-but-not-visible-.md](./issue-65639-sql-binding-fails-silently-when-sql-is-too-long-binding-created-but-not-visible-.md) | +| #65556 | open | type/bug | 1 | [issue-65556-cost-model-v2-indexhashjoin-cost-underestimation.md](./issue-65556-cost-model-v2-indexhashjoin-cost-underestimation.md) | +| #65489 | closed | type/bug | 3 | [issue-65489-memory-leak-when-analyze-table-is-the-last-statement-in-a-session.md](./issue-65489-memory-leak-when-analyze-table-is-the-last-statement-in-a-session.md) | +| #65381 | closed | type/bug | 3 | [issue-65381-planner-the-optimizer-can-t-use-the-latest-tidb-mem-quota-binding-cache-value-to.md](./issue-65381-planner-the-optimizer-can-t-use-the-latest-tidb-mem-quota-binding-cache-value-to.md) | +| #65208 | open | type/enhancement | 1 | [issue-65208-support-using-ordering-property-to-get-better-join-order.md](./issue-65208-support-using-ordering-property-to-get-better-join-order.md) | +| #65183 | closed | type/bug | 2 | [issue-65183-planner-row-count-incorrectly-forced-to-1-for-indexjoin-when-matching-only-a-pre.md](./issue-65183-planner-row-count-incorrectly-forced-to-1-for-indexjoin-when-matching-only-a-pre.md) | +| #65166 | closed | type/bug | 4 | [issue-65166-planner-outer-join-cannot-be-eliminated-due-to-the-impact-of-the-expression-inde.md](./issue-65166-planner-outer-join-cannot-be-eliminated-due-to-the-impact-of-the-expression-inde.md) | +| #65165 | closed | type/enhancement | 1 | [issue-65165-planner-outer-join-should-be-eliminated-when-join-keys-do-not-use-nulleq-and-can.md](./issue-65165-planner-outer-join-should-be-eliminated-when-join-keys-do-not-use-nulleq-and-can.md) | +| #65100 | closed | type/enhancement | 3 | [issue-65100-planner-unnecessary-overhead-caused-by-the-optimizer-matching-bindings-twice-whe.md](./issue-65100-planner-unnecessary-overhead-caused-by-the-optimizer-matching-bindings-twice-whe.md) | +| #65059 | closed | type/question | 2 | [issue-65059-tiflash-plan-returns-no-access-path-error-in-rc-isolation-level.md](./issue-65059-tiflash-plan-returns-no-access-path-error-in-rc-isolation-level.md) | +| #65032 | closed | type/enhancement | 0 | [issue-65032-planner-set-the-default-value-of-tidb-opt-ordering-index-selectivity-threshold-t.md](./issue-65032-planner-set-the-default-value-of-tidb-opt-ordering-index-selectivity-threshold-t.md) | +| #64809 | open | type/enhancement | 0 | [issue-64809-planner-out-of-range-estimation-enhancement.md](./issue-64809-planner-out-of-range-estimation-enhancement.md) | +| #64712 | open | type/enhancement | 0 | [issue-64712-build-better-range-with-not-expression.md](./issue-64712-build-better-range-with-not-expression.md) | +| #64707 | open | type/enhancement | 0 | [issue-64707-planner-extract-common-leading-predicates-in-dnf-and-push-them-to-indexscan-to-f.md](./issue-64707-planner-extract-common-leading-predicates-in-dnf-and-push-them-to-indexscan-to-f.md) | +| #64649 | open | type/enhancement | 0 | [issue-64649-planner-index-join-cardinality-estimation-issue-because-of-filter-re-estimates-c.md](./issue-64649-planner-index-join-cardinality-estimation-issue-because-of-filter-re-estimates-c.md) | +| #64648 | open | type/enhancement | 0 | [issue-64648-planner-optimizer-uses-total-table-size-for-partition-table-scan-cost-estimation.md](./issue-64648-planner-optimizer-uses-total-table-size-for-partition-table-scan-cost-estimation.md) | +| #64643 | closed | type/bug | 3 | [issue-64643-planner-read-from-storage-hint-table-alias-matching-problem-when-using-crossdb-b.md](./issue-64643-planner-read-from-storage-hint-table-alias-matching-problem-when-using-crossdb-b.md) | +| #64572 | closed | type/bug | 3 | [issue-64572-planner-unexpected-low-out-of-range-estimation-on-index-stats.md](./issue-64572-planner-unexpected-low-out-of-range-estimation-on-index-stats.md) | +| #64463 | open | type/bug | 0 | [issue-64463-tidb-report-error-some-columns-of-limit-69-cannot-find-the-reference-from-its-ch.md](./issue-64463-tidb-report-error-some-columns-of-limit-69-cannot-find-the-reference-from-its-ch.md) | +| #64423 | closed | type/bug | 3 | [issue-64423-analyze-a-small-table-costs-a-lot-of-memory.md](./issue-64423-analyze-a-small-table-costs-a-lot-of-memory.md) | +| #64375 | closed | type/enhancement | 2 | [issue-64375-planner-a-more-fine-grained-track-of-optimization-time.md](./issue-64375-planner-a-more-fine-grained-track-of-optimization-time.md) | +| #64345 | open | type/enhancement | 0 | [issue-64345-recommend-index-statements-may-be-slow-and-can-t-be-killed.md](./issue-64345-recommend-index-statements-may-be-slow-and-can-t-be-killed.md) | +| #64329 | closed | type/bug | 1 | [issue-64329-xxxx-is-null-prevent-the-optimizer-from-choosing-the-best-join-order-and-indexjo.md](./issue-64329-xxxx-is-null-prevent-the-optimizer-from-choosing-the-best-join-order-and-indexjo.md) | +| #64278 | closed | type/bug | 2 | [issue-64278-planner-chooses-wrong-plan-due-to-the-overestimation-of-equal-condition-on-index.md](./issue-64278-planner-chooses-wrong-plan-due-to-the-overestimation-of-equal-condition-on-index.md) | +| #64276 | open | type/bug | 0 | [issue-64276-planner-incorrect-estrows-on-projection-with-pointget.md](./issue-64276-planner-incorrect-estrows-on-projection-with-pointget.md) | +| #64274 | closed | type/bug | 1 | [issue-64274-async-load-may-panic-when-processing-dropped-indexes.md](./issue-64274-async-load-may-panic-when-processing-dropped-indexes.md) | +| #64263 | closed | type/bug | 2 | [issue-64263-plan-cache-can-t-work-after-update-to-v8-5-3.md](./issue-64263-plan-cache-can-t-work-after-update-to-v8-5-3.md) | +| #64250 | closed | type/bug | 4 | [issue-64250-global-binding-cache-may-be-incorrect-after-concurrent-binding-operations-on-mul.md](./issue-64250-global-binding-cache-may-be-incorrect-after-concurrent-binding-operations-on-mul.md) | +| #64217 | open | type/enhancement | 0 | [issue-64217-planner-large-estimation-error-of-probe-side-of-indexjoin-when-table-filter-sele.md](./issue-64217-planner-large-estimation-error-of-probe-side-of-indexjoin-when-table-filter-sele.md) | +| #64205 | open | type/enhancement | 1 | [issue-64205-support-dump-plan-replayer-for-prepare-stmt.md](./issue-64205-support-dump-plan-replayer-for-prepare-stmt.md) | +| #64137 | closed | type/bug | 3 | [issue-64137-planner-consider-empty-histogram-and-newly-emerging-values-in-out-of-range-estim.md](./issue-64137-planner-consider-empty-histogram-and-newly-emerging-values-in-out-of-range-estim.md) | +| #63985 | closed | type/bug | 4 | [issue-63985-plan-changed-unexpectedly-because-expression-index-wasn-t-chosen-by-default.md](./issue-63985-plan-changed-unexpectedly-because-expression-index-wasn-t-chosen-by-default.md) | +| #63805 | open | type/bug | 1 | [issue-63805-prepare-query-with-order-by-clause-does-not-work-with-sql-binding.md](./issue-63805-prepare-query-with-order-by-clause-does-not-work-with-sql-binding.md) | +| #63602 | closed | type/bug | 2 | [issue-63602-statement-with-scalar-subquery-result-wrong.md](./issue-63602-statement-with-scalar-subquery-result-wrong.md) | +| #63573 | open | type/bug | 0 | [issue-63573-true-or-expressions-are-not-always-short-circuit.md](./issue-63573-true-or-expressions-are-not-always-short-circuit.md) | +| #63492 | closed | type/bug | 2 | [issue-63492-hash-aggregation-may-not-be-pushed-down-to-tikv-during-admin-check-table.md](./issue-63492-hash-aggregation-may-not-be-pushed-down-to-tikv-during-admin-check-table.md) | +| #63417 | open | type/enhancement | 0 | [issue-63417-planner-expand-the-dnf-expression-to-construct-precise-scan-ranges.md](./issue-63417-planner-expand-the-dnf-expression-to-construct-precise-scan-ranges.md) | +| #63410 | open | type/bug | 0 | [issue-63410-binding-create-binding-from-plan-digest-can-t-distinguish-different-plans-with-s.md](./issue-63410-binding-create-binding-from-plan-digest-can-t-distinguish-different-plans-with-s.md) | +| #63370 | closed | type/bug | 4 | [issue-63370-limit-doesn-t-pushdown-to-tikv-automatically.md](./issue-63370-limit-doesn-t-pushdown-to-tikv-automatically.md) | +| #63369 | closed | type/bug | 3 | [issue-63369-tiflash-query-result-materialization-can-t-take-effect-after-setting-sql-mode-by.md](./issue-63369-tiflash-query-result-materialization-can-t-take-effect-after-setting-sql-mode-by.md) | +| #63353 | closed | type/bug | 2 | [issue-63353-dp-join-reorder-on-the-graph-that-is-not-totally-connect-may-fail-to-execute.md](./issue-63353-dp-join-reorder-on-the-graph-that-is-not-totally-connect-may-fail-to-execute.md) | +| #63336 | open | type/enhancement | 2 | [issue-63336-planner-a-new-risk-aware-index-selection-heuristic-rule-for-indexlookup-in-skyli.md](./issue-63336-planner-a-new-risk-aware-index-selection-heuristic-rule-for-indexlookup-in-skyli.md) | +| #63314 | closed | type/enhancement | 8 | [issue-63314-planner-join-order-algo-can-t-select-the-lowest-cost-join-order-since-join-keys-.md](./issue-63314-planner-join-order-algo-can-t-select-the-lowest-cost-join-order-since-join-keys-.md) | +| #63290 | closed | type/enhancement | 2 | [issue-63290-planner-the-greedy-join-order-algorithm-can-t-select-the-lowest-cost-plan-since-.md](./issue-63290-planner-the-greedy-join-order-algorithm-can-t-select-the-lowest-cost-plan-since-.md) | +| #63280 | closed | type/enhancement | 2 | [issue-63280-planner-utilize-prefix-of-columns-in-order-limit-clause-to-avoid-full-table-scan.md](./issue-63280-planner-utilize-prefix-of-columns-in-order-limit-clause-to-avoid-full-table-scan.md) | +| #63235 | closed | type/bug | 12 | [issue-63235-with-the-introduction-of-issaferange-cpu-resources-are-consumed-more-heavily.md](./issue-63235-with-the-introduction-of-issaferange-cpu-resources-are-consumed-more-heavily.md) | +| #63075 | closed | type/bug | 5 | [issue-63075-select-1-from-dual-can-t-hit-instance-level-plan-cache.md](./issue-63075-select-1-from-dual-can-t-hit-instance-level-plan-cache.md) | +| #62978 | closed | type/question | 0 | [issue-62978-unexpected-warning-when-using-read-from-storage-hint.md](./issue-62978-unexpected-warning-when-using-read-from-storage-hint.md) | +| #62975 | open | type/enhancement | 0 | [issue-62975-make-it-possible-to-push-agg-down-cross-unionscan.md](./issue-62975-make-it-possible-to-push-agg-down-cross-unionscan.md) | +| #62917 | closed | type/bug | 3 | [issue-62917-planner-mergejoin-cost-formula-should-consider-othercond.md](./issue-62917-planner-mergejoin-cost-formula-should-consider-othercond.md) | +| #62766 | open | type/enhancement | 1 | [issue-62766-same-predicate-estimation-of-two-approximate-stats-is-quite-different.md](./issue-62766-same-predicate-estimation-of-two-approximate-stats-is-quite-different.md) | +| #62722 | closed | type/enhancement | 1 | [issue-62722-enable-set-var-hint-support-for-tidb-allow-tiflash-cop.md](./issue-62722-enable-set-var-hint-support-for-tidb-allow-tiflash-cop.md) | +| #62672 | closed | type/bug | 6 | [issue-62672-only-full-group-by-check-should-be-case-sensitive.md](./issue-62672-only-full-group-by-check-should-be-case-sensitive.md) | +| #62665 | closed | type/bug | 2 | [issue-62665-equal-predicate-at-end-of-range-or-out-of-range-does-not-recognize-modifycount.md](./issue-62665-equal-predicate-at-end-of-range-or-out-of-range-does-not-recognize-modifycount.md) | +| #62556 | closed | type/bug | 2 | [issue-62556-group-by-in-prepared-statement-causes-error-in-tidb-mysql-compatible.md](./issue-62556-group-by-in-prepared-statement-causes-error-in-tidb-mysql-compatible.md) | +| #62551 | closed | type/bug | 4 | [issue-62551-rollup-will-output-more-rows-than-expected.md](./issue-62551-rollup-will-output-more-rows-than-expected.md) | +| #62499 | closed | type/enhancement | 5 | [issue-62499-planner-consider-the-number-of-ranges-and-the-seek-operation-cost-for-indexjoin-.md](./issue-62499-planner-consider-the-number-of-ranges-and-the-seek-operation-cost-for-indexjoin-.md) | +| #62448 | open | type/enhancement | 0 | [issue-62448-tidb-can-t-push-down-join-with-nulleq-key-condition-to-tiflash.md](./issue-62448-tidb-can-t-push-down-join-with-nulleq-key-condition-to-tiflash.md) | +| #62438 | closed | type/bug | 4 | [issue-62438-the-same-count-tablerowidscan-may-generate-different-cost-evaluation.md](./issue-62438-the-same-count-tablerowidscan-may-generate-different-cost-evaluation.md) | +| #62216 | closed | type/bug | 1 | [issue-62216-stats-analyze-panic-with-error-index-out-of-range-10-with-length-10-include-virt.md](./issue-62216-stats-analyze-panic-with-error-index-out-of-range-10-with-length-10-include-virt.md) | +| #62050 | closed | type/bug | 4 | [issue-62050-cannot-simplify-predicate-isnull-not-null-column.md](./issue-62050-cannot-simplify-predicate-isnull-not-null-column.md) | +| #61990 | open | type/question | 0 | [issue-61990-planner-index-predicates-based-on-column-prefix-are-not-pushed-down-to-index-sid.md](./issue-61990-planner-index-predicates-based-on-column-prefix-are-not-pushed-down-to-index-sid.md) | +| #61889 | closed | type/bug | 3 | [issue-61889-tidb-load-binding-timeout-cannot-rightly-init-the-default-value.md](./issue-61889-tidb-load-binding-timeout-cannot-rightly-init-the-default-value.md) | +| #61822 | closed | type/bug | 4 | [issue-61822-ttl-scan-cannot-trigger-sync-async-load-generateruntimefilter.md](./issue-61822-ttl-scan-cannot-trigger-sync-async-load-generateruntimefilter.md) | +| #61785 | closed | type/bug | 4 | [issue-61785-analyze-s-analyzecolumnspushdownv2-create-too-much-concurrency.md](./issue-61785-analyze-s-analyzecolumnspushdownv2-create-too-much-concurrency.md) | +| #61716 | open | type/enhancement | 2 | [issue-61716-avoid-indexmerge-double-read-for-multi-valued-index.md](./issue-61716-avoid-indexmerge-double-read-for-multi-valued-index.md) | +| #61669 | closed | type/bug | 2 | [issue-61669-planner-invalid-column-error-when-building-indexjoin-for-sub-query-with-multiple.md](./issue-61669-planner-invalid-column-error-when-building-indexjoin-for-sub-query-with-multiple.md) | +| #61606 | closed | type/bug | 6 | [issue-61606-tidb-s-analyze-command-cannot-correctly-handle-a-virtual-generated-column-if-it-.md](./issue-61606-tidb-s-analyze-command-cannot-correctly-handle-a-virtual-generated-column-if-it-.md) | +| #61602 | closed | type/bug | 6 | [issue-61602-planner-wrong-join-estimation-depending-on-uninitialized-or-missing-stats.md](./issue-61602-planner-wrong-join-estimation-depending-on-uninitialized-or-missing-stats.md) | +| #61568 | open | type/enhancement | 0 | [issue-61568-planner-remove-duplicated-predicates-in-cnf-or-dnf.md](./issue-61568-planner-remove-duplicated-predicates-in-cnf-or-dnf.md) | +| #61565 | closed | type/bug | 7 | [issue-61565-internal-sql-wrongly-use-get-plan-cost-v1.md](./issue-61565-internal-sql-wrongly-use-get-plan-cost-v1.md) | +| #61501 | open | type/enhancement | 1 | [issue-61501-planner-support-a-hint-or-var-like-index-join-first-to-indicate-the-planner-to-p.md](./issue-61501-planner-support-a-hint-or-var-like-index-join-first-to-indicate-the-planner-to-p.md) | +| #61339 | closed | type/enhancement | 4 | [issue-61339-totalcount-spends-too-much-time.md](./issue-61339-totalcount-spends-too-much-time.md) | +| #61305 | closed | type/bug | 2 | [issue-61305-full-table-scan-hash-join-index-join-when-using-select-distinct-constant-instead.md](./issue-61305-full-table-scan-hash-join-index-join-when-using-select-distinct-constant-instead.md) | +| #61093 | closed | type/enhancement | 2 | [issue-61093-index-merge-alternative-picking-should-consider-the-countafteraccess-more-than-t.md](./issue-61093-index-merge-alternative-picking-should-consider-the-countafteraccess-more-than-t.md) | +| #60985 | closed | type/bug | 5 | [issue-60985-tidb-panic-when-using-plan-replayer-dump-with-a-table-using-foreign-keys.md](./issue-60985-tidb-panic-when-using-plan-replayer-dump-with-a-table-using-foreign-keys.md) | +| #60984 | open | type/bug | 0 | [issue-60984-wrongly-enable-plan-cache-with-tiflash.md](./issue-60984-wrongly-enable-plan-cache-with-tiflash.md) | +| #60692 | closed | type/bug | 2 | [issue-60692-tidb-reports-can-t-find-column-error-from-v8-5-0-when-there-are-multiple-joins-i.md](./issue-60692-tidb-reports-can-t-find-column-error-from-v8-5-0-when-there-are-multiple-joins-i.md) | +| #60655 | closed | type/bug | 4 | [issue-60655-throw-can-t-find-column-xxx-in-schema-column-error-when-execute-sql.md](./issue-60655-throw-can-t-find-column-xxx-in-schema-column-error-when-execute-sql.md) | +| #60455 | open | type/bug | 0 | [issue-60455-unknown-column-with-only-full-group-by.md](./issue-60455-unknown-column-with-only-full-group-by.md) | +| #60380 | open | type/enhancement | 0 | [issue-60380-plan-cache-the-different-in-value-numbers-in-query-cause-too-much-memory-in-tidb.md](./issue-60380-plan-cache-the-different-in-value-numbers-in-query-cause-too-much-memory-in-tidb.md) | +| #60357 | closed | type/enhancement | 8 | [issue-60357-require-an-ability-to-externally-influence-the-optimizer-cost-model.md](./issue-60357-require-an-ability-to-externally-influence-the-optimizer-cost-model.md) | +| #60242 | closed | type/enhancement | 9 | [issue-60242-planner-need-to-ignore-the-ordering-index-path-even-if-the-index-selectivity-is-.md](./issue-60242-planner-need-to-ignore-the-ordering-index-path-even-if-the-index-selectivity-is-.md) | +| #60137 | closed | type/enhancement | 1 | [issue-60137-planner-prevent-pseudo-stats-from-dominating-the-stats-cache.md](./issue-60137-planner-prevent-pseudo-stats-from-dominating-the-stats-cache.md) | +| #60076 | closed | type/enhancement | 5 | [issue-60076-planner-keep-redundant-join-key-predicates-when-propagating-eq-predicates.md](./issue-60076-planner-keep-redundant-join-key-predicates-when-propagating-eq-predicates.md) | +| #60037 | closed | type/bug | 5 | [issue-60037-auto-analyze-merge-global-statistics-failed-when-some-partition-statistics-are-m.md](./issue-60037-auto-analyze-merge-global-statistics-failed-when-some-partition-statistics-are-m.md) | +| #59986 | closed | type/enhancement | 1 | [issue-59986-planner-support-long-binding-statement.md](./issue-59986-planner-support-long-binding-statement.md) | +| #59972 | closed | type/enhancement | 3 | [issue-59972-planner-selection-splits-the-join-group-to-two-parts-forbidding-join-reorder-to-.md](./issue-59972-planner-selection-splits-the-join-group-to-two-parts-forbidding-join-reorder-to-.md) | +| #59902 | closed | type/bug | 2 | [issue-59902-the-estrows-is-wrong-for-inner-operator-of-index-join.md](./issue-59902-the-estrows-is-wrong-for-inner-operator-of-index-join.md) | +| #59869 | open | type/enhancement | 0 | [issue-59869-unionall-should-be-able-to-reserve-the-order-property-comming-from-its-children.md](./issue-59869-unionall-should-be-able-to-reserve-the-order-property-comming-from-its-children.md) | +| #59759 | closed | type/bug | 7 | [issue-59759-data-too-long-field-len-x-error-when-loading-stats-for-a-bit-column.md](./issue-59759-data-too-long-field-len-x-error-when-loading-stats-for-a-bit-column.md) | +| #59652 | open | type/enhancement | 3 | [issue-59652-rewrite-exprs-like-a-10-or-a-20-or-a-30-to-a-in-10-20-30-for-better-expr-evaluat.md](./issue-59652-rewrite-exprs-like-a-10-or-a-20-or-a-30-to-a-in-10-20-30-for-better-expr-evaluat.md) | +| #59560 | closed | type/bug | 7 | [issue-59560-memory-leak-risk-in-session-pool-usage-in-stats-sync-load.md](./issue-59560-memory-leak-risk-in-session-pool-usage-in-stats-sync-load.md) | +| #59524 | closed | type/bug | 8 | [issue-59524-release-the-internal-session-which-may-meet-error-like-54022.md](./issue-59524-release-the-internal-session-which-may-meet-error-like-54022.md) | +| #59427 | closed | type/bug | 3 | [issue-59427-comments-style-optimizer-hints-can-t-work-for-point-get-plan-sometimes.md](./issue-59427-comments-style-optimizer-hints-can-t-work-for-point-get-plan-sometimes.md) | +| #59406 | open | type/bug | 0 | [issue-59406-invalid-setting-for-the-variable-tidb-record-plan-in-slow-log-and-the-config-rec.md](./issue-59406-invalid-setting-for-the-variable-tidb-record-plan-in-slow-log-and-the-config-rec.md) | +| #59211 | closed | type/bug | 8 | [issue-59211-new-only-full-group-by-mode-can-not-tell-special-union-all-case.md](./issue-59211-new-only-full-group-by-mode-can-not-tell-special-union-all-case.md) | +| #58371 | open | type/bug | 0 | [issue-58371-inaccurate-estimated-rows-in-mixed-cnf-and-dnf-conditions.md](./issue-58371-inaccurate-estimated-rows-in-mixed-cnf-and-dnf-conditions.md) | +| #58284 | closed | type/bug | 4 | [issue-58284-cannot-get-right-max-table-id-when-to-init-stats.md](./issue-58284-cannot-get-right-max-table-id-when-to-init-stats.md) | +| #58016 | closed | type/bug | 5 | [issue-58016-planner-tidb-server-can-t-start-due-to-loading-binding-panic.md](./issue-58016-planner-tidb-server-can-t-start-due-to-loading-binding-panic.md) | +| #57992 | open | type/bug | 0 | [issue-57992-planner-the-global-binding-is-not-working-when-using-prepared-statement-with-sel.md](./issue-57992-planner-the-global-binding-is-not-working-when-using-prepared-statement-with-sel.md) | +| #57991 | open | type/enhancement | 0 | [issue-57991-planner-the-estimation-results-of-same-predicate-on-different-index-statistics-a.md](./issue-57991-planner-the-estimation-results-of-same-predicate-on-different-index-statistics-a.md) | +| #57975 | open | type/enhancement | 1 | [issue-57975-planner-assumed-data-distribution-of-out-of-range-estimation-might-not-be-correc.md](./issue-57975-planner-assumed-data-distribution-of-out-of-range-estimation-might-not-be-correc.md) | +| #57948 | open | type/feature-request | 6 | [issue-57948-update-table-statistics-before-publishing-new-schema-version-after-ddl.md](./issue-57948-update-table-statistics-before-publishing-new-schema-version-after-ddl.md) | +| #57631 | closed | type/bug | 5 | [issue-57631-analyze-is-too-slow-when-run-dml-in-million-tables-scenario.md](./issue-57631-analyze-is-too-slow-when-run-dml-in-million-tables-scenario.md) | +| #57544 | closed | type/bug | 4 | [issue-57544-sync-load-cannot-load-common-column-when-to-concurrently-init-stats.md](./issue-57544-sync-load-cannot-load-common-column-when-to-concurrently-init-stats.md) | +| #57427 | open | type/bug | 0 | [issue-57427-error-8066-when-create-binding-for-select-statement-with-high-priority.md](./issue-57427-error-8066-when-create-binding-for-select-statement-with-high-priority.md) | +| #57090 | closed | type/bug | 2 | [issue-57090-optimizer-cannot-correctly-display-the-statistics-info-for-stored-generated-inde.md](./issue-57090-optimizer-cannot-correctly-display-the-statistics-info-for-stored-generated-inde.md) | +| #57085 | closed | type/bug | 2 | [issue-57085-planner-test-testcostmodelver2scanrowsize-tablescan-row-size-is-incorrect.md](./issue-57085-planner-test-testcostmodelver2scanrowsize-tablescan-row-size-is-incorrect.md) | +| #56741 | open | type/enhancement | 0 | [issue-56741-memory-sync-consistency-across-tidb-node-stats-bindings-for-example.md](./issue-56741-memory-sync-consistency-across-tidb-node-stats-bindings-for-example.md) | +| #56724 | open | type/enhancement | 0 | [issue-56724-use-toja-can-not-be-applied-to-a-named-query-block-leading-hint-can-not-be-appli.md](./issue-56724-use-toja-can-not-be-applied-to-a-named-query-block-leading-hint-can-not-be-appli.md) | +| #56683 | open | type/enhancement | 0 | [issue-56683-the-pushed-operator-should-scale-with-regions-index-lookup-will-issue-too-much-c.md](./issue-56683-the-pushed-operator-should-scale-with-regions-index-lookup-will-issue-too-much-c.md) | +| #56615 | open | type/enhancement | 1 | [issue-56615-wrong-original-sql-display-for-bindings.md](./issue-56615-wrong-original-sql-display-for-bindings.md) | +| #56603 | closed | type/bug | 7 | [issue-56603-parsing-cte-error-with-doesn-t-yet-support-order-by-limit-select-distinct-in-rec.md](./issue-56603-parsing-cte-error-with-doesn-t-yet-support-order-by-limit-select-distinct-in-rec.md) | +| #56480 | closed | type/bug | 4 | [issue-56480-tidb-throws-error-when-querying-zero-datetime-with-table-statistics.md](./issue-56480-tidb-throws-error-when-querying-zero-datetime-with-table-statistics.md) | +| #56217 | closed | type/bug | 5 | [issue-56217-read-from-storage-hint-doesn-t-effect-while-plan-choose-the-index-merge-plan.md](./issue-56217-read-from-storage-hint-doesn-t-effect-while-plan-choose-the-index-merge-plan.md) | +| #55807 | open | type/enhancement | 0 | [issue-55807-planner-doesn-t-rewrite-conflict-expressions-to-dual-in-utf8mb4-general-ci-colla.md](./issue-55807-planner-doesn-t-rewrite-conflict-expressions-to-dual-in-utf8mb4-general-ci-colla.md) | +| #55803 | closed | type/bug | 5 | [issue-55803-query-results-of-information-schema-tables-not-accurate.md](./issue-55803-query-results-of-information-schema-tables-not-accurate.md) | +| #55551 | closed | type/bug | 6 | [issue-55551-cte-hang-or-result-wrong-when-multiple-apply-is-used.md](./issue-55551-cte-hang-or-result-wrong-when-multiple-apply-is-used.md) | +| #55191 | closed | type/bug | 0 | [issue-55191-with-the-same-sql-the-results-on-v6-5-are-inconsistent-with-the-results-on-v7-5-.md](./issue-55191-with-the-same-sql-the-results-on-v6-5-are-inconsistent-with-the-results-on-v7-5-.md) | +| #55126 | closed | type/bug | 13 | [issue-55126-planner-tot-col-size-in-mysql-stats-histograms-might-be-a-negative-number.md](./issue-55126-planner-tot-col-size-in-mysql-stats-histograms-might-be-a-negative-number.md) | +| #54968 | open | type/bug | 1 | [issue-54968-tidb-enforce-mpp-doesn-t-work-when-keep-order-is-true-for-table-scan.md](./issue-54968-tidb-enforce-mpp-doesn-t-work-when-keep-order-is-true-for-table-scan.md) | +| #54870 | closed | type/bug | 7 | [issue-54870-planner-can-t-push-down-predicates-with-generated-columns-down-through-unionscan.md](./issue-54870-planner-can-t-push-down-predicates-with-generated-columns-down-through-unionscan.md) | +| #54812 | closed | type/enhancement | 1 | [issue-54812-planner-ndv-scaling-estimation-formula-is-too-naive.md](./issue-54812-planner-ndv-scaling-estimation-formula-is-too-naive.md) | +| #54652 | closed | type/bug | 12 | [issue-54652-planner-txn-select-for-update-using-plan-cache-can-not-lock-data-correctly-in-so.md](./issue-54652-planner-txn-select-for-update-using-plan-cache-can-not-lock-data-correctly-in-so.md) | +| #54454 | open | type/enhancement | 0 | [issue-54454-optimal-range-is-not-built-for-a-b-in-1-1-2-2-and-c-10-and-c-50-on-b-c-index.md](./issue-54454-optimal-range-is-not-built-for-a-b-in-1-1-2-2-and-c-10-and-c-50-on-b-c-index.md) | +| #54213 | closed | type/bug | 7 | [issue-54213-optimizer-access-table-unnecessarily.md](./issue-54213-optimizer-access-table-unnecessarily.md) | +| #54188 | closed | type/enhancement | 4 | [issue-54188-optimizer-does-not-consider-using-ordered-index-scan-when-is-null-is-used.md](./issue-54188-optimizer-does-not-consider-using-ordered-index-scan-when-is-null-is-used.md) | +| #54125 | open | type/enhancement | 0 | [issue-54125-join-reorder-using-greedy-algorithm-use-wrong-rule-for-the-cost-of-limit-operato.md](./issue-54125-join-reorder-using-greedy-algorithm-use-wrong-rule-for-the-cost-of-limit-operato.md) | +| #54064 | closed | type/bug | 5 | [issue-54064-indexmergejoin-get-different-result-when-using-different-charset.md](./issue-54064-indexmergejoin-get-different-result-when-using-different-charset.md) | +| #54022 | closed | type/bug | 8 | [issue-54022-tidb-memory-leak-after-meets-the-fail-to-get-stats-version-for-this-histogram.md](./issue-54022-tidb-memory-leak-after-meets-the-fail-to-get-stats-version-for-this-histogram.md) | +| #53951 | closed | type/bug | 10 | [issue-53951-projection-containing-virtual-columns-should-be-evaluated-before-the-unionscan-d.md](./issue-53951-projection-containing-virtual-columns-should-be-evaluated-before-the-unionscan-d.md) | +| #53834 | closed | type/bug | 11 | [issue-53834-memory-quota-hint-doesn-t-work-well-with-global-binding.md](./issue-53834-memory-quota-hint-doesn-t-work-well-with-global-binding.md) | +| #53652 | closed | type/bug | 5 | [issue-53652-a-new-table-with-fk-will-not-create-stats-meta-by-ddl-event.md](./issue-53652-a-new-table-with-fk-will-not-create-stats-meta-by-ddl-event.md) | +| #53387 | closed | type/bug | 1 | [issue-53387-outer-join-reorder-will-produce-unexpected-join-sequence.md](./issue-53387-outer-join-reorder-will-produce-unexpected-join-sequence.md) | +| #53236 | closed | type/bug | 5 | [issue-53236-failed-to-execute-sql-but-succeed-in-mysql-v8-0.md](./issue-53236-failed-to-execute-sql-but-succeed-in-mysql-v8-0.md) | +| #52831 | open | type/enhancement | 2 | [issue-52831-tracker-for-sync-load.md](./issue-52831-tracker-for-sync-load.md) | +| #52826 | open | type/enhancement | 0 | [issue-52826-recordhistoricalstatstostorage-used-too-much-memory.md](./issue-52826-recordhistoricalstatstostorage-used-too-much-memory.md) | +| #52813 | closed | type/bug | 2 | [issue-52813-no-join-related-hints-don-t-work-in-sql-binding.md](./issue-52813-no-join-related-hints-don-t-work-in-sql-binding.md) | +| #52789 | open | type/enhancement | 0 | [issue-52789-leading-hint-cannot-take-effect-when-the-subquery-is-the-result-of-joins.md](./issue-52789-leading-hint-cannot-take-effect-when-the-subquery-is-the-result-of-joins.md) | +| #52772 | closed | type/bug | 4 | [issue-52772-tidb-panic-when-new-collation-is-disabled-and-two-expressions-use-different-coll.md](./issue-52772-tidb-panic-when-new-collation-is-disabled-and-two-expressions-use-different-coll.md) | +| #52653 | closed | type/bug | 2 | [issue-52653-unexpected-can-t-find-column-error-for-query-with-expression-index.md](./issue-52653-unexpected-can-t-find-column-error-for-query-with-expression-index.md) | +| #52650 | open | type/enhancement | 0 | [issue-52650-correlated-subquery-will-block-the-join-reorder-and-the-correlated-subquery-shou.md](./issue-52650-correlated-subquery-will-block-the-join-reorder-and-the-correlated-subquery-shou.md) | +| #52601 | closed | type/bug | 6 | [issue-52601-negative-waitgroup-counter-during-the-analyze-cause-tidb-is-killed.md](./issue-52601-negative-waitgroup-counter-during-the-analyze-cause-tidb-is-killed.md) | +| #52294 | closed | type/bug | 6 | [issue-52294-wrong-singleflight-implementation-for-stats-syncload.md](./issue-52294-wrong-singleflight-implementation-for-stats-syncload.md) | +| #52036 | open | type/enhancement | 1 | [issue-52036-tidb-opt-ordering-index-selectivity-threshold-should-also-be-valid-for-clustered.md](./issue-52036-tidb-opt-ordering-index-selectivity-threshold-should-also-be-valid-for-clustered.md) | +| #51873 | closed | type/bug | 5 | [issue-51873-inconsistent-query-result-after-changing-the-order-of-cte-statements.md](./issue-51873-inconsistent-query-result-after-changing-the-order-of-cte-statements.md) | +| #51700 | closed | type/bug | 4 | [issue-51700-planner-constant-propagation-supports-sub-queries-in-update-statements.md](./issue-51700-planner-constant-propagation-supports-sub-queries-in-update-statements.md) | +| #51384 | open | type/enhancement | 0 | [issue-51384-join-order-hints-don-t-work-due-to-left-condition-on-left-join-conditions.md](./issue-51384-join-order-hints-don-t-work-due-to-left-condition-on-left-join-conditions.md) | +| #51379 | open | type/enhancement | 0 | [issue-51379-planner-large-cardinality-estimation-error-for-indexjoin-with-indexlookup-when-j.md](./issue-51379-planner-large-cardinality-estimation-error-for-indexjoin-with-indexlookup-when-j.md) | +| #51362 | closed | type/bug | 5 | [issue-51362-planner-invalid-agg-mode-on-mpp-plans-with-sub-queries-accessing-partitioning-ta.md](./issue-51362-planner-invalid-agg-mode-on-mpp-plans-with-sub-queries-accessing-partitioning-ta.md) | +| #51358 | closed | type/bug | 2 | [issue-51358-panic-when-to-disable-lite-init-stats.md](./issue-51358-panic-when-to-disable-lite-init-stats.md) | +| #51116 | closed | type/unknown | 5 | [issue-51116-planner-the-optimizer-can-t-produce-an-optimal-plan-for-queries-with-sub-queries.md](./issue-51116-planner-the-optimizer-can-t-produce-an-optimal-plan-for-queries-with-sub-queries.md) | +| #51098 | open | type/enhancement | 1 | [issue-51098-wrong-estimated-row-count-for-tidb-rowid.md](./issue-51098-wrong-estimated-row-count-for-tidb-rowid.md) | +| #50507 | closed | type/bug | 2 | [issue-50507-planner-set-var-fix-control-is-incompatible-with-binding.md](./issue-50507-planner-set-var-fix-control-is-incompatible-with-binding.md) | +| #50080 | open | type/bug | 8 | [issue-50080-row-count-estimation-on-datetime-type-is-over-estimated-when-time-span-across-ye.md](./issue-50080-row-count-estimation-on-datetime-type-is-over-estimated-when-time-span-across-ye.md) | +| #50068 | closed | type/bug | 4 | [issue-50068-planner-set-var-cannot-take-effect-in-union-all-statements.md](./issue-50068-planner-set-var-cannot-take-effect-in-union-all-statements.md) | +| #50067 | closed | type/bug | 15 | [issue-50067-planner-leading-hint-cannot-take-effect-in-union-all-statements.md](./issue-50067-planner-leading-hint-cannot-take-effect-in-union-all-statements.md) | +| #50012 | closed | type/bug | 4 | [issue-50012-multi-statements-prefetchpointplankeys-panic.md](./issue-50012-multi-statements-prefetchpointplankeys-panic.md) | +| #49986 | closed | type/bug | 4 | [issue-49986-error-1105-hy000-runtime-error-index-out-of-range-0-with-length-0.md](./issue-49986-error-1105-hy000-runtime-error-index-out-of-range-0-with-length-0.md) | diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-49986-error-1105-hy000-runtime-error-index-out-of-range-0-with-length-0.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-49986-error-1105-hy000-runtime-error-index-out-of-range-0-with-length-0.md new file mode 100644 index 0000000..3307f33 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-49986-error-1105-hy000-runtime-error-index-out-of-range-0-with-length-0.md @@ -0,0 +1,59 @@ +# Issue #49986: ERROR 1105 (HY000): runtime error: index out of range [0] with length 0 + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/49986 +- Status: closed +- Type: type/bug +- Created At: 2024-01-02T12:51:20Z +- Closed At: 2024-01-03T07:03:34Z +- Labels: affects-6.5, affects-7.1, affects-7.5, found/gs, report/community, report/customer, severity/moderate, sig/planner, type/bug, type/regression + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Related PR #50002: planner: fix agg push down rule mistake order by item inside agg function + URL: https://github.com/pingcap/tidb/pull/50002 + State: closed + Merged At: 2024-01-03T07:03:32Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/rule_aggregation_push_down.go + - tests/integrationtest/r/expression/issues.result + - tests/integrationtest/t/expression/issues.test + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #50016: planner: fix agg push down rule mistake order by item inside agg function (#50002) + URL: https://github.com/pingcap/tidb/pull/50016 + State: closed + Merged At: 2024-01-25T10:34:51Z + Changed Files Count: 2 + Main Modules: planner/core, statistics/integration_test.go + Sample Changed Files: + - planner/core/rule_aggregation_push_down.go + - statistics/integration_test.go + PR Summary: This is an automated cherry-pick of #50002 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #50017: planner: fix agg push down rule mistake order by item inside agg function (#50002) + URL: https://github.com/pingcap/tidb/pull/50017 + State: closed + Merged At: 2024-01-25T09:25:52Z + Changed Files Count: 2 + Main Modules: planner/core, statistics/integration_test.go + Sample Changed Files: + - planner/core/rule_aggregation_push_down.go + - statistics/integration_test.go + PR Summary: This is an automated cherry-pick of #50002 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #50018: planner: fix agg push down rule mistake order by item inside agg function (#50002) + URL: https://github.com/pingcap/tidb/pull/50018 + State: closed + Merged At: 2024-01-26T05:24:21Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/rule_aggregation_push_down.go + - tests/integrationtest/r/statistics/integration.result + - tests/integrationtest/t/statistics/integration.test + PR Summary: This is an automated cherry-pick of #50002 What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50012-multi-statements-prefetchpointplankeys-panic.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50012-multi-statements-prefetchpointplankeys-panic.md new file mode 100644 index 0000000..c3f2a67 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50012-multi-statements-prefetchpointplankeys-panic.md @@ -0,0 +1,61 @@ +# Issue #50012: Multi-statements `prefetchPointPlanKeys` panic + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/50012 +- Status: closed +- Type: type/bug +- Created At: 2024-01-03T06:12:49Z +- Closed At: 2024-01-03T12:59:04Z +- Labels: affects-6.5, affects-7.1, affects-7.2, affects-7.3, affects-7.4, affects-7.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #50037: server: fix decode issue for prefetch point plan index keys + URL: https://github.com/pingcap/tidb/pull/50037 + State: closed + Merged At: 2024-01-03T12:59:03Z + Changed Files Count: 3 + Main Modules: pkg/server + Sample Changed Files: + - pkg/server/BUILD.bazel + - pkg/server/conn.go + - pkg/server/internal/testserverclient/server_client.go + PR Summary: What problem does this PR solve? Problem Summary: The second parameter of indicates whether index KVs belong to a clustered-index table. Previously, it was always , causing decoding error and panic. What changed and how does it work? Set the parameter properly. +- Fix PR #50046: server: fix decode issue for prefetch point plan index keys (#50037) + URL: https://github.com/pingcap/tidb/pull/50046 + State: closed + Merged At: 2024-01-23T15:03:21Z + Changed Files Count: 3 + Main Modules: server/BUILD.bazel, server/conn.go, server/server_test.go + Sample Changed Files: + - server/BUILD.bazel + - server/conn.go + - server/server_test.go + PR Summary: This is an automated cherry-pick of #50037 What problem does this PR solve? Problem Summary: The second parameter of indicates whether index KVs belong to a clustered-index table. Previously, it was always , causing decoding error and panic. What changed and how does it work? +- Fix PR #50047: server: fix decode issue for prefetch point plan index keys (#50037) + URL: https://github.com/pingcap/tidb/pull/50047 + State: closed + Merged At: 2024-02-27T12:04:59Z + Changed Files Count: 3 + Main Modules: server/BUILD.bazel, server/conn.go, server/server_test.go + Sample Changed Files: + - server/BUILD.bazel + - server/conn.go + - server/server_test.go + PR Summary: This is an automated cherry-pick of #50037 What problem does this PR solve? Problem Summary: The second parameter of indicates whether index KVs belong to a clustered-index table. Previously, it was always , causing decoding error and panic. What changed and how does it work? +- Fix PR #50048: server: fix decode issue for prefetch point plan index keys (#50037) + URL: https://github.com/pingcap/tidb/pull/50048 + State: closed + Merged At: 2024-02-18T08:48:00Z + Changed Files Count: 3 + Main Modules: pkg/server + Sample Changed Files: + - pkg/server/BUILD.bazel + - pkg/server/conn.go + - pkg/server/internal/testserverclient/server_client.go + PR Summary: This is an automated cherry-pick of #50037 What problem does this PR solve? Problem Summary: The second parameter of indicates whether index KVs belong to a clustered-index table. Previously, it was always , causing decoding error and panic. What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50067-planner-leading-hint-cannot-take-effect-in-union-all-statements.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50067-planner-leading-hint-cannot-take-effect-in-union-all-statements.md new file mode 100644 index 0000000..b180668 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50067-planner-leading-hint-cannot-take-effect-in-union-all-statements.md @@ -0,0 +1,249 @@ +# Issue #50067: planner: leading hint cannot take effect in UNION ALL statements + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/50067 +- Status: closed +- Type: type/bug +- Created At: 2024-01-04T05:52:53Z +- Closed At: 2024-01-11T08:11:11Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-7.6, epic/hint, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #50145: planner: fix leading hint cannot take effect in UNION ALL statements + URL: https://github.com/pingcap/tidb/pull/50145 + State: closed + Merged At: 2024-01-10T07:18:24Z + Changed Files Count: 8 + Main Modules: pkg/planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/util/hint/hint.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50257: planner: fix leading hint cannot take effect in UNION ALL statements (#50145) + URL: https://github.com/pingcap/tidb/pull/50257 + State: closed + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/planner/core, planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/util/hint/hint.go + - planner/core/logical_plan_builder.go + - planner/core/planbuilder.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: This is an automated cherry-pick of #50145 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50258: planner: fix leading hint cannot take effect in UNION ALL statements (#50145) + URL: https://github.com/pingcap/tidb/pull/50258 + State: closed + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/planner/core, planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/util/hint/hint.go + - planner/core/logical_plan_builder.go + - planner/core/planbuilder.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: This is an automated cherry-pick of #50145 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50259: planner: fix leading hint cannot take effect in UNION ALL statements (#50145) + URL: https://github.com/pingcap/tidb/pull/50259 + State: closed + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/util/hint/hint.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: This is an automated cherry-pick of #50145 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50260: Revert "planner: fix leading hint cannot take effect in UNION ALL statements" + URL: https://github.com/pingcap/tidb/pull/50260 + State: closed + Merged At: 2024-01-10T08:32:08Z + Changed Files Count: 8 + Main Modules: pkg/planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/util/hint/hint.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: Reverts pingcap/tidb#50145 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50268: Revert "planner: fix leading hint cannot take effect in UNION ALL statements" (#50260) + URL: https://github.com/pingcap/tidb/pull/50268 + State: closed + Merged At: not merged + Changed Files Count: 3 + Main Modules: pkg/planner/core, pkg/util + Sample Changed Files: + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/util/hint/hint.go + PR Summary: This is an automated cherry-pick of #50260 Reverts pingcap/tidb#50145 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50270: Revert "planner: fix leading hint cannot take effect in UNION ALL statements" (#50260) + URL: https://github.com/pingcap/tidb/pull/50270 + State: closed + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/planner/core, planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/util/hint/hint.go + - planner/core/logical_plan_builder.go + - planner/core/planbuilder.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: This is an automated cherry-pick of #50260 Reverts pingcap/tidb#50145 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50272: Revert "planner: fix leading hint cannot take effect in UNION ALL statements" (#50260) + URL: https://github.com/pingcap/tidb/pull/50272 + State: closed + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/planner/core, planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/util/hint/hint.go + - planner/core/logical_plan_builder.go + - planner/core/planbuilder.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: This is an automated cherry-pick of #50260 Reverts pingcap/tidb#50145 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50277: planner: fix leading hint cannot take effect in UNION ALL statements + URL: https://github.com/pingcap/tidb/pull/50277 + State: closed + Merged At: 2024-01-11T08:11:10Z + Changed Files Count: 9 + Main Modules: pkg/planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/planner/core/rule_join_reorder_greedy.go + - pkg/util/hint/hint.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? the root cause is that the will be overwritten by push/pop in the +- Fix PR #50320: planner: fix leading hint cannot take effect in UNION ALL statements (#50277) + URL: https://github.com/pingcap/tidb/pull/50320 + State: closed + Merged At: 2024-01-16T07:15:16Z + Changed Files Count: 5 + Main Modules: planner/core + Sample Changed Files: + - planner/core/logical_plan_builder.go + - planner/core/planbuilder.go + - planner/core/rule_join_reorder_greedy.go + - planner/core/testdata/integration_suite_out.json + - planner/core/testdata/join_reorder_suite_out.json + PR Summary: This is an automated cherry-pick of #50277 What problem does this PR solve? Problem Summary: What changed and how does it work? the root cause is that the will be overwritten by push/pop in the +- Fix PR #50322: planner: fix leading hint cannot take effect in UNION ALL statements (#50277) + URL: https://github.com/pingcap/tidb/pull/50322 + State: closed + Merged At: 2024-01-12T04:10:55Z + Changed Files Count: 5 + Main Modules: planner/core + Sample Changed Files: + - planner/core/casetest/testdata/integration_suite_out.json + - planner/core/casetest/testdata/join_reorder_suite_out.json + - planner/core/logical_plan_builder.go + - planner/core/planbuilder.go + - planner/core/rule_join_reorder_greedy.go + PR Summary: This is an automated cherry-pick of #50277 What problem does this PR solve? Problem Summary: What changed and how does it work? the root cause is that the will be overwritten by push/pop in the +- Fix PR #50323: planner: fix leading hint cannot take effect in UNION ALL statements (#50277) + URL: https://github.com/pingcap/tidb/pull/50323 + State: closed + Merged At: 2024-01-12T04:06:26Z + Changed Files Count: 8 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/planner/core/rule_join_reorder_greedy.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: This is an automated cherry-pick of #50277 What problem does this PR solve? Problem Summary: What changed and how does it work? the root cause is that the will be overwritten by push/pop in the +- Fix PR #50376: planner: fix leading hint cannot take effect in UNION ALL statements (#50277) + URL: https://github.com/pingcap/tidb/pull/50376 + State: closed + Merged At: not merged + Changed Files Count: 9 + Main Modules: pkg/planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/planner/core/rule_join_reorder_greedy.go + - pkg/util/hint/hint.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: This is an automated cherry-pick of #50277 What problem does this PR solve? Problem Summary: What changed and how does it work? the root cause is that the will be overwritten by push/pop in the +- Fix PR #50721: planner: fix leading hint cannot take effect in UNION ALL statements (#50145) + URL: https://github.com/pingcap/tidb/pull/50721 + State: closed + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/util/hint/hint.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: This is an automated cherry-pick of #50145 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50722: planner: fix leading hint cannot take effect in UNION ALL statements (#50277) + URL: https://github.com/pingcap/tidb/pull/50722 + State: closed + Merged At: 2024-01-25T08:37:51Z + Changed Files Count: 9 + Main Modules: pkg/planner/core, pkg/util, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/casetest/hint/testdata/integration_suite_in.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/planner/core/rule_join_reorder_greedy.go + - pkg/util/hint/hint.go + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + PR Summary: This is an automated cherry-pick of #50277 What problem does this PR solve? Problem Summary: What changed and how does it work? the root cause is that the will be overwritten by push/pop in the + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50068-planner-set-var-cannot-take-effect-in-union-all-statements.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50068-planner-set-var-cannot-take-effect-in-union-all-statements.md new file mode 100644 index 0000000..2a9148d --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50068-planner-set-var-cannot-take-effect-in-union-all-statements.md @@ -0,0 +1,57 @@ +# Issue #50068: planner: set_var cannot take effect in UNION ALL statements + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/50068 +- Status: closed +- Type: type/bug +- Created At: 2024-01-04T05:58:51Z +- Closed At: 2024-01-04T14:00:35Z +- Labels: affects-6.5, affects-7.1, affects-7.5, epic/hint, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- The second query's execution time exceeds the limitation. The reason is that UNION ALL's node is not considered when extracting statement hints: ![img_v3_026p_32b954e0-92b3-4a15-8aac-c847fca3026g]() + +## Linked PRs +- Fix PR #50070: bindinfo: extract the table hint from the union statement + URL: https://github.com/pingcap/tidb/pull/50070 + State: closed + Merged At: 2024-01-04T14:00:34Z + Changed Files Count: 2 + Main Modules: pkg/server, pkg/util + Sample Changed Files: + - pkg/server/conn_test.go + - pkg/util/hint/hint_processor.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50095: bindinfo: extract the table hint from the union statement (#50070) + URL: https://github.com/pingcap/tidb/pull/50095 + State: closed + Merged At: 2024-01-23T07:19:50Z + Changed Files Count: 2 + Main Modules: server/conn_test.go, util/hint + Sample Changed Files: + - server/conn_test.go + - util/hint/hint_processor.go + PR Summary: This is an automated cherry-pick of #50070 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50096: bindinfo: extract the table hint from the union statement (#50070) + URL: https://github.com/pingcap/tidb/pull/50096 + State: closed + Merged At: 2024-01-05T07:20:04Z + Changed Files Count: 2 + Main Modules: pkg/server, pkg/util + Sample Changed Files: + - pkg/server/conn_test.go + - pkg/util/hint/hint_processor.go + PR Summary: This is an automated cherry-pick of #50070 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #50097: bindinfo: extract the table hint from the union statement (#50070) + URL: https://github.com/pingcap/tidb/pull/50097 + State: closed + Merged At: 2024-01-23T07:15:21Z + Changed Files Count: 2 + Main Modules: server/conn_test.go, util/hint + Sample Changed Files: + - server/conn_test.go + - util/hint/hint_processor.go + PR Summary: This is an automated cherry-pick of #50070 What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50080-row-count-estimation-on-datetime-type-is-over-estimated-when-time-span-across-ye.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50080-row-count-estimation-on-datetime-type-is-over-estimated-when-time-span-across-ye.md new file mode 100644 index 0000000..63d3cf5 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50080-row-count-estimation-on-datetime-type-is-over-estimated-when-time-span-across-ye.md @@ -0,0 +1,115 @@ +# Issue #50080: row count estimation on DATETIME type is over-estimated when time span across years/months + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/50080 +- Status: open +- Type: type/bug +- Created At: 2024-01-04T08:53:30Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, epic/cardinality-estimation, found/gs, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) [test.zip]() + +## Linked PRs +- Fix PR #58661: planner: fix wrong count when is out of range with datetime + URL: https://github.com/pingcap/tidb/pull/58661 + State: closed + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/planner/core, pkg/planner, pkg/statistics, build/nogo_config.json + Sample Changed Files: + - build/nogo_config.json + - pkg/planner/cardinality/row_count_column.go + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/core/issuetest/BUILD.bazel + - pkg/planner/core/issuetest/planner_issue_test.go + - pkg/planner/core/issuetest/testdata/test.json + - pkg/statistics/histogram.go + - pkg/statistics/scalar.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? First, we found that this issue shows a significant difference between datetime with an index and without an index. Therefore, one of the goals of this PR is to reduce the error between them. The issue actually occurs at this point: we found that datetime is first specially encoded into a uint64, and this uint64 is different from a timestamp — it cannot be directly subtracted. +- Fix PR #55600: Planner: Single col out of range use original type + URL: https://github.com/pingcap/tidb/pull/55600 + State: closed + Merged At: 2025-04-29T16:41:17Z + Changed Files Count: 5 + Main Modules: tests/integrationtest, pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - tests/integrationtest/r/explain_complex.result + - tests/integrationtest/s.zip + - tests/integrationtest/t/explain_complex.test + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Index histograms are stored as type string to support the potential for multiple columns with different types to have their histograms combined into a consistent type. This conversion to string can result in a loss of precision when comparing out of range estimation. This fix will use the original column histogram data for out of range estimation. +- Fix PR #61364: Planner: Single col out of range use original type (#55600) + URL: https://github.com/pingcap/tidb/pull/61364 + State: closed + Merged At: not merged + Changed Files Count: 5 + Main Modules: cmd/explaintest, pkg/planner, tests/integrationtest + Sample Changed Files: + - cmd/explaintest/r/explain_complex.result + - cmd/explaintest/t/explain_complex.test + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - tests/integrationtest/s.zip + PR Summary: This is an automated cherry-pick of #55600 What problem does this PR solve? Problem Summary: What changed and how does it work? Index histograms are stored as type string to support the potential for multiple columns with different types to have their histograms combined into a consistent type. This conversion to string can result in a loss of precision when comparing out of range estimation. This fix will use the original column histogram data for out of range estimation. +- Fix PR #61365: Planner: Single col out of range use original type (#55600) + URL: https://github.com/pingcap/tidb/pull/61365 + State: open + Merged At: not merged + Changed Files Count: 6 + Main Modules: cmd/explaintest, statistics/testdata, statistics/index.go + Sample Changed Files: + - cmd/explaintest/r/explain_complex_update_date.result + - cmd/explaintest/s.zip + - cmd/explaintest/t/explain_complex_update_date.test + - statistics/index.go + - statistics/testdata/stats_suite_out.json + - statistics/testdata/trace_suite_out.json + PR Summary: This is an automated cherry-pick of #55600 What problem does this PR solve? Problem Summary: What changed and how does it work? Index histograms are stored as type string to support the potential for multiple columns with different types to have their histograms combined into a consistent type. This conversion to string can result in a loss of precision when comparing out of range estimation. This fix will use the original column histogram data for out of range estimation. +- Fix PR #62156: Planner: Single col out of range use original type (#55600) + URL: https://github.com/pingcap/tidb/pull/62156 + State: closed + Merged At: 2025-07-07T21:51:27Z + Changed Files Count: 2 + Main Modules: pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + PR Summary: This is an automated cherry-pick of #55600 What problem does this PR solve? Problem Summary: What changed and how does it work? Index histograms are stored as type string to support the potential for multiple columns with different types to have their histograms combined into a consistent type. This conversion to string can result in a loss of precision when comparing out of range estimation. This fix will use the original column histogram data for out of range estimation. +- Fix PR #62259: Planner: Single col out of range use original type (#55600) + URL: https://github.com/pingcap/tidb/pull/62259 + State: closed + Merged At: 2025-07-08T16:08:24Z + Changed Files Count: 2 + Main Modules: pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + PR Summary: This is an automated cherry-pick of #55600 What problem does this PR solve? Problem Summary: What changed and how does it work? Index histograms are stored as type string to support the potential for multiple columns with different types to have their histograms combined into a consistent type. This conversion to string can result in a loss of precision when comparing out of range estimation. This fix will use the original column histogram data for out of range estimation. +- Fix PR #62260: Planner: Single col out of range use original type (#55600) + URL: https://github.com/pingcap/tidb/pull/62260 + State: closed + Merged At: 2025-07-08T23:45:36Z + Changed Files Count: 2 + Main Modules: pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + PR Summary: This is an automated cherry-pick of #55600 What problem does this PR solve? Problem Summary: What changed and how does it work? Index histograms are stored as type string to support the potential for multiple columns with different types to have their histograms combined into a consistent type. This conversion to string can result in a loss of precision when comparing out of range estimation. This fix will use the original column histogram data for out of range estimation. +- Fix PR #65667: Planner: Single col out of range use original type (#55600) + URL: https://github.com/pingcap/tidb/pull/65667 + State: closed + Merged At: 2026-01-20T09:46:14Z + Changed Files Count: 5 + Main Modules: pkg/privilege, pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - pkg/privilege/privileges/ldap/test/ca.crt + - pkg/privilege/privileges/ldap/test/ldap.crt + - pkg/privilege/privileges/ldap/test/ldap.key + PR Summary: Manual cherry-pick of #62259 What problem does this PR solve? Problem Summary: What changed and how does it work? Index histograms are stored as type string to support the potential for multiple columns with different types to have their histograms combined into a consistent type. This conversion to string can result in a loss of precision when comparing out of range estimation. This fix will use the original column histogram data for out of range estimation. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50507-planner-set-var-fix-control-is-incompatible-with-binding.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50507-planner-set-var-fix-control-is-incompatible-with-binding.md new file mode 100644 index 0000000..a138054 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-50507-planner-set-var-fix-control-is-incompatible-with-binding.md @@ -0,0 +1,41 @@ +# Issue #50507: planner: set_var(fix_control) is incompatible with binding + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/50507 +- Status: closed +- Type: type/bug +- Created At: 2024-01-17T08:44:55Z +- Closed At: 2024-01-18T03:52:48Z +- Labels: affects-7.5, epic/sql-plan-management, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #50515: parser: restore set_var value to string instead of plain text + URL: https://github.com/pingcap/tidb/pull/50515 + State: closed + Merged At: 2024-01-18T03:52:47Z + Changed Files Count: 3 + Main Modules: pkg/parser, pkg/bindinfo + Sample Changed Files: + - pkg/bindinfo/global_handle_test.go + - pkg/parser/ast/misc.go + - pkg/parser/parser_test.go + PR Summary: What problem does this PR solve? Problem Summary: parser: restore set_var value to string instead of plain text What changed and how does it work? +- Fix PR #50536: parser: restore set_var value to string instead of plain text (#50515) + URL: https://github.com/pingcap/tidb/pull/50536 + State: closed + Merged At: 2024-02-20T13:03:56Z + Changed Files Count: 5 + Main Modules: pkg/bindinfo, pkg/parser + Sample Changed Files: + - pkg/bindinfo/BUILD.bazel + - pkg/bindinfo/handle_test.go + - pkg/bindinfo/tests/bind_test.go + - pkg/parser/ast/misc.go + - pkg/parser/parser_test.go + PR Summary: This is an automated cherry-pick of #50515 What problem does this PR solve? Problem Summary: parser: restore set_var value to string instead of plain text What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51098-wrong-estimated-row-count-for-tidb-rowid.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51098-wrong-estimated-row-count-for-tidb-rowid.md new file mode 100644 index 0000000..8f2f555 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51098-wrong-estimated-row-count-for-tidb-rowid.md @@ -0,0 +1,29 @@ +# Issue #51098: Wrong estimated row count for _tidb_rowid + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/51098 +- Status: open +- Type: type/enhancement +- Created At: 2024-02-13T10:10:51Z +- Labels: affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, component/statistics, epic/cardinality-estimation, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- The is close to the total row count of the table. Thus, the cost of is equal to and the optimizer sometimes chooses , which is very slow and even causes TiKV OOM. + +## Linked PRs +- Fix PR #51111: *: fix wrong estimated row count for _tidb_rowid + URL: https://github.com/pingcap/tidb/pull/51111 + State: closed + Merged At: not merged + Changed Files Count: 5 + Main Modules: pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/row_count_column.go + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/core/casetest/planstats/plan_stats_test.go + - pkg/planner/core/internal/base/plan.go + - pkg/planner/core/planbuilder.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51116-planner-the-optimizer-can-t-produce-an-optimal-plan-for-queries-with-sub-queries.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51116-planner-the-optimizer-can-t-produce-an-optimal-plan-for-queries-with-sub-queries.md new file mode 100644 index 0000000..46ab3f7 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51116-planner-the-optimizer-can-t-produce-an-optimal-plan-for-queries-with-sub-queries.md @@ -0,0 +1,81 @@ +# Issue #51116: planner: the optimizer can't produce an optimal plan for queries with sub-queries in select list since it always decorrelate sub-queries + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/51116 +- Status: closed +- Type: type/unknown +- Created At: 2024-02-18T08:46:57Z +- Closed At: 2025-09-03T18:34:01Z +- Labels: affects-8.5, report/customer, severity/moderate, sig/planner + +## Customer-Facing Phenomenon +- A better plan could be (how Oracle executes it): + +## Linked PRs +- Fix PR #63204: planner: Add variable for no_decorrelate in select list + URL: https://github.com/pingcap/tidb/pull/63204 + State: closed + Merged At: 2025-09-03T18:34:00Z + Changed Files Count: 9 + Main Modules: pkg/session, pkg/planner/core, pkg/bindinfo + Sample Changed Files: + - pkg/bindinfo/binding_plan_generation.go + - pkg/bindinfo/binding_plan_generation_test.go + - pkg/planner/core/expression_rewriter.go + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/setvar_affect.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Customer requirement to allow decorrelation of subqueries in the select list. Added a variable to control this behavior. +- Fix PR #63273: planner: no_decorrelate triggered by cost factors + URL: https://github.com/pingcap/tidb/pull/63273 + State: closed + Merged At: not merged + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/expression_rewriter.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Prototype PR for test scenario. Not planned for integration into master branch. The planned PR for master branch is: +- Fix PR #63287: planner: allow correlated exists subqueries to early-out | tidb-test=pr/2593 + URL: https://github.com/pingcap/tidb/pull/63287 + State: closed + Merged At: 2025-09-10T20:50:27Z + Changed Files Count: 4 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/expression_rewriter.go + - pkg/planner/core/logical_plans_test.go + - tests/integrationtest/r/cte.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Enhancement to add LIMIT 1 to EXISTS correlated subqueries (where NO_DECORRELATE hint exists) if no existing LIMIT applies within the subquery. +- Fix PR #63541: planner: Add variable for no_decorrelate in select list (#63204) + URL: https://github.com/pingcap/tidb/pull/63541 + State: closed + Merged At: 2025-09-22T03:46:24Z + Changed Files Count: 7 + Main Modules: pkg/session, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/expression_rewriter.go + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/planbuilder.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/setvar_affect.go + - pkg/sessionctx/variable/sysvar.go + - pkg/sessionctx/variable/tidb_vars.go + PR Summary: This is an automated cherry-pick of #63204 What problem does this PR solve? Problem Summary: What changed and how does it work? Customer requirement to allow decorrelation of subqueries in the select list. Added a variable to control this behavior. +- Fix PR #63634: planner: allow correlated exists subqueries to early-out (#63287) | tidb-test=pr/2611 + URL: https://github.com/pingcap/tidb/pull/63634 + State: closed + Merged At: 2025-09-23T01:12:22Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/expression_rewriter.go + - tests/integrationtest/r/cte.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + PR Summary: This is an automated cherry-pick of #63287 What problem does this PR solve? Problem Summary: What changed and how does it work? Enhancement to add LIMIT 1 to EXISTS correlated subqueries (where NO_DECORRELATE hint exists) if no existing LIMIT applies within the subquery. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51358-panic-when-to-disable-lite-init-stats.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51358-panic-when-to-disable-lite-init-stats.md new file mode 100644 index 0000000..6e3a237 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51358-panic-when-to-disable-lite-init-stats.md @@ -0,0 +1,43 @@ +# Issue #51358: panic when to disable lite-init-stats + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/51358 +- Status: closed +- Type: type/bug +- Created At: 2024-02-27T08:46:01Z +- Closed At: 2024-02-27T10:47:32Z +- Labels: affects-7.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #51357: statistics: fix panic when to enable force-init-stats + URL: https://github.com/pingcap/tidb/pull/51357 + State: closed + Merged At: 2024-02-27T10:47:08Z + Changed Files Count: 5 + Main Modules: pkg/statistics + Sample Changed Files: + - pkg/statistics/handle/bootstrap.go + - pkg/statistics/handle/cache/BUILD.bazel + - pkg/statistics/handle/cache/statscache.go + - pkg/statistics/handle/handletest/statstest/BUILD.bazel + - pkg/statistics/handle/handletest/statstest/stats_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? If we get nil from cache, we shouldn't do anything. +- Fix PR #51369: statistics: fix panic when to enable force-init-stats (#51357) + URL: https://github.com/pingcap/tidb/pull/51369 + State: closed + Merged At: 2024-02-27T11:23:02Z + Changed Files Count: 5 + Main Modules: pkg/statistics + Sample Changed Files: + - pkg/statistics/handle/bootstrap.go + - pkg/statistics/handle/cache/BUILD.bazel + - pkg/statistics/handle/cache/statscache.go + - pkg/statistics/handle/handletest/statstest/BUILD.bazel + - pkg/statistics/handle/handletest/statstest/stats_test.go + PR Summary: This is an automated cherry-pick of #51357 What problem does this PR solve? Problem Summary: What changed and how does it work? If we get nil from cache, we shouldn't do anything. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51362-planner-invalid-agg-mode-on-mpp-plans-with-sub-queries-accessing-partitioning-ta.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51362-planner-invalid-agg-mode-on-mpp-plans-with-sub-queries-accessing-partitioning-ta.md new file mode 100644 index 0000000..d46e8f8 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51362-planner-invalid-agg-mode-on-mpp-plans-with-sub-queries-accessing-partitioning-ta.md @@ -0,0 +1,75 @@ +# Issue #51362: planner: invalid Agg mode on MPP plans with sub-queries accessing partitioning tables under static partition mode + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/51362 +- Status: closed +- Type: type/bug +- Created At: 2024-02-27T09:17:30Z +- Closed At: 2024-05-22T06:46:18Z +- Labels: affects-7.5, affects-8.1, report/customer, severity/moderate, sig/planner, type/bug, type/regression + +## Customer-Facing Phenomenon +- ERROR 1105 (HY000): other error for mpp stream: Code: 0, e.displayText() = DB::TiFlashException: Different aggregation mode detected, e.what() = DB::TiFlashException, + +## Linked PRs +- Related PR #53455: planner: fix mpp final agg couldn't co-exist with other non-final mode + URL: https://github.com/pingcap/tidb/pull/53455 + State: closed + Merged At: 2024-05-22T06:46:17Z + Changed Files Count: 5 + Main Modules: pkg/expression, pkg/planner/core + Sample Changed Files: + - pkg/expression/aggregation/BUILD.bazel + - pkg/expression/aggregation/aggregation.go + - pkg/expression/aggregation/explain.go + - pkg/planner/core/enforce_mpp_test.go + - pkg/planner/core/exhaust_physical_plans.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #53475: planner: fix mpp final agg couldn't co-exist with other non-final mode (#53455) + URL: https://github.com/pingcap/tidb/pull/53475 + State: closed + Merged At: 2024-05-28T07:28:21Z + Changed Files Count: 5 + Main Modules: pkg/expression, pkg/planner/core + Sample Changed Files: + - pkg/expression/aggregation/BUILD.bazel + - pkg/expression/aggregation/aggregation.go + - pkg/expression/aggregation/explain.go + - pkg/planner/core/enforce_mpp_test.go + - pkg/planner/core/exhaust_physical_plans.go + PR Summary: This is an automated cherry-pick of #53455 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #53476: planner: fix mpp final agg couldn't co-exist with other non-final mode (#53455) + URL: https://github.com/pingcap/tidb/pull/53476 + State: closed + Merged At: 2024-08-01T02:26:52Z + Changed Files Count: 6 + Main Modules: pkg/expression, pkg/executor, pkg/planner/core + Sample Changed Files: + - pkg/executor/test/tiflashtest/BUILD.bazel + - pkg/executor/test/tiflashtest/tiflash_test.go + - pkg/expression/aggregation/BUILD.bazel + - pkg/expression/aggregation/aggregation.go + - pkg/expression/aggregation/explain.go + - pkg/planner/core/exhaust_physical_plans.go + PR Summary: This is an automated cherry-pick of #53455 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #53484: planner: make TestMppAggShouldAlignFinalMode test case stable + URL: https://github.com/pingcap/tidb/pull/53484 + State: closed + Merged At: 2024-05-22T09:23:48Z + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/enforce_mpp_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #54666: planner: make TestMppAggShouldAlignFinalMode test case stable (#53484) + URL: https://github.com/pingcap/tidb/pull/54666 + State: closed + Merged At: not merged + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/enforce_mpp_test.go + PR Summary: This is an automated cherry-pick of #53484 What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51379-planner-large-cardinality-estimation-error-for-indexjoin-with-indexlookup-when-j.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51379-planner-large-cardinality-estimation-error-for-indexjoin-with-indexlookup-when-j.md new file mode 100644 index 0000000..5ddb369 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51379-planner-large-cardinality-estimation-error-for-indexjoin-with-indexlookup-when-j.md @@ -0,0 +1,17 @@ +# Issue #51379: planner: large cardinality estimation error for IndexJoin with IndexLookup when join estimation closes to zero + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/51379 +- Status: open +- Type: type/enhancement +- Created At: 2024-02-28T04:07:52Z +- Labels: affects-8.1, epic/cardinality-estimation, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- The estimation error of is huge, vs , which might lead to suboptimal plans. + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51384-join-order-hints-don-t-work-due-to-left-condition-on-left-join-conditions.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51384-join-order-hints-don-t-work-due-to-left-condition-on-left-join-conditions.md new file mode 100644 index 0000000..f80f187 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51384-join-order-hints-don-t-work-due-to-left-condition-on-left-join-conditions.md @@ -0,0 +1,17 @@ +# Issue #51384: Join order hints don't work due to left condition on left join conditions + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/51384 +- Status: open +- Type: type/enhancement +- Created At: 2024-02-28T05:24:30Z +- Labels: epic/hint, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51700-planner-constant-propagation-supports-sub-queries-in-update-statements.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51700-planner-constant-propagation-supports-sub-queries-in-update-statements.md new file mode 100644 index 0000000..c91b83c --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51700-planner-constant-propagation-supports-sub-queries-in-update-statements.md @@ -0,0 +1,68 @@ +# Issue #51700: planner: constant propagation supports sub-queries in update statements + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/51700 +- Status: closed +- Type: type/bug +- Created At: 2024-03-12T08:28:46Z +- Closed At: 2025-07-18T10:27:50Z +- Labels: affects-7.5, affects-8.1, affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- The constant is only propagated to t2 side for select sql + +## Linked PRs +- Fix PR #61909: planner: constant propagation supports more join type in the logical plan builder + URL: https://github.com/pingcap/tidb/pull/61909 + State: closed + Merged At: 2025-07-18T10:27:49Z + Changed Files Count: 5 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/issuetest/planner_issue_test.go + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/operator/logicalop/logical_join.go + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + - tests/integrationtest/t/planner/core/rule_constant_propagation.test + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? The flag for the constant propagation rule was set too conservatively before, which led to the inability to execute constant propagation in some places. +- Fix PR #62518: planner: constant propagation supports more join type in the logical plan builder (#61909) + URL: https://github.com/pingcap/tidb/pull/62518 + State: closed + Merged At: 2025-07-21T16:07:30Z + Changed Files Count: 6 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/issuetest/planner_issue_test.go + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/operator/logicalop/logical_join.go + - tests/integrationtest/r/cte.result + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + - tests/integrationtest/t/planner/core/rule_constant_propagation.test + PR Summary: This is an automated cherry-pick of #61909 What problem does this PR solve? Problem Summary: What changed and how does it work? The flag for the constant propagation rule was set too conservatively before, which led to the inability to execute constant propagation in some places. +- Fix PR #62534: planner: constant propagation supports more join type in the logical plan builder (#61909) + URL: https://github.com/pingcap/tidb/pull/62534 + State: closed + Merged At: 2025-07-24T16:28:20Z + Changed Files Count: 4 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/logical_plan_builder.go + - tests/integrationtest/r/cte.result + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + - tests/integrationtest/t/planner/core/rule_constant_propagation.test + PR Summary: This is an automated cherry-pick of #61909 What problem does this PR solve? Problem Summary: What changed and how does it work? The flag for the constant propagation rule was set too conservatively before, which led to the inability to execute constant propagation in some places. +- Fix PR #62535: planner: constant propagation supports more join type in the logical plan builder (#61909) + URL: https://github.com/pingcap/tidb/pull/62535 + State: closed + Merged At: 2025-07-21T15:07:27Z + Changed Files Count: 4 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/logical_plan_builder.go + - tests/integrationtest/r/cte.result + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + - tests/integrationtest/t/planner/core/rule_constant_propagation.test + PR Summary: This is an automated cherry-pick of #61909 What problem does this PR solve? Problem Summary: What changed and how does it work? The flag for the constant propagation rule was set too conservatively before, which led to the inability to execute constant propagation in some places. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51873-inconsistent-query-result-after-changing-the-order-of-cte-statements.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51873-inconsistent-query-result-after-changing-the-order-of-cte-statements.md new file mode 100644 index 0000000..3880519 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-51873-inconsistent-query-result-after-changing-the-order-of-cte-statements.md @@ -0,0 +1,74 @@ +# Issue #51873: Inconsistent query result after changing the order of cte statements + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/51873 +- Status: closed +- Type: type/bug +- Created At: 2024-03-19T02:27:22Z +- Closed At: 2024-03-19T13:38:14Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.0, report/customer, severity/critical, sig/planner, type/bug, type/regression + +## Customer-Facing Phenomenon +- Query A: Empty set, 1 warning (0.00 sec) Query B: +---------------+ + +## Linked PRs +- Fix PR #51903: planner: apply rule_partition_pruning when optimizing CTE under static mode + URL: https://github.com/pingcap/tidb/pull/51903 + State: closed + Merged At: 2024-03-19T13:38:13Z + Changed Files Count: 3 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/BUILD.bazel + - pkg/planner/core/casetest/integration_test.go + - pkg/planner/core/optimizer.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #52058: planner: apply rule_partition_pruning when optimizing CTE under static mode (#51903) + URL: https://github.com/pingcap/tidb/pull/52058 + State: closed + Merged At: 2024-04-01T06:24:16Z + Changed Files Count: 3 + Main Modules: planner/core, executor/sample_test.go + Sample Changed Files: + - executor/sample_test.go + - planner/core/integration_test.go + - planner/core/optimizer.go + PR Summary: This is an automated cherry-pick of #51903 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #52148: planner: apply rule_partition_pruning when optimizing CTE under static mode (#51903) + URL: https://github.com/pingcap/tidb/pull/52148 + State: closed + Merged At: 2024-03-27T09:32:18Z + Changed Files Count: 3 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/BUILD.bazel + - pkg/planner/core/casetest/integration_test.go + - pkg/planner/core/optimizer.go + PR Summary: This is an automated cherry-pick of #51903 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #52269: planner: apply rule_partition_pruning when optimizing CTE under static mode (#51903) + URL: https://github.com/pingcap/tidb/pull/52269 + State: closed + Merged At: 2024-04-18T13:36:08Z + Changed Files Count: 3 + Main Modules: planner/core, executor/sample_test.go + Sample Changed Files: + - executor/sample_test.go + - planner/core/integration_test.go + - planner/core/optimizer.go + PR Summary: This is an automated cherry-pick of #51903 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #52270: planner: apply rule_partition_pruning when optimizing CTE under static mode (#51903) + URL: https://github.com/pingcap/tidb/pull/52270 + State: closed + Merged At: 2024-05-21T07:58:17Z + Changed Files Count: 5 + Main Modules: pkg/planner/core, pkg/executor + Sample Changed Files: + - pkg/executor/sample_test.go + - pkg/planner/core/casetest/BUILD.bazel + - pkg/planner/core/casetest/integration_test.go + - pkg/planner/core/integration_test.go + - pkg/planner/core/optimizer.go + PR Summary: This is an automated cherry-pick of #51903 What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-52036-tidb-opt-ordering-index-selectivity-threshold-should-also-be-valid-for-clustered.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-52036-tidb-opt-ordering-index-selectivity-threshold-should-also-be-valid-for-clustered.md new file mode 100644 index 0000000..2865796 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-52036-tidb-opt-ordering-index-selectivity-threshold-should-also-be-valid-for-clustered.md @@ -0,0 +1,44 @@ +# Issue #52036: `tidb_opt_ordering_index_selectivity_threshold` should also be valid for clustered index + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/52036 +- Status: open +- Type: type/enhancement +- Created At: 2024-03-22T10:50:09Z +- Labels: affects-7.1, affects-7.5, affects-8.1, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- If a table has a clustered index as an option of order index, the choice between the clustered index and another index using selection condition is not controlled by the variable . It'd be better to support this case, to avoid choosing bad plan for the same case (but for clustered index) as described in + +## Linked PRs +- Related PR #61506: planner: apply ordering ratio to tablerangescan | tidb-test=pr/2527 + URL: https://github.com/pingcap/tidb/pull/61506 + State: closed + Merged At: 2025-07-26T16:49:45Z + Changed Files Count: 22 + Main Modules: tests/integrationtest, pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/cross_estimation.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_out.json + - pkg/planner/core/casetest/pushdown/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/plan_normalized_suite_out.json + - pkg/planner/core/casetest/testdata/plan_normalized_suite_xut.json + - pkg/planner/core/find_best_task.go + - pkg/planner/core/plan_cost_ver2.go + - tests/integrationtest/r/access_path_selection.result + - tests/integrationtest/r/executor/issues.result + - tests/integrationtest/r/explain_easy.result + - tests/integrationtest/r/explain_easy_stats.result + - tests/integrationtest/r/planner/cascades/integration.result + - tests/integrationtest/r/planner/core/casetest/hint/hint.result + - tests/integrationtest/r/planner/core/casetest/index/index.result + - tests/integrationtest/r/planner/core/casetest/integration.result + - tests/integrationtest/r/planner/core/casetest/partition/integration_partition.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/casetest/rule/rule_result_reorder.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Similar to other PRs that have utilized tidb_opt_ordering_index_selectivity_ratio - this PR will allow this variable to adjust the estimated rows for an ordering plan estimate for the following: table range scan - where the table range scan provides order, + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-52294-wrong-singleflight-implementation-for-stats-syncload.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-52294-wrong-singleflight-implementation-for-stats-syncload.md new file mode 100644 index 0000000..a617272 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-52294-wrong-singleflight-implementation-for-stats-syncload.md @@ -0,0 +1,95 @@ +# Issue #52294: wrong singleflight implementation for stats' syncload + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/52294 +- Status: closed +- Type: type/bug +- Created At: 2024-04-01T13:03:40Z +- Closed At: 2024-04-03T10:16:56Z +- Labels: affects-6.5, affects-7.1, affects-7.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- ![image]() We use the for sync load's duplicate task detection. ![image]() It will return a boolean value directly after it finds duplicates. ![image]() And we can see that the also returns directly by writing an ok result to the . + +## Linked PRs +- Fix PR #52301: statistics: fix wrong singleflight implementation for stats' syncload + URL: https://github.com/pingcap/tidb/pull/52301 + State: closed + Merged At: 2024-04-03T10:16:55Z + Changed Files Count: 5 + Main Modules: pkg/statistics, pkg/parser + Sample Changed Files: + - pkg/parser/model/model.go + - pkg/statistics/handle/syncload/stats_syncload.go + - pkg/statistics/handle/syncload/stats_syncload_test.go + - pkg/statistics/handle/types/BUILD.bazel + - pkg/statistics/handle/types/interfaces.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? the problem has been described in the issue. we refactor the singleflight using the golang official library. +- Fix PR #52374: statistics: fix wrong singleflight implementation for stats' syncload (#52301) + URL: https://github.com/pingcap/tidb/pull/52374 + State: closed + Merged At: 2024-04-07T08:51:20Z + Changed Files Count: 5 + Main Modules: pkg/statistics, pkg/parser + Sample Changed Files: + - pkg/parser/model/model.go + - pkg/statistics/handle/BUILD.bazel + - pkg/statistics/handle/handle.go + - pkg/statistics/handle/handle_hist.go + - pkg/statistics/handle/handle_hist_test.go + PR Summary: This is an automated cherry-pick of #52301 What problem does this PR solve? Problem Summary: What changed and how does it work? the problem has been described in the issue. we refactor the singleflight using the golang official library. +- Fix PR #52382: statistics: fix wrong singleflight implementation for stats' syncload (#52301) + URL: https://github.com/pingcap/tidb/pull/52382 + State: closed + Merged At: 2024-08-27T13:48:17Z + Changed Files Count: 5 + Main Modules: statistics/handle, parser/model + Sample Changed Files: + - parser/model/model.go + - statistics/handle/BUILD.bazel + - statistics/handle/handle.go + - statistics/handle/handle_hist.go + - statistics/handle/handle_hist_test.go + PR Summary: This is an automated cherry-pick of #52301 What problem does this PR solve? Problem Summary: What changed and how does it work? the problem has been described in the issue. we refactor the singleflight using the golang official library. +- Fix PR #52383: statistics: fix wrong singleflight implementation for stats' syncload (#52301) + URL: https://github.com/pingcap/tidb/pull/52383 + State: closed + Merged At: 2024-06-04T14:10:28Z + Changed Files Count: 5 + Main Modules: statistics/handle, parser/model + Sample Changed Files: + - parser/model/model.go + - statistics/handle/BUILD.bazel + - statistics/handle/handle.go + - statistics/handle/handle_hist.go + - statistics/handle/handle_hist_test.go + PR Summary: This is an automated cherry-pick of #52301 What problem does this PR solve? Problem Summary: What changed and how does it work? the problem has been described in the issue. we refactor the singleflight using the golang official library. +- Related PR #54339: statistics: fix wrong singleflight implementation for stats' syncload + URL: https://github.com/pingcap/tidb/pull/54339 + State: closed + Merged At: 2024-07-01T06:10:26Z + Changed Files Count: 5 + Main Modules: statistics/handle, parser/model + Sample Changed Files: + - parser/model/model.go + - statistics/handle/BUILD.bazel + - statistics/handle/handle.go + - statistics/handle/handle_hist.go + - statistics/handle/handle_hist_test.go + PR Summary: close pingcap/tidb#52294 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #55726: statistics: fix wrong singleflight implementation for stats' syncload(#52301) (#52382) + URL: https://github.com/pingcap/tidb/pull/55726 + State: closed + Merged At: 2024-08-28T12:27:18Z + Changed Files Count: 5 + Main Modules: statistics/handle, parser/model + Sample Changed Files: + - parser/model/model.go + - statistics/handle/BUILD.bazel + - statistics/handle/handle.go + - statistics/handle/handle_hist.go + - statistics/handle/handle_hist_test.go + PR Summary: cherry pick #52382 close pingcap/tidb#52294 What problem does this PR solve? Problem Summary: + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-52601-negative-waitgroup-counter-during-the-analyze-cause-tidb-is-killed.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-52601-negative-waitgroup-counter-during-the-analyze-cause-tidb-is-killed.md new file mode 100644 index 0000000..d2859cf --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-52601-negative-waitgroup-counter-during-the-analyze-cause-tidb-is-killed.md @@ -0,0 +1,81 @@ +# Issue #52601: Negative WaitGroup counter during the analyze cause TiDB is killed + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/52601 +- Status: closed +- Type: type/bug +- Created At: 2024-04-15T07:18:33Z +- Closed At: 2024-04-16T11:12:07Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.1, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- image 2 should simplify to a1=a2 and a1=3 and a2=3. +- Fix PR #63162: planner: Convert OR to IN predicate | tidb-test=pr/2588 + URL: https://github.com/pingcap/tidb/pull/63162 + State: open + Merged At: not merged + Changed Files Count: 29 + Main Modules: tests/integrationtest, pkg/planner/core, pkg/executor + Sample Changed Files: + - pkg/executor/explainfor_test.go + - pkg/executor/sortexec/parallel_sort_test.go + - pkg/planner/core/casetest/partition/testdata/integration_partition_suite_out.json + - pkg/planner/core/casetest/partition/testdata/integration_partition_suite_xut.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_out.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_xut.json + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_out.json + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/rule/rule_predicate_simplification.go + - pkg/planner/core/testdata/index_merge_suite_out.json + - pkg/planner/core/testdata/index_merge_suite_xut.json + - pkg/planner/core/testdata/plan_suite_unexported_out.json + - tests/integrationtest/r/executor/explainfor.result + - tests/integrationtest/r/executor/partition/partition_boundaries.result + - tests/integrationtest/r/executor/partition/partition_with_expression.result + - tests/integrationtest/r/explain_cte.result + - tests/integrationtest/r/explain_shard_index.result + - tests/integrationtest/r/expression/issues.result + PR Summary: What problem does this PR solve? Problem Summary: Initial motivation in is that IN list equivalent to OR list produce better outcomes with index-selection and producing a unified canonical form using IN list produce better plans and make it consistent. What changed and how does it work? Predicate simplification is extended to convert OR list to IN list as the last rewrite. This uncovered other limitations in predicate simplification that does not cover IN list and IR list equally. Additional changes were made: (1) flip flop between OR and IN lists to make sure existing prediate simplification is applied fully. (2) Use equivalence classes to support simplification. For example a1=a2 and a2 in (1,2,3) and a1 > 2 should simplify to a1=a2 and a1=3 and a2=3. +- Related PR #63317: planner : merge index scan ranges | tidb-test=pr/2591 + URL: https://github.com/pingcap/tidb/pull/63317 + State: open + Merged At: not merged + Changed Files Count: 21 + Main Modules: tests/integrationtest, pkg/planner/core, pkg/util, pkg/executor + Sample Changed Files: + - pkg/executor/test/indexmergereadtest/index_merge_reader_test.go + - pkg/planner/core/casetest/index/testdata/index_range_out.json + - pkg/planner/core/casetest/index/testdata/index_range_xut.json + - pkg/planner/core/casetest/partition/testdata/integration_partition_suite_out.json + - pkg/planner/core/casetest/partition/testdata/integration_partition_suite_xut.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_out.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_xut.json + - pkg/util/ranger/detacher.go + - pkg/util/ranger/ranger_test.go + - tests/integrationtest/r/black_list.result + - tests/integrationtest/r/explain_complex.result + - tests/integrationtest/r/explain_complex_stats.result + - tests/integrationtest/r/explain_generate_column_substitute.result + - tests/integrationtest/r/expression/vitess_hash.result + - tests/integrationtest/r/generated_columns.result + - tests/integrationtest/r/planner/core/casetest/index/index.result + - tests/integrationtest/r/planner/core/casetest/partition/partition_pruner.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/casetest/rule/rule_result_reorder.result + - tests/integrationtest/r/util/ranger.result + PR Summary: What problem does this PR solve? Problem Summary: This is a an enhancement for the planner range derivation logic for IN list which produces more ranges than the equivalent OR predicate. See example below. The ranges [1,1] [2,2] for the IN predicate compared to [1,2] for the equivalent OR predicate. Both forms scan the same data but less ranges could perform better especially if the number of ranges is high. This issue addresses of on the potential side effect of issue that aims to produce one canonical form for IN and irs equivalent OR list. What changed and how does it work? + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59759-data-too-long-field-len-x-error-when-loading-stats-for-a-bit-column.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59759-data-too-long-field-len-x-error-when-loading-stats-for-a-bit-column.md new file mode 100644 index 0000000..bdc6778 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59759-data-too-long-field-len-x-error-when-loading-stats-for-a-bit-column.md @@ -0,0 +1,119 @@ +# Issue #59759: `Data Too Long, field len X` error when loading stats for a `bit` column + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/59759 +- Status: closed +- Type: type/bug +- Created At: 2025-02-25T13:38:14Z +- Closed At: 2025-04-15T08:57:08Z +- Labels: affects-7.1, affects-7.5, affects-8.1, affects-8.5, report/customer, severity/major, sig/planner, type/bug, type/regression + +## Customer-Facing Phenomenon +- Both async and sync load failed. In tidb log: + +## Linked PRs +- Related PR #59768: table: Revert "table: fix the issue that the default value for `BIT` column is wrong (#57303)" + URL: https://github.com/pingcap/tidb/pull/59768 + State: closed + Merged At: 2025-02-26T04:38:33Z + Changed Files Count: 5 + Main Modules: ddl/ddl_api.go, ddl/integration_test.go, executor/write_test.go, types/datum.go, types/datum_test.go + Sample Changed Files: + - ddl/ddl_api.go + - ddl/integration_test.go + - executor/write_test.go + - types/datum.go + - types/datum_test.go + PR Summary: Reverts pingcap/tidb#57354 because of close #59769 We'll need to reconsider the solution / bugfix for both issues before the next release of 6.5. +- Fix PR #59791: stats: use an alternative function to read the bound from `BLOB` stored in `mysql.stats_buckets`. | tidb-test=pr/2503 + URL: https://github.com/pingcap/tidb/pull/59791 + State: closed + Merged At: 2025-04-15T08:57:06Z + Changed Files Count: 5 + Main Modules: pkg/statistics, pkg/types + Sample Changed Files: + - pkg/statistics/handle/handletest/BUILD.bazel + - pkg/statistics/handle/handletest/handle_test.go + - pkg/statistics/handle/storage/read.go + - pkg/statistics/handle/storage/save.go + - pkg/types/datum.go + PR Summary: What problem does this PR solve? Problem Summary: The stats logic relies on the behavior that type conversion from A to Blob to A is still equal to the original value. However, it's not true for after What changed and how does it work? Add an extra function to handle the load of bound specially. It'll not affect the behavior of any expression, but only modify the stats part. +- Related PR #59840: table: Revert "table: fix the issue that the default value for `BIT` column is wrong (#57303)" + URL: https://github.com/pingcap/tidb/pull/59840 + State: closed + Merged At: 2025-02-28T09:26:30Z + Changed Files Count: 8 + Main Modules: tests/integrationtest, pkg/types, pkg/ddl, pkg/executor + Sample Changed Files: + - pkg/ddl/ddl_api.go + - pkg/executor/test/writetest/write_test.go + - pkg/types/datum.go + - pkg/types/datum_test.go + - tests/integrationtest/r/ddl/column.result + - tests/integrationtest/r/table/tables.result + - tests/integrationtest/t/ddl/column.test + - tests/integrationtest/t/table/tables.test + PR Summary: Reverts pingcap/tidb#57356 because of What problem does this PR solve? We'll need to reconsider the solution / bugfix for both issues before the next release of 7.5. +- Fix PR #60579: stats: use an alternative function to read the bound from `BLOB` stored in `mysql.stats_buckets`. | tidb-test=pr/2503 (#59791) + URL: https://github.com/pingcap/tidb/pull/60579 + State: open + Merged At: not merged + Changed Files Count: 5 + Main Modules: pkg/statistics, types/datum.go + Sample Changed Files: + - pkg/statistics/handle/handletest/BUILD.bazel + - pkg/statistics/handle/handletest/handle_test.go + - pkg/statistics/handle/storage/read.go + - pkg/statistics/handle/storage/save.go + - types/datum.go + PR Summary: This is an automated cherry-pick of #59791 What problem does this PR solve? Problem Summary: The stats logic relies on the behavior that type conversion from A to Blob to A is still equal to the original value. However, it's not true for after What changed and how does it work? +- Fix PR #60580: stats: use an alternative function to read the bound from `BLOB` stored in `mysql.stats_buckets`. | tidb-test=pr/2503 (#59791) + URL: https://github.com/pingcap/tidb/pull/60580 + State: closed + Merged At: 2025-07-14T23:59:45Z + Changed Files Count: 13 + Main Modules: pkg/statistics, tests/integrationtest, pkg/types, pkg/ddl, pkg/executor + Sample Changed Files: + - pkg/ddl/ddl_api.go + - pkg/executor/test/writetest/write_test.go + - pkg/statistics/handle/handletest/BUILD.bazel + - pkg/statistics/handle/handletest/handle_test.go + - pkg/statistics/handle/storage/read.go + - pkg/statistics/handle/storage/save.go + - pkg/types/BUILD.bazel + - pkg/types/datum.go + - pkg/types/datum_test.go + - tests/integrationtest/r/ddl/column.result + - tests/integrationtest/r/table/tables.result + - tests/integrationtest/t/ddl/column.test + - tests/integrationtest/t/table/tables.test + PR Summary: This is an automated cherry-pick of #59791 What problem does this PR solve? Problem Summary: The stats logic relies on the behavior that type conversion from A to Blob to A is still equal to the original value. However, it's not true for after What changed and how does it work? +- Fix PR #60581: stats: use an alternative function to read the bound from `BLOB` stored in `mysql.stats_buckets`. | tidb-test=pr/2503 (#59791) + URL: https://github.com/pingcap/tidb/pull/60581 + State: open + Merged At: not merged + Changed Files Count: 5 + Main Modules: pkg/statistics, pkg/types + Sample Changed Files: + - pkg/statistics/handle/handletest/BUILD.bazel + - pkg/statistics/handle/handletest/handle_test.go + - pkg/statistics/handle/storage/read.go + - pkg/statistics/handle/storage/save.go + - pkg/types/datum.go + PR Summary: This is an automated cherry-pick of #59791 What problem does this PR solve? Problem Summary: The stats logic relies on the behavior that type conversion from A to Blob to A is still equal to the original value. However, it's not true for after What changed and how does it work? +- Fix PR #60583: stats: use an alternative function to read the bound from `BLOB` stored in `mysql.stats_buckets`. (#59791) + URL: https://github.com/pingcap/tidb/pull/60583 + State: closed + Merged At: 2025-04-16T06:32:09Z + Changed Files Count: 5 + Main Modules: pkg/statistics, pkg/types + Sample Changed Files: + - pkg/statistics/handle/handletest/BUILD.bazel + - pkg/statistics/handle/handletest/handle_test.go + - pkg/statistics/handle/storage/read.go + - pkg/statistics/handle/storage/save.go + - pkg/types/datum.go + PR Summary: This is an automated cherry-pick of #59791 What problem does this PR solve? Problem Summary: The stats logic relies on the behavior that type conversion from A to Blob to A is still equal to the original value. However, it's not true for after What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59869-unionall-should-be-able-to-reserve-the-order-property-comming-from-its-children.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59869-unionall-should-be-able-to-reserve-the-order-property-comming-from-its-children.md new file mode 100644 index 0000000..139595d --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59869-unionall-should-be-able-to-reserve-the-order-property-comming-from-its-children.md @@ -0,0 +1,17 @@ +# Issue #59869: UnionAll should be able to reserve the order property comming from its children + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/59869 +- Status: open +- Type: type/enhancement +- Created At: 2025-03-03T09:56:15Z +- Labels: report/customer, sig/execution, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- suppose that we have table + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59902-the-estrows-is-wrong-for-inner-operator-of-index-join.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59902-the-estrows-is-wrong-for-inner-operator-of-index-join.md new file mode 100644 index 0000000..57bba20 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59902-the-estrows-is-wrong-for-inner-operator-of-index-join.md @@ -0,0 +1,45 @@ +# Issue #59902: The estrows is wrong for inner operator of index join + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/59902 +- Status: closed +- Type: type/bug +- Created At: 2025-03-05T02:49:08Z +- Closed At: 2025-03-25T13:26:12Z +- Labels: affects-8.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- create table t1(a int primary key, b int); create table t2(a int, b int, key idx(a)); set tidb_enable_inl_join_inner_multi_pattern=on; explain select t1.b,(select count(*) from t2 where t2.a=t1.a) as a from t1 where t1.a=1; + +## Linked PRs +- Fix PR #60071: planner: fix wrong HashAgg estrows for inner operator of index join + URL: https://github.com/pingcap/tidb/pull/60071 + State: closed + Merged At: 2025-03-25T13:26:11Z + Changed Files Count: 6 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_out.json + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/issuetest/BUILD.bazel + - pkg/planner/core/issuetest/planner_issue_test.go + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/indexjoin.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? in the , we forget to transfer the parents stats by the child stats. +- Fix PR #60336: planner: fix wrong HashAgg estrows for inner operator of index join (#60071) + URL: https://github.com/pingcap/tidb/pull/60336 + State: closed + Merged At: 2025-04-16T16:36:22Z + Changed Files Count: 6 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_out.json + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/issuetest/BUILD.bazel + - pkg/planner/core/issuetest/planner_issue_test.go + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/indexjoin.result + PR Summary: This is an automated cherry-pick of #60071 What problem does this PR solve? Problem Summary: What changed and how does it work? in the , we forget to transfer the parents stats by the child stats. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59972-planner-selection-splits-the-join-group-to-two-parts-forbidding-join-reorder-to-.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59972-planner-selection-splits-the-join-group-to-two-parts-forbidding-join-reorder-to-.md new file mode 100644 index 0000000..594f98c --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59972-planner-selection-splits-the-join-group-to-two-parts-forbidding-join-reorder-to-.md @@ -0,0 +1,64 @@ +# Issue #59972: planner: selection splits the join group to two parts, forbidding join reorder to get best order + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/59972 +- Status: closed +- Type: type/enhancement +- Created At: 2025-03-10T03:10:02Z +- Closed At: 2026-01-22T04:18:57Z +- Labels: affects-6.5, affects-8.5, planner/cascades, planner/join-order, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- See the case below, the best join order is joining and first, but both our join-order and can't support it: + +## Linked PRs +- Related PR #63522: planner: Allow leading ordered table to survive (WIP) + URL: https://github.com/pingcap/tidb/pull/63522 + State: open + Merged At: not merged + Changed Files Count: 11 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/operator/physicalop/physical_index_join.go + - pkg/planner/core/operator/physicalop/physical_topn.go + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/rule_join_reorder.go + - pkg/planner/core/rule_join_reorder_greedy.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #64535: planner: handle the selection between the join group + URL: https://github.com/pingcap/tidb/pull/64535 + State: closed + Merged At: 2026-01-22T04:18:56Z + Changed Files Count: 6 + Main Modules: pkg/session, tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/rule_join_reorder.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + - tests/integrationtest/r/planner/core/join_reorder2.result + - tests/integrationtest/t/planner/core/join_reorder2.test + PR Summary: What problem does this PR solve? Problem Summary: Handle the selection between the join group. Let more join group to participate the join order phase. What changed and how does it work? The plan node prevent the function to get more join groups. +- Related PR #65742: planner: handle the selection between the join group (#64535) + URL: https://github.com/pingcap/tidb/pull/65742 + State: closed + Merged At: 2026-01-23T16:11:02Z + Changed Files Count: 6 + Main Modules: pkg/session, tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/rule_join_reorder.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + - pkg/sessionctx/variable/tidb_vars.go + - tests/integrationtest/r/planner/core/join_reorder2.result + - tests/integrationtest/t/planner/core/join_reorder2.test + PR Summary: This is an automated cherry-pick of #64535 What problem does this PR solve? Problem Summary: Handle the selection between the join group. Let more join group to participate the join order phase. What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59986-planner-support-long-binding-statement.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59986-planner-support-long-binding-statement.md new file mode 100644 index 0000000..2a95be3 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-59986-planner-support-long-binding-statement.md @@ -0,0 +1,29 @@ +# Issue #59986: planner: support long binding statement + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/59986 +- Status: closed +- Type: type/enhancement +- Created At: 2025-03-10T08:04:29Z +- Closed At: 2025-03-12T10:46:51Z +- Labels: epic/sql-plan-management, impact/upgrade, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- At least we can update its type to , which can support SQL with a maximum length of . + +## Linked PRs +- Fix PR #60007: planner: update bind_info binding columns from TEXT to LONGTEXT to support long bindings + URL: https://github.com/pingcap/tidb/pull/60007 + State: closed + Merged At: 2025-03-12T10:46:50Z + Changed Files Count: 4 + Main Modules: pkg/bindinfo, br/pkg, pkg/session + Sample Changed Files: + - br/pkg/restore/snap_client/systable_restore_test.go + - pkg/bindinfo/BUILD.bazel + - pkg/bindinfo/binding_operator_test.go + - pkg/session/bootstrap.go + PR Summary: What problem does this PR solve? Problem Summary: planner: update bind_info binding columns from TEXT to LONGTEXT to support long bindings What changed and how does it work? planner: update bind_info binding columns from TEXT to LONGTEXT to support long bindings + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60037-auto-analyze-merge-global-statistics-failed-when-some-partition-statistics-are-m.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60037-auto-analyze-merge-global-statistics-failed-when-some-partition-statistics-are-m.md new file mode 100644 index 0000000..e9e32c7 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60037-auto-analyze-merge-global-statistics-failed-when-some-partition-statistics-are-m.md @@ -0,0 +1,67 @@ +# Issue #60037: auto analyze merge global statistics failed when some partition statistics are missing + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60037 +- Status: closed +- Type: type/bug +- Created At: 2025-03-12T10:08:36Z +- Closed At: 2025-03-13T04:29:03Z +- Labels: affects-6.5, affects-7.1, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- merge global stats failed + +## Linked PRs +- Fix PR #60035: statistics: sync TiDBSkipMissingPartitionStats value in the auto analyze + URL: https://github.com/pingcap/tidb/pull/60035 + State: closed + Merged At: 2025-03-18T14:13:39Z + Changed Files Count: 2 + Main Modules: statistics/handle + Sample Changed Files: + - statistics/handle/handle.go + - statistics/handle/update_test.go + PR Summary: …essionVar What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #60038: planner: add test cases for auto analyze's tidb_skip_missing_partition_stats + URL: https://github.com/pingcap/tidb/pull/60038 + State: closed + Merged At: 2025-03-13T04:29:02Z + Changed Files Count: 2 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/tests/analyze/BUILD.bazel + - pkg/planner/core/tests/analyze/analyze_test.go + PR Summary: … What problem does this PR solve? Problem Summary: What changed and how does it work? add test cases for auto analyze's tidb_skip_missing_partition_stats +- Fix PR #60046: statistics: sync TiDBSkipMissingPartitionStats value in the auto analyze (#60038) + URL: https://github.com/pingcap/tidb/pull/60046 + State: closed + Merged At: 2025-04-25T08:17:38Z + Changed Files Count: 2 + Main Modules: statistics/handle + Sample Changed Files: + - statistics/handle/updatetest/BUILD.bazel + - statistics/handle/updatetest/update_test.go + PR Summary: This is an automated cherry-pick of #60038 … What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #60049: planner: add test cases for auto analyze's tidb_skip_missing_partition_stats (#60038) + URL: https://github.com/pingcap/tidb/pull/60049 + State: closed + Merged At: not merged + Changed Files Count: 2 + Main Modules: planner/core + Sample Changed Files: + - planner/core/tests/cte/BUILD.bazel + - planner/core/tests/cte/main_test.go + PR Summary: This is an automated cherry-pick of #60038 … What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #60270: statistics: sync TiDBSkipMissingPartitionStats value in the auto analyze | tidb-test=920d9bf1b1137cda1272bdd59ae527aee8067944 + URL: https://github.com/pingcap/tidb/pull/60270 + State: closed + Merged At: 2025-03-27T02:58:12Z + Changed Files Count: 2 + Main Modules: statistics/handle + Sample Changed Files: + - statistics/handle/handle.go + - statistics/handle/update_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60076-planner-keep-redundant-join-key-predicates-when-propagating-eq-predicates.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60076-planner-keep-redundant-join-key-predicates-when-propagating-eq-predicates.md new file mode 100644 index 0000000..078b87d --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60076-planner-keep-redundant-join-key-predicates-when-propagating-eq-predicates.md @@ -0,0 +1,108 @@ +# Issue #60076: planner: keep redundant Join Key predicates when propagating EQ predicates + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60076 +- Status: closed +- Type: type/enhancement +- Created At: 2025-03-14T02:29:55Z +- Closed At: 2025-09-10T02:07:49Z +- Labels: planner/join-order, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- See the case below, before adding , the can work well, while after adding it, the can't work and there are multiple dangerous Cartesian Joins. + +## Linked PRs +- Fix PR #62996: expression: avoid removing the equal condition when to propagte constant | tidb-test=pr/2581 + URL: https://github.com/pingcap/tidb/pull/62996 + State: closed + Merged At: not merged + Changed Files Count: 45 + Main Modules: pkg/planner/core, pkg/expression, tests/integrationtest, pkg/executor, pkg/planner, pkg/testkit + Sample Changed Files: + - pkg/executor/builder.go + - pkg/executor/explainfor_test.go + - pkg/executor/testdata/prepare_suite_out.json + - pkg/expression/builtin_compare_test.go + - pkg/expression/constant_propagation.go + - pkg/expression/constant_test.go + - pkg/expression/scalar_function.go + - pkg/planner/cascades/old/transformation_rules.go + - pkg/planner/core/casetest/dag/testdata/plan_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_out.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_xut.json + - pkg/planner/core/casetest/rule/rule_predicate_simplification_test.go + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_in.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_out.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/find_best_task.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? When we perform constant propagation, sometimes we remove the equal condition, which is actually very dangerous. So the fix is that if it detects that the condition being processed is an equal condition, then the new expression generated from this expression should be appended to the slices, not replaced. +- Fix PR #63375: planner: keep join keys for join optimization in constant propagation + URL: https://github.com/pingcap/tidb/pull/63375 + State: closed + Merged At: not merged + Changed Files Count: 3 + Main Modules: pkg/planner/core, pkg/expression + Sample Changed Files: + - pkg/expression/constant_propagation.go + - pkg/planner/core/casetest/join/BUILD.bazel + - pkg/planner/core/casetest/join/join_test.go + PR Summary: What problem does this PR solve? Problem Summary: planner: keep join keys for join optimization in constant propagation What changed and how does it work? +- Related PR #63385: planner: refactoring, use the special function to propagate constant for Joins + URL: https://github.com/pingcap/tidb/pull/63385 + State: closed + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/rule/rule_init.go + - pkg/planner/core/rule/rule_predicate_simplification.go + - pkg/planner/core/rule/util/misc.go + PR Summary: What problem does this PR solve? Problem Summary: planner: refactoring, use the special function to propagate constant for Joins What changed and how does it work? planner: refactoring, use the special function to propagate constant for Joins +- Fix PR #63404: planner: keep join keys for join optimization in constant propagation | tidb-test=pr/2600 + URL: https://github.com/pingcap/tidb/pull/63404 + State: closed + Merged At: 2025-09-10T02:07:48Z + Changed Files Count: 12 + Main Modules: pkg/planner/core, pkg/session, pkg/expression, pkg/bindinfo, pkg/planner + Sample Changed Files: + - pkg/bindinfo/binding_plan_generation.go + - pkg/expression/constant_propagation.go + - pkg/expression/constant_test.go + - pkg/planner/cascades/old/transformation_rules.go + - pkg/planner/core/casetest/join/BUILD.bazel + - pkg/planner/core/casetest/join/join_test.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/rule/rule_predicate_simplification.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/setvar_affect.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: What problem does this PR solve? Problem Summary: planner: keep join keys for join optimization in constant propagation What changed and how does it work? +- Fix PR #64799: planner: keep join keys for join optimization in constant propagation | tidb-test=pr/2600 (#63404) + URL: https://github.com/pingcap/tidb/pull/64799 + State: open + Merged At: not merged + Changed Files Count: 12 + Main Modules: pkg/planner/core, pkg/session, pkg/expression, pkg/bindinfo, pkg/planner + Sample Changed Files: + - pkg/bindinfo/binding_plan_generation.go + - pkg/expression/constant_propagation.go + - pkg/expression/constant_test.go + - pkg/planner/cascades/transformation_rules.go + - pkg/planner/core/casetest/join/BUILD.bazel + - pkg/planner/core/casetest/join/join_test.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/rule/rule_predicate_simplification.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/setvar_affect.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: This is an automated cherry-pick of #63404 What problem does this PR solve? Problem Summary: planner: keep join keys for join optimization in constant propagation What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60137-planner-prevent-pseudo-stats-from-dominating-the-stats-cache.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60137-planner-prevent-pseudo-stats-from-dominating-the-stats-cache.md new file mode 100644 index 0000000..3117ab4 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60137-planner-prevent-pseudo-stats-from-dominating-the-stats-cache.md @@ -0,0 +1,31 @@ +# Issue #60137: planner: prevent pseudo-stats from dominating the Stats Cache + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60137 +- Status: closed +- Type: type/enhancement +- Created At: 2025-03-18T04:35:37Z +- Closed At: 2025-09-09T20:29:04Z +- Labels: component/statistics, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- In a customer case below, pseudo-stats costs 1.8GB (40%) memory: ![Image]() + +## Linked PRs +- Fix PR #63160: stats: optimize memory footprint of pseudo stats table + URL: https://github.com/pingcap/tidb/pull/63160 + State: closed + Merged At: 2025-09-09T20:29:03Z + Changed Files Count: 6 + Main Modules: pkg/statistics, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/planbuilder.go + - pkg/statistics/BUILD.bazel + - pkg/statistics/handle/handle.go + - pkg/statistics/histogram.go + - pkg/statistics/histogram_test.go + - pkg/statistics/table.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? A significant part of memory consumption of pseudo table is creating new chunk for every pseudo table creation. These new chunks are read only and only to provide compatibility to the code of using real stats table struct. We can make it a static object and shared by all pseudo tables to reduce memory usage. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60242-planner-need-to-ignore-the-ordering-index-path-even-if-the-index-selectivity-is-.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60242-planner-need-to-ignore-the-ordering-index-path-even-if-the-index-selectivity-is-.md new file mode 100644 index 0000000..474fb42 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60242-planner-need-to-ignore-the-ordering-index-path-even-if-the-index-selectivity-is-.md @@ -0,0 +1,125 @@ +# Issue #60242: Planner: Need to ignore the ordering index path even if the index selectivity is 1. + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60242 +- Status: closed +- Type: type/enhancement +- Created At: 2025-03-24T17:42:27Z +- Closed At: 2025-03-26T05:56:01Z +- Labels: affects-7.1, affects-7.5, affects-8.1, affects-8.5, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #60255: planner: change the semantic of `tidb_opt_ordering_index_selectivity_threshold` from `<` to `<=` + URL: https://github.com/pingcap/tidb/pull/60255 + State: closed + Merged At: 2025-03-26T05:56:00Z + Changed Files Count: 4 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/cardinality/testdata/cardinality_suite_in.json + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - pkg/planner/core/find_best_task.go + PR Summary: What problem does this PR solve? Problem Summary: As said in #42060 and the [doc](), this variable was originally introduced to control the choice between a "filter index" that satisfies the filter conditions and an "ordering index" that satisfies the clause. The allowed range for this variable is . Obviously, it's meaningless to choose a "filter index" when the selectivity is 1, which basically means there are no filters. So we made the semantic of this variable "less than". This provides a benefit: when setting it to 0, the behavior is not changed, so we can safely use 0 as the default value. However, in a recent ticket, we met another case where there were no indexes satisfying the filters, and the choice was between a tiflash path and an "ordering index" path. In this specific ticket, +- Fix PR #60282: planner: change the semantic of `tidb_opt_ordering_index_selectivity_threshold` from `<` to `<=` (#60255) + URL: https://github.com/pingcap/tidb/pull/60282 + State: closed + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/planner, pkg/planner/core, statistics/testdata + Sample Changed Files: + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/cardinality/testdata/cardinality_suite_in.json + - pkg/planner/core/find_best_task.go + - statistics/testdata/trace_suite_out.json + PR Summary: This is an automated cherry-pick of #60255 What problem does this PR solve? Problem Summary: As said in #42060 and the [doc](), this variable was originally introduced to control the choice between a "filter index" that satisfies the filter conditions and an "ordering index" that satisfies the clause. The allowed range for this variable is . Obviously, it's meaningless to choose a "filter index" when the selectivity is 1, which basically means there are no filters. So we made the semantic of this variable "less than". This provides a benefit: when setting it to 0, the behavior is not changed, so we can safely use 0 as the default value. +- Fix PR #60285: planner: change the semantic of `tidb_opt_ordering_index_selectivity_threshold` from `<` to `<=` (#60255) + URL: https://github.com/pingcap/tidb/pull/60285 + State: closed + Merged At: 2025-03-27T04:30:07Z + Changed Files Count: 4 + Main Modules: statistics/testdata, planner/core, statistics/integration_test.go + Sample Changed Files: + - planner/core/find_best_task.go + - statistics/integration_test.go + - statistics/testdata/integration_suite_in.json + - statistics/testdata/integration_suite_out.json + PR Summary: Manual cherry-pick of #60255 What problem does this PR solve? Problem Summary: As said in #42060 and the [doc](), this variable was originally introduced to control the choice between a "filter index" that satisfies the filter conditions and an "ordering index" that satisfies the clause. The allowed range for this variable is . +- Fix PR #60445: planner: change the semantic of `tidb_opt_ordering_index_selectivity_threshold` from `<` to `<=` (#60255) + URL: https://github.com/pingcap/tidb/pull/60445 + State: closed + Merged At: 2025-04-14T10:29:04Z + Changed Files Count: 4 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/cardinality/testdata/cardinality_suite_in.json + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - pkg/planner/core/find_best_task.go + PR Summary: This is an automated cherry-pick of #60255 What problem does this PR solve? Problem Summary: As said in #42060 and the [doc](), this variable was originally introduced to control the choice between a "filter index" that satisfies the filter conditions and an "ordering index" that satisfies the clause. The allowed range for this variable is . Obviously, it's meaningless to choose a "filter index" when the selectivity is 1, which basically means there are no filters. So we made the semantic of this variable "less than". This provides a benefit: when setting it to 0, the behavior is not changed, so we can safely use 0 as the default value. +- Fix PR #60537: planner: change the semantic of `tidb_opt_ordering_index_selectivity_threshold` from `<` to `<=` (#60255) + URL: https://github.com/pingcap/tidb/pull/60537 + State: closed + Merged At: 2025-04-16T16:25:57Z + Changed Files Count: 4 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/cardinality/testdata/cardinality_suite_in.json + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - pkg/planner/core/find_best_task.go + PR Summary: This is an automated cherry-pick of #60255 What problem does this PR solve? Problem Summary: As said in #42060 and the [doc](), this variable was originally introduced to control the choice between a "filter index" that satisfies the filter conditions and an "ordering index" that satisfies the clause. The allowed range for this variable is . Obviously, it's meaningless to choose a "filter index" when the selectivity is 1, which basically means there are no filters. So we made the semantic of this variable "less than". This provides a benefit: when setting it to 0, the behavior is not changed, so we can safely use 0 as the default value. +- Fix PR #60538: planner: change the semantic of `tidb_opt_ordering_index_selectivity_threshold` from `<` to `<=` (#60255) + URL: https://github.com/pingcap/tidb/pull/60538 + State: closed + Merged At: 2025-04-16T16:26:00Z + Changed Files Count: 4 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/cardinality/testdata/cardinality_suite_in.json + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - pkg/planner/core/find_best_task.go + PR Summary: This is an automated cherry-pick of #60255 What problem does this PR solve? Problem Summary: As said in #42060 and the [doc](), this variable was originally introduced to control the choice between a "filter index" that satisfies the filter conditions and an "ordering index" that satisfies the clause. The allowed range for this variable is . Obviously, it's meaningless to choose a "filter index" when the selectivity is 1, which basically means there are no filters. So we made the semantic of this variable "less than". This provides a benefit: when setting it to 0, the behavior is not changed, so we can safely use 0 as the default value. +- Fix PR #60542: planner: change the semantic of `tidb_opt_ordering_index_selectivity_threshold` from `<` to `<=` (#60255) + URL: https://github.com/pingcap/tidb/pull/60542 + State: closed + Merged At: 2025-04-16T16:25:54Z + Changed Files Count: 4 + Main Modules: statistics/testdata, planner/core, statistics/integration_test.go + Sample Changed Files: + - planner/core/find_best_task.go + - statistics/integration_test.go + - statistics/testdata/integration_suite_in.json + - statistics/testdata/integration_suite_out.json + PR Summary: Manual cherry-pick of #60255 What problem does this PR solve? Problem Summary: As said in #42060 and the [doc](), this variable was originally introduced to control the choice between a "filter index" that satisfies the filter conditions and an "ordering index" that satisfies the clause. The allowed range for this variable is . +- Fix PR #60588: planner: change the semantic of `tidb_opt_ordering_index_selectivity_threshold` from `<` to `<=` (#60255) + URL: https://github.com/pingcap/tidb/pull/60588 + State: closed + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/planner, pkg/planner/core, statistics/testdata + Sample Changed Files: + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/cardinality/testdata/cardinality_suite_in.json + - pkg/planner/core/find_best_task.go + - statistics/testdata/trace_suite_out.json + PR Summary: This is an automated cherry-pick of #60255 What problem does this PR solve? Problem Summary: As said in #42060 and the [doc](), this variable was originally introduced to control the choice between a "filter index" that satisfies the filter conditions and an "ordering index" that satisfies the clause. The allowed range for this variable is . Obviously, it's meaningless to choose a "filter index" when the selectivity is 1, which basically means there are no filters. So we made the semantic of this variable "less than". This provides a benefit: when setting it to 0, the behavior is not changed, so we can safely use 0 as the default value. +- Fix PR #60733: planner: change the semantic of `tidb_opt_ordering_index_selectivity_threshold` from `<` to `<=` (#60255) + URL: https://github.com/pingcap/tidb/pull/60733 + State: closed + Merged At: 2025-04-23T12:54:25Z + Changed Files Count: 4 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/cardinality/testdata/cardinality_suite_in.json + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - pkg/planner/core/find_best_task.go + PR Summary: Manual cherry-pick of #60255 What problem does this PR solve? Problem Summary: As said in #42060 and the [doc](), this variable was originally introduced to control the choice between a "filter index" that satisfies the filter conditions and an "ordering index" that satisfies the clause. The allowed range for this variable is . + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60357-require-an-ability-to-externally-influence-the-optimizer-cost-model.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60357-require-an-ability-to-externally-influence-the-optimizer-cost-model.md new file mode 100644 index 0000000..ed30ec9 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60357-require-an-ability-to-externally-influence-the-optimizer-cost-model.md @@ -0,0 +1,122 @@ +# Issue #60357: Require an ability to externally influence the optimizer cost model + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60357 +- Status: closed +- Type: type/enhancement +- Created At: 2025-04-02T00:50:13Z +- Closed At: 2025-04-10T21:56:34Z +- Labels: affects-8.5, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- The TiDB optimizer would benefit from having an ability to influence the optimizer's cost of each physical operator in a query plan. This could be used externally to influence query plans (by customers/users and also TiDB engineers), and could be used by TiDB engineers to test and adjust the optimizer's cost model. + +## Linked PRs +- Fix PR #60333: planner: add optimizer cost factors + URL: https://github.com/pingcap/tidb/pull/60333 + State: closed + Merged At: 2025-04-10T21:56:33Z + Changed Files Count: 9 + Main Modules: tests/integrationtest, pkg/session, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/plan_cost_ver2_test.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + - tests/integrationtest/r/executor/explain.result + - tests/integrationtest/r/planner/core/cbo.result + - tests/integrationtest/r/planner/core/plan_cache.result + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Add a global variable "cost factor" to the more common optimizer physical operators. This allows an ability to externally control the optimizer's plan choice(s) - by either increasing the cost factor for one operation (to discourage that plan choice), or decreasing the cost factor of an operation (to encourage that plan choice). +- Fix PR #60512: planner: add optimizer cost factor tests + URL: https://github.com/pingcap/tidb/pull/60512 + State: closed + Merged At: 2025-04-14T03:55:06Z + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Add testcases for recent optimizer cost factors. +- Fix PR #60558: planner: add hint support for optimizer cost factors + URL: https://github.com/pingcap/tidb/pull/60558 + State: closed + Merged At: 2025-04-16T04:27:58Z + Changed Files Count: 4 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/plan_cost_ver2_test.go + - pkg/sessionctx/variable/setvar_affect.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #61765: planner: add optimizer cost factors (#60333) + URL: https://github.com/pingcap/tidb/pull/61765 + State: closed + Merged At: 2025-06-25T15:23:21Z + Changed Files Count: 9 + Main Modules: tests/integrationtest, pkg/session, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/plan_cost_ver2_test.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + - pkg/sessionctx/variable/tidb_vars.go + - tests/integrationtest/r/executor/explain.result + - tests/integrationtest/r/planner/core/cbo.result + - tests/integrationtest/r/planner/core/plan_cache.result + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + PR Summary: This is an automated cherry-pick of #60333 What problem does this PR solve? Problem Summary: What changed and how does it work? Add a global variable "cost factor" to the more common optimizer physical operators. This allows an ability to externally control the optimizer's plan choice(s) - by either increasing the cost factor for one operation (to discourage that plan choice), or decreasing the cost factor of an operation (to encourage that plan choice). +- Fix PR #61903: planner: add optimizer cost factor tests (#60512) + URL: https://github.com/pingcap/tidb/pull/61903 + State: closed + Merged At: 2025-07-02T19:48:27Z + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2_test.go + PR Summary: This is an automated cherry-pick of #60512 What problem does this PR solve? Problem Summary: What changed and how does it work? Add testcases for recent optimizer cost factors. +- Fix PR #61904: planner: add hint support for optimizer cost factors (#60558) + URL: https://github.com/pingcap/tidb/pull/61904 + State: closed + Merged At: 2025-07-07T13:43:33Z + Changed Files Count: 4 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/plan_cost_ver2_test.go + - pkg/sessionctx/variable/setvar_affect.go + PR Summary: This is an automated cherry-pick of #60558 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #64553: planner: add optimizer cost factors + URL: https://github.com/pingcap/tidb/pull/64553 + State: closed + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/session, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + - pkg/sessionctx/variable/tidb_vars.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #64564: planner: add optimizer cost factors | tidb-test=release-8.5-20250114-v8.5.0 + URL: https://github.com/pingcap/tidb/pull/64564 + State: closed + Merged At: 2025-11-19T15:38:41Z + Changed Files Count: 9 + Main Modules: tests/integrationtest, pkg/session, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/plan_cost_ver2_test.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + - pkg/sessionctx/variable/tidb_vars.go + - tests/integrationtest/r/executor/explain.result + - tests/integrationtest/r/planner/core/cbo.result + - tests/integrationtest/r/planner/core/plan_cache.result + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + PR Summary: What problem does this PR solve? Problem Summary: cherry pick to v8.5.0 for hotfix What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60380-plan-cache-the-different-in-value-numbers-in-query-cause-too-much-memory-in-tidb.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60380-plan-cache-the-different-in-value-numbers-in-query-cause-too-much-memory-in-tidb.md new file mode 100644 index 0000000..2a473ab --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60380-plan-cache-the-different-in-value-numbers-in-query-cause-too-much-memory-in-tidb.md @@ -0,0 +1,17 @@ +# Issue #60380: Plan Cache: The different IN value numbers in query cause too much memory in TiDB + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60380 +- Status: open +- Type: type/enhancement +- Created At: 2025-04-02T12:43:35Z +- Labels: epic/plan-cache, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- 2. Use the prepare stmt and query with different IN values number 3. The memory keeps rising and cannot be released + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60455-unknown-column-with-only-full-group-by.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60455-unknown-column-with-only-full-group-by.md new file mode 100644 index 0000000..bb395da --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60455-unknown-column-with-only-full-group-by.md @@ -0,0 +1,17 @@ +# Issue #60455: Unknown column with only-full-group-by + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60455 +- Status: open +- Type: type/bug +- Created At: 2025-04-09T09:06:11Z +- Labels: affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60655-throw-can-t-find-column-xxx-in-schema-column-error-when-execute-sql.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60655-throw-can-t-find-column-xxx-in-schema-column-error-when-execute-sql.md new file mode 100644 index 0000000..d510c13 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60655-throw-can-t-find-column-xxx-in-schema-column-error-when-execute-sql.md @@ -0,0 +1,78 @@ +# Issue #60655: Throw Can't find column xxx in schema Column Error when execute SQL + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60655 +- Status: closed +- Type: type/bug +- Created At: 2025-04-18T08:54:17Z +- Closed At: 2025-05-07T17:23:51Z +- Labels: affects-7.5, affects-8.1, affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #60822: planner: fix wrong TopN's ByItem with expression.ScalarFunction when to PushDownTopN + URL: https://github.com/pingcap/tidb/pull/60822 + State: closed + Merged At: 2025-05-07T17:23:50Z + Changed Files Count: 9 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/executor + Sample Changed Files: + - pkg/executor/infoschema_reader_test.go + - pkg/planner/core/casetest/vectorsearch/testdata/ann_index_suite_in.json + - pkg/planner/core/casetest/vectorsearch/testdata/ann_index_suite_out.json + - pkg/planner/core/integration_test.go + - pkg/planner/core/operator/logicalop/logical_projection.go + - pkg/planner/core/operator/logicalop/logicalop_test/BUILD.bazel + - pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go + - tests/integrationtest/r/planner/core/plan.result + - tests/integrationtest/t/planner/core/plan.test + PR Summary: What problem does this PR solve? Problem Summary: In the There is a column in topN.ByItems is generated by proj, when topN is pushed down below proj, topN cannot obtain the column from datasource, to fix this, we can check whether topN.ByItems contains a column(with ID=0) generated by proj, if so, proj will prevent the optimizer from pushing topN down. 1. A column with ID=0 indicates that the column cannot be resolved by data source. +- Fix PR #61004: planner: fix wrong TopN's ByItem with expression.ScalarFunction when to PushDownTopN (#60822) + URL: https://github.com/pingcap/tidb/pull/61004 + State: closed + Merged At: 2025-07-09T01:43:09Z + Changed Files Count: 4 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/integration_test.go + - pkg/planner/core/rule_topn_push_down.go + - tests/integrationtest/r/planner/core/plan.result + - tests/integrationtest/t/planner/core/plan.test + PR Summary: This is an automated cherry-pick of #60822 What problem does this PR solve? Problem Summary: In the There is a column in topN.ByItems is generated by proj, when topN is pushed down below proj, topN cannot obtain the column from datasource, to fix this, we can check whether topN.ByItems contains a column(with ID=0) generated by proj, if so, proj will prevent the optimizer from pushing topN down. +- Fix PR #61005: planner: fix wrong TopN's ByItem with expression.ScalarFunction when to PushDownTopN (#60822) + URL: https://github.com/pingcap/tidb/pull/61005 + State: open + Merged At: not merged + Changed Files Count: 9 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/executor + Sample Changed Files: + - pkg/executor/infoschema_reader_test.go + - pkg/planner/core/casetest/vectorsearch/testdata/ann_index_suite_in.json + - pkg/planner/core/casetest/vectorsearch/testdata/ann_index_suite_out.json + - pkg/planner/core/integration_test.go + - pkg/planner/core/operator/logicalop/logical_projection.go + - pkg/planner/core/operator/logicalop/logicalop_test/BUILD.bazel + - pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go + - tests/integrationtest/r/planner/core/plan.result + - tests/integrationtest/t/planner/core/plan.test + PR Summary: This is an automated cherry-pick of #60822 What problem does this PR solve? Problem Summary: In the There is a column in topN.ByItems is generated by proj, when topN is pushed down below proj, topN cannot obtain the column from datasource, to fix this, we can check whether topN.ByItems contains a column(with ID=0) generated by proj, if so, proj will prevent the optimizer from pushing topN down. +- Fix PR #61006: planner: fix wrong TopN's ByItem with expression.ScalarFunction when to PushDownTopN (#60822) + URL: https://github.com/pingcap/tidb/pull/61006 + State: closed + Merged At: 2025-07-08T03:13:09Z + Changed Files Count: 7 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/executor + Sample Changed Files: + - pkg/executor/infoschema_reader_test.go + - pkg/planner/core/integration_test.go + - pkg/planner/core/operator/logicalop/logical_projection.go + - pkg/planner/core/operator/logicalop/logicalop_test/BUILD.bazel + - pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go + - tests/integrationtest/r/planner/core/plan.result + - tests/integrationtest/t/planner/core/plan.test + PR Summary: This is an automated cherry-pick of #60822 What problem does this PR solve? Problem Summary: In the There is a column in topN.ByItems is generated by proj, when topN is pushed down below proj, topN cannot obtain the column from datasource, to fix this, we can check whether topN.ByItems contains a column(with ID=0) generated by proj, if so, proj will prevent the optimizer from pushing topN down. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60692-tidb-reports-can-t-find-column-error-from-v8-5-0-when-there-are-multiple-joins-i.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60692-tidb-reports-can-t-find-column-error-from-v8-5-0-when-there-are-multiple-joins-i.md new file mode 100644 index 0000000..7e93afd --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60692-tidb-reports-can-t-find-column-error-from-v8-5-0-when-there-are-multiple-joins-i.md @@ -0,0 +1,39 @@ +# Issue #60692: tidb reports "Can't find column" error from v8.5.0 when there are multiple joins in the query + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60692 +- Status: closed +- Type: type/bug +- Created At: 2025-04-21T14:31:50Z +- Closed At: 2025-04-23T07:54:57Z +- Labels: affects-8.5, affects-9.0, report/customer, severity/major, sig/planner, type/bug, type/regression + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #60694: planner: add back children's `Schema` when checking `LogicalJoin`'s used columns in column pruning + URL: https://github.com/pingcap/tidb/pull/60694 + State: closed + Merged At: 2025-04-23T07:54:56Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/operator/logicalop/logical_join.go + - tests/integrationtest/r/planner/core/issuetest/planner_issue.result + - tests/integrationtest/t/planner/core/issuetest/planner_issue.test + PR Summary: What problem does this PR solve? Problem Summary: As said in the issue, sometimes the may contain outdated s after the projection elimination. We should have fixed the , but considering the long-standing problems in schema maintenance and column pruning in tidb, we can't make sure we won't introduce any new risk. This PR is likely to be cherry-picked to the LTS version quickly. So it's better to just restore the previous checking logic, i.e., use the to do the check. +- Fix PR #60738: planner: add back children's `Schema` when checking `LogicalJoin`'s used columns in column pruning (#60694) + URL: https://github.com/pingcap/tidb/pull/60738 + State: closed + Merged At: 2025-04-25T03:59:56Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/operator/logicalop/logical_join.go + - tests/integrationtest/r/planner/core/issuetest/planner_issue.result + - tests/integrationtest/t/planner/core/issuetest/planner_issue.test + PR Summary: This is an automated cherry-pick of #60694 What problem does this PR solve? Problem Summary: As said in the issue, sometimes the may contain outdated s after the projection elimination. We should have fixed the , but considering the long-standing problems in schema maintenance and column pruning in tidb, we can't make sure we won't introduce any new risk. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60984-wrongly-enable-plan-cache-with-tiflash.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60984-wrongly-enable-plan-cache-with-tiflash.md new file mode 100644 index 0000000..8f1090c --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60984-wrongly-enable-plan-cache-with-tiflash.md @@ -0,0 +1,17 @@ +# Issue #60984: wrongly enable plan cache with tiflash + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60984 +- Status: open +- Type: type/bug +- Created At: 2025-05-07T03:11:48Z +- Labels: affects-6.1, affects-6.5, report/customer, severity/minor, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60985-tidb-panic-when-using-plan-replayer-dump-with-a-table-using-foreign-keys.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60985-tidb-panic-when-using-plan-replayer-dump-with-a-table-using-foreign-keys.md new file mode 100644 index 0000000..610a119 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-60985-tidb-panic-when-using-plan-replayer-dump-with-a-table-using-foreign-keys.md @@ -0,0 +1,67 @@ +# Issue #60985: TiDB panic when using plan replayer dump with a table using foreign keys + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/60985 +- Status: closed +- Type: type/bug +- Created At: 2025-05-07T03:19:32Z +- Closed At: 2025-05-07T09:07:38Z +- Labels: affects-7.5, affects-8.1, affects-8.5, impact/panic, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #60987: domain: fix the issue where defining FKs in a circular manner causes an infinite loop + URL: https://github.com/pingcap/tidb/pull/60987 + State: closed + Merged At: 2025-05-07T09:07:37Z + Changed Files Count: 2 + Main Modules: pkg/domain, pkg/server + Sample Changed Files: + - pkg/domain/plan_replayer_dump.go + - pkg/server/handler/optimizor/plan_replayer_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Foreign keys cannot form a cycle during the table definition phase. However, it is possible to disable FK checks to force a cycle. When traversing the definitions, we need to skip parts that have already been traversed to avoid an infinite loop +- Fix PR #60998: domain: fix the issue where defining FKs in a circular manner causes an infinite loop (#60987) + URL: https://github.com/pingcap/tidb/pull/60998 + State: closed + Merged At: 2025-07-08T10:30:12Z + Changed Files Count: 2 + Main Modules: pkg/domain, pkg/server + Sample Changed Files: + - pkg/domain/plan_replayer_dump.go + - pkg/server/handler/optimizor/plan_replayer_test.go + PR Summary: This is an automated cherry-pick of #60987 What problem does this PR solve? Problem Summary: What changed and how does it work? Foreign keys cannot form a cycle during the table definition phase. However, it is possible to disable FK checks to force a cycle. When traversing the definitions, we need to skip parts that have already been traversed to avoid an infinite loop +- Fix PR #60999: domain: fix the issue where defining FKs in a circular manner causes an infinite loop (#60987) + URL: https://github.com/pingcap/tidb/pull/60999 + State: open + Merged At: not merged + Changed Files Count: 2 + Main Modules: pkg/domain, pkg/server + Sample Changed Files: + - pkg/domain/plan_replayer_dump.go + - pkg/server/handler/optimizor/plan_replayer_test.go + PR Summary: This is an automated cherry-pick of #60987 What problem does this PR solve? Problem Summary: What changed and how does it work? Foreign keys cannot form a cycle during the table definition phase. However, it is possible to disable FK checks to force a cycle. When traversing the definitions, we need to skip parts that have already been traversed to avoid an infinite loop +- Fix PR #61000: domain: fix the issue where defining FKs in a circular manner causes an infinite loop (#60987) + URL: https://github.com/pingcap/tidb/pull/61000 + State: closed + Merged At: 2025-06-30T04:06:33Z + Changed Files Count: 2 + Main Modules: pkg/domain, pkg/server + Sample Changed Files: + - pkg/domain/plan_replayer_dump.go + - pkg/server/handler/optimizor/plan_replayer_test.go + PR Summary: This is an automated cherry-pick of #60987 What problem does this PR solve? Problem Summary: What changed and how does it work? Foreign keys cannot form a cycle during the table definition phase. However, it is possible to disable FK checks to force a cycle. When traversing the definitions, we need to skip parts that have already been traversed to avoid an infinite loop +- Related PR #61028: domain: fix the issue where defining FKs in a circular manner causes … + URL: https://github.com/pingcap/tidb/pull/61028 + State: closed + Merged At: not merged + Changed Files Count: 2 + Main Modules: pkg/domain, pkg/server + Sample Changed Files: + - pkg/domain/plan_replayer_dump.go + - pkg/server/handler/optimizor/plan_replayer_test.go + PR Summary: …an infinite loop (#60987) close pingcap/tidb#60985 What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61093-index-merge-alternative-picking-should-consider-the-countafteraccess-more-than-t.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61093-index-merge-alternative-picking-should-consider-the-countafteraccess-more-than-t.md new file mode 100644 index 0000000..1b7d471 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61093-index-merge-alternative-picking-should-consider-the-countafteraccess-more-than-t.md @@ -0,0 +1,57 @@ +# Issue #61093: index merge alternative picking should consider the countAfterAccess more than the allSingleIndex limitation + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61093 +- Status: closed +- Type: type/enhancement +- Created At: 2025-05-13T10:25:57Z +- Closed At: 2025-06-05T10:19:42Z +- Labels: affects-8.5, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- phenomenon: The plan cost from the explain format verbose with index merge hint is much lower than the plan without, and while the index merge plan wasn't be picked. + +## Linked PRs +- Related PR #61372: planner: fix index merge skyline pruning may be prior to choose distinct partial index rather than the low count one. + URL: https://github.com/pingcap/tidb/pull/61372 + State: closed + Merged At: 2025-06-05T10:19:41Z + Changed Files Count: 12 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/find_best_task.go + - pkg/planner/core/logical_plans_test.go + - pkg/planner/core/testdata/index_merge_suite_out.json + - tests/integrationtest/r/explain_indexmerge_stats.result + - tests/integrationtest/r/index_merge.result + - tests/integrationtest/r/planner/core/casetest/index/index.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/indexmerge_path.result + - tests/integrationtest/r/planner/core/issuetest/planner_issue.result + - tests/integrationtest/s.zip + - tests/integrationtest/t/explain_indexmerge_stats.test + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #61562: planner: fix index merge skyline pruning may be prior to choose distinct partial index rather than the low count one. (#61372) + URL: https://github.com/pingcap/tidb/pull/61562 + State: closed + Merged At: 2025-06-25T09:41:21Z + Changed Files Count: 12 + Main Modules: tests/integrationtest, pkg/planner/core, tests/realtikvtest, pkg/disttask + Sample Changed Files: + - pkg/disttask/framework/scheduler/scheduler_manager.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/logical_plans_test.go + - pkg/planner/core/testdata/index_merge_suite_out.json + - tests/integrationtest/r/explain_indexmerge_stats.result + - tests/integrationtest/r/index_merge.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/issuetest/planner_issue.result + - tests/integrationtest/s.zip + - tests/integrationtest/t/explain_indexmerge_stats.test + - tests/realtikvtest/addindextest1/BUILD.bazel + - tests/realtikvtest/addindextest1/disttask_test.go + PR Summary: This is an automated cherry-pick of #61372 What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61305-full-table-scan-hash-join-index-join-when-using-select-distinct-constant-instead.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61305-full-table-scan-hash-join-index-join-when-using-select-distinct-constant-instead.md new file mode 100644 index 0000000..6398835 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61305-full-table-scan-hash-join-index-join-when-using-select-distinct-constant-instead.md @@ -0,0 +1,42 @@ +# Issue #61305: Full table scan / Hash Join+Index Join when using select distinct constant, instead of column + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61305 +- Status: closed +- Type: type/bug +- Created At: 2025-05-24T19:41:48Z +- Closed At: 2025-09-01T06:55:26Z +- Labels: affects-6.1, affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, planner/cascades, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- Added Hash Join and Index Join (to do non-needed left outer join?) When it should been more similar to: + +## Linked PRs +- Fix PR #61478: planner: outer join pruning for constants + URL: https://github.com/pingcap/tidb/pull/61478 + State: closed + Merged At: 2025-06-06T14:23:46Z + Changed Files Count: 4 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/plan_test.go + - pkg/planner/core/rule/util/misc.go + - pkg/planner/core/rule_join_elimination.go + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Issue 61305 exposed limitations in the outer join pruning if constants are in the select list. If a constant is in the select list, then we don't recognize that this column is irrelvant - and thus if you ONLY otherwise have columns from the left side, we don't prune right side tables due to the constant. Enhanced outer join pruning to cover these cases. +- Fix PR #62166: planner: outer join pruning for constants (#61478) + URL: https://github.com/pingcap/tidb/pull/62166 + State: closed + Merged At: 2025-07-03T17:46:42Z + Changed Files Count: 5 + Main Modules: pkg/planner/core, pkg/bindinfo, tests/integrationtest + Sample Changed Files: + - pkg/bindinfo/BUILD.bazel + - pkg/planner/core/casetest/BUILD.bazel + - pkg/planner/core/casetest/plan_test.go + - pkg/planner/core/rule_join_elimination.go + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + PR Summary: This is an automated cherry-pick of #61478 What problem does this PR solve? Problem Summary: What changed and how does it work? Issue 61305 exposed limitations in the outer join pruning if constants are in the select list. If a constant is in the select list, then we don't recognize that this column is irrelvant - and thus if you ONLY otherwise have columns from the left side, we don't prune right side tables due to the constant. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61339-totalcount-spends-too-much-time.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61339-totalcount-spends-too-much-time.md new file mode 100644 index 0000000..ce9a333 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61339-totalcount-spends-too-much-time.md @@ -0,0 +1,57 @@ +# Issue #61339: TotalCount spends too much time + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61339 +- Status: closed +- Type: type/enhancement +- Created At: 2025-05-27T03:06:40Z +- Closed At: 2025-05-28T02:10:56Z +- Labels: affects-8.5, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- The issue is as shown in this flame graph. + +## Linked PRs +- Fix PR #61340: statistics: the totalCount/minCount of TopN is calculated only once + URL: https://github.com/pingcap/tidb/pull/61340 + State: closed + Merged At: 2025-05-28T02:10:55Z + Changed Files Count: 3 + Main Modules: pkg/statistics + Sample Changed Files: + - pkg/statistics/builder.go + - pkg/statistics/cmsketch.go + - pkg/statistics/cmsketch_test.go + PR Summary: What problem does this PR solve? Problem Summary: The issue is as shown in this flame graph. You can see that calculating the total for topn takes a lot of time, because it needs to traverse topn each time to compute the total count. This is very time-consuming. Moreover, the result of each calculation is the same, so you can cache this result to avoid performance issues. What changed and how does it work? +- Fix PR #61371: statistics: the totalCount/minCount of TopN is calculated only once (#61340) + URL: https://github.com/pingcap/tidb/pull/61371 + State: closed + Merged At: 2025-07-01T06:17:57Z + Changed Files Count: 3 + Main Modules: pkg/statistics + Sample Changed Files: + - pkg/statistics/builder.go + - pkg/statistics/cmsketch.go + - pkg/statistics/cmsketch_test.go + PR Summary: This is an automated cherry-pick of #61340 What problem does this PR solve? Problem Summary: The issue is as shown in this flame graph. You can see that calculating the total for topn takes a lot of time, because it needs to traverse topn each time to compute the total count. This is very time-consuming. Moreover, the result of each calculation is the same, so you can cache this result to avoid performance issues. +- Fix PR #61421: planner: Only count all topn when necessary + URL: https://github.com/pingcap/tidb/pull/61421 + State: closed + Merged At: 2025-06-03T18:11:35Z + Changed Files Count: 1 + Main Modules: pkg/statistics + Sample Changed Files: + - pkg/statistics/builder.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? This PR does the following in addition to a prior PR targeted at this issue: 1) It removes the need to accumulate the count of all topN for the purposes of determining if we have all rows in the TopN. +- Fix PR #61996: planner: Only count all topn when necessary (#61421) + URL: https://github.com/pingcap/tidb/pull/61996 + State: closed + Merged At: 2025-07-02T17:06:03Z + Changed Files Count: 1 + Main Modules: pkg/statistics + Sample Changed Files: + - pkg/statistics/builder.go + PR Summary: This is an automated cherry-pick of #61421 What problem does this PR solve? Problem Summary: What changed and how does it work? This PR does the following in addition to a prior PR targeted at this issue: + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61501-planner-support-a-hint-or-var-like-index-join-first-to-indicate-the-planner-to-p.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61501-planner-support-a-hint-or-var-like-index-join-first-to-indicate-the-planner-to-p.md new file mode 100644 index 0000000..d91e9d5 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61501-planner-support-a-hint-or-var-like-index-join-first-to-indicate-the-planner-to-p.md @@ -0,0 +1,34 @@ +# Issue #61501: planner: support a hint or var like `index-join-first` to indicate the planner to prefer `IndexJoin` over other joins + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61501 +- Status: open +- Type: type/enhancement +- Created At: 2025-06-05T01:05:18Z +- Labels: report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- We've met this problems multiple times, so it might be better to support a hint or variable like to indicate the optimizer always choose if possible in such scenarios: + +## Linked PRs +- Fix PR #66596: planner: support index_join_first() hint to prefer index join + URL: https://github.com/pingcap/tidb/pull/66596 + State: open + Merged At: not merged + Changed Files Count: 10 + Main Modules: pkg/planner/core, pkg/parser, pkg/util + Sample Changed Files: + - pkg/parser/ast/misc.go + - pkg/parser/hintparser.go + - pkg/parser/hintparser.y + - pkg/parser/misc.go + - pkg/planner/core/casetest/hint/BUILD.bazel + - pkg/planner/core/casetest/hint/hint_test.go + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/rule_join_reorder.go + - pkg/util/hint/hint.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? support index_join_first() hint to prefer index join + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61565-internal-sql-wrongly-use-get-plan-cost-v1.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61565-internal-sql-wrongly-use-get-plan-cost-v1.md new file mode 100644 index 0000000..da5ad94 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61565-internal-sql-wrongly-use-get-plan-cost-v1.md @@ -0,0 +1,89 @@ +# Issue #61565: internal SQL wrongly use get plan cost v1 + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61565 +- Status: closed +- Type: type/bug +- Created At: 2025-06-06T11:12:24Z +- Closed At: 2025-06-10T05:22:29Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- When you enable the flame graph, you will find that some of the internal SQL stacks have stacks that call , which is clearly incorrect. ![Image]() + +## Linked PRs +- Fix PR #61566: session: set right cost model version + URL: https://github.com/pingcap/tidb/pull/61566 + State: closed + Merged At: not merged + Changed Files Count: 1 + Main Modules: pkg/session + Sample Changed Files: + - pkg/session/session.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #61608: planner: set the default of the tidb_cost_model_version correctly + URL: https://github.com/pingcap/tidb/pull/61608 + State: closed + Merged At: 2025-06-10T05:22:28Z + Changed Files Count: 3 + Main Modules: pkg/planner/core, pkg/session, pkg/util + Sample Changed Files: + - pkg/planner/core/plan_cost_ver1.go + - pkg/sessionctx/variable/session.go + - pkg/util/mock/context.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? We don't set the default of the tidb_cost_model_version in the right place. so it led many problems in the internal SQL. The internal SQL will not use the cost model v2. +- Fix PR #61895: planner: set the default of the tidb_cost_model_version correctly (#61608) + URL: https://github.com/pingcap/tidb/pull/61895 + State: closed + Merged At: 2025-07-15T22:58:57Z + Changed Files Count: 2 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/plan_cost_ver1.go + - pkg/sessionctx/variable/session.go + PR Summary: This is an automated cherry-pick of #61608 What problem does this PR solve? Problem Summary: What changed and how does it work? We don't set the default of the tidb_cost_model_version in the right place. so it led many problems in the internal SQL. The internal SQL will not use the cost model v2. +- Fix PR #61896: planner: set the default of the tidb_cost_model_version correctly (#61608) + URL: https://github.com/pingcap/tidb/pull/61896 + State: open + Merged At: not merged + Changed Files Count: 2 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/plan_cost_ver1.go + - pkg/sessionctx/variable/session.go + PR Summary: This is an automated cherry-pick of #61608 What problem does this PR solve? Problem Summary: What changed and how does it work? We don't set the default of the tidb_cost_model_version in the right place. so it led many problems in the internal SQL. The internal SQL will not use the cost model v2. +- Fix PR #61897: planner: set the default of the tidb_cost_model_version correctly (#61608) + URL: https://github.com/pingcap/tidb/pull/61897 + State: closed + Merged At: 2025-07-09T16:05:03Z + Changed Files Count: 3 + Main Modules: pkg/planner/core, pkg/session, pkg/util + Sample Changed Files: + - pkg/planner/core/plan_cost_ver1.go + - pkg/sessionctx/variable/session.go + - pkg/util/mock/context.go + PR Summary: This is an automated cherry-pick of #61608 What problem does this PR solve? Problem Summary: What changed and how does it work? We don't set the default of the tidb_cost_model_version in the right place. so it led many problems in the internal SQL. The internal SQL will not use the cost model v2. +- Fix PR #62184: planner: set the default of the tidb_cost_model_version correctly (#61608) + URL: https://github.com/pingcap/tidb/pull/62184 + State: closed + Merged At: 2025-07-04T01:23:18Z + Changed Files Count: 2 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/plan_cost_ver1.go + - pkg/sessionctx/variable/session.go + PR Summary: This is an automated cherry-pick of #61608 What problem does this PR solve? Problem Summary: What changed and how does it work? We don't set the default of the tidb_cost_model_version in the right place. so it led many problems in the internal SQL. The internal SQL will not use the cost model v2. +- Fix PR #63799: planner: set the default of the tidb_cost_model_version correctly (#61608) | tidb-test=release-8.5.3 + URL: https://github.com/pingcap/tidb/pull/63799 + State: closed + Merged At: 2025-09-29T14:48:22Z + Changed Files Count: 3 + Main Modules: pkg/planner/core, pkg/session, pkg/util + Sample Changed Files: + - pkg/planner/core/plan_cost_ver1.go + - pkg/sessionctx/variable/session.go + - pkg/util/mock/context.go + PR Summary: This is an automated cherry-pick of #61608 What problem does this PR solve? Problem Summary: What changed and how does it work? We don't set the default of the tidb_cost_model_version in the right place. so it led many problems in the internal SQL. The internal SQL will not use the cost model v2. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61568-planner-remove-duplicated-predicates-in-cnf-or-dnf.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61568-planner-remove-duplicated-predicates-in-cnf-or-dnf.md new file mode 100644 index 0000000..c48847e --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61568-planner-remove-duplicated-predicates-in-cnf-or-dnf.md @@ -0,0 +1,17 @@ +# Issue #61568: planner: remove duplicated predicates in CNF or DNF + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61568 +- Status: open +- Type: type/enhancement +- Created At: 2025-06-07T03:11:17Z +- Labels: report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- The DNF in this query is generated by code, so there might be some duplicated predicates: . Since there are 3 duplicated predicates, we have to execute the same three times. The optimizer should remove these duplicated predicates automatically, and the final optimal plan should be: + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61602-planner-wrong-join-estimation-depending-on-uninitialized-or-missing-stats.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61602-planner-wrong-join-estimation-depending-on-uninitialized-or-missing-stats.md new file mode 100644 index 0000000..f0ce5e1 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61602-planner-wrong-join-estimation-depending-on-uninitialized-or-missing-stats.md @@ -0,0 +1,93 @@ +# Issue #61602: planner: wrong Join estimation depending on uninitialized or missing stats + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61602 +- Status: closed +- Type: type/bug +- Created At: 2025-06-09T11:14:05Z +- Closed At: 2025-06-11T11:46:21Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, epic/cardinality-estimation, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- Root Cause: 1. In join estimation, we use . 2. When deriving NDV from index-stats, we didn't check whether it was loaded or missing. After adding a new index without analyzing, the index-stats might be missing, 3. If it's missing, we'll use the wrong to calculate join estimation, and finally get a high error result. + +## Linked PRs +- Fix PR #61604: planner: fix the wrong join estimation depending on missing or uninitialized stats + URL: https://github.com/pingcap/tidb/pull/61604 + State: closed + Merged At: 2025-06-11T11:46:19Z + Changed Files Count: 5 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/stats.go + - tests/integrationtest/r/executor/index_lookup_merge_join.result + - tests/integrationtest/r/explain_complex.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/indexjoin.result + PR Summary: What problem does this PR solve? Problem Summary: planner: fix the wrong join estimation depending on missing or uninitialized stats What changed and how does it work? planner: fix the wrong join estimation depending on missing or uninitialized stats It's hard to construct test cases for this issue since it depends on stats cache's status. So I tested it locally, and this PR can work for this scenario: +- Fix PR #61673: planner: fix the wrong join estimation depending on missing or uninitialized stats (#61604) + URL: https://github.com/pingcap/tidb/pull/61673 + State: open + Merged At: not merged + Changed Files Count: 5 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/stats.go + - tests/integrationtest/r/executor/index_lookup_merge_join.result + - tests/integrationtest/r/explain_complex.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/indexjoin.result + PR Summary: This is an automated cherry-pick of #61604 What problem does this PR solve? Problem Summary: planner: fix the wrong join estimation depending on missing or uninitialized stats What changed and how does it work? planner: fix the wrong join estimation depending on missing or uninitialized stats +- Fix PR #61674: planner: fix the wrong join estimation depending on missing or uninitialized stats (#61604) + URL: https://github.com/pingcap/tidb/pull/61674 + State: closed + Merged At: not merged + Changed Files Count: 5 + Main Modules: tests/integrationtest, cmd/explaintest, pkg/planner/core + Sample Changed Files: + - cmd/explaintest/r/explain_complex.result + - pkg/planner/core/stats.go + - tests/integrationtest/r/executor/index_lookup_merge_join.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/indexjoin.result + PR Summary: This is an automated cherry-pick of #61604 What problem does this PR solve? Problem Summary: planner: fix the wrong join estimation depending on missing or uninitialized stats What changed and how does it work? planner: fix the wrong join estimation depending on missing or uninitialized stats +- Fix PR #61675: planner: fix the wrong join estimation depending on missing or uninitialized stats (#61604) + URL: https://github.com/pingcap/tidb/pull/61675 + State: open + Merged At: not merged + Changed Files Count: 5 + Main Modules: tests/integrationtest, cmd/explaintest, pkg/planner/core + Sample Changed Files: + - cmd/explaintest/r/explain_complex.result + - pkg/planner/core/stats.go + - tests/integrationtest/r/executor/index_lookup_merge_join.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/indexjoin.result + PR Summary: This is an automated cherry-pick of #61604 What problem does this PR solve? Problem Summary: planner: fix the wrong join estimation depending on missing or uninitialized stats What changed and how does it work? planner: fix the wrong join estimation depending on missing or uninitialized stats +- Fix PR #61676: planner: fix the wrong join estimation depending on missing or uninitialized stats (#61604) + URL: https://github.com/pingcap/tidb/pull/61676 + State: closed + Merged At: 2025-07-22T02:43:20Z + Changed Files Count: 3 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_out.json + - pkg/planner/core/stats.go + - tests/integrationtest/r/explain_complex.result + PR Summary: This is an automated cherry-pick of #61604 What problem does this PR solve? Problem Summary: planner: fix the wrong join estimation depending on missing or uninitialized stats What changed and how does it work? planner: fix the wrong join estimation depending on missing or uninitialized stats +- Fix PR #61857: planner: fix the wrong join estimation depending on missing or uninitialized stats (#61604) + URL: https://github.com/pingcap/tidb/pull/61857 + State: closed + Merged At: 2025-07-09T09:18:42Z + Changed Files Count: 5 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/stats.go + - tests/integrationtest/r/executor/index_lookup_merge_join.result + - tests/integrationtest/r/explain_complex.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/indexjoin.result + PR Summary: This is an automated cherry-pick of #61604 What problem does this PR solve? Problem Summary: planner: fix the wrong join estimation depending on missing or uninitialized stats What changed and how does it work? planner: fix the wrong join estimation depending on missing or uninitialized stats + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61606-tidb-s-analyze-command-cannot-correctly-handle-a-virtual-generated-column-if-it-.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61606-tidb-s-analyze-command-cannot-correctly-handle-a-virtual-generated-column-if-it-.md new file mode 100644 index 0000000..59d0630 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61606-tidb-s-analyze-command-cannot-correctly-handle-a-virtual-generated-column-if-it-.md @@ -0,0 +1,83 @@ +# Issue #61606: TiDB's analyze command cannot correctly handle a virtual generated column if it is the first column + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61606 +- Status: closed +- Type: type/bug +- Created At: 2025-06-09T14:03:32Z +- Closed At: 2025-07-10T07:53:53Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, component/statistics, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Create the table: 2. Insert some data: 3. Analyze the table: + +## Linked PRs +- Fix PR #62333: executor: fix the issue during analyze when first col is virtual col + URL: https://github.com/pingcap/tidb/pull/62333 + State: closed + Merged At: 2025-07-10T07:53:48Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/executor + Sample Changed Files: + - pkg/executor/analyze_col_v2.go + - tests/integrationtest/r/executor/analyze.result + - tests/integrationtest/t/executor/analyze.test + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? The samples should append for the virtual generated column first for later use. +- Fix PR #62348: executor: fix the issue during analyze when first col is virtual col (#62333) + URL: https://github.com/pingcap/tidb/pull/62348 + State: closed + Merged At: 2025-07-11T16:59:03Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/executor + Sample Changed Files: + - pkg/executor/analyze_col_v2.go + - tests/integrationtest/r/executor/analyze.result + - tests/integrationtest/t/executor/analyze.test + PR Summary: This is an automated cherry-pick of #62333 What problem does this PR solve? Problem Summary: What changed and how does it work? The samples should append for the virtual generated column first for later use. +- Fix PR #62370: executor: fix the issue during analyze when first col is virtual col (#62333) + URL: https://github.com/pingcap/tidb/pull/62370 + State: open + Merged At: not merged + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/executor + Sample Changed Files: + - pkg/executor/analyze_col_v2.go + - tests/integrationtest/r/executor/analyze.result + - tests/integrationtest/t/executor/analyze.test + PR Summary: This is an automated cherry-pick of #62333 What problem does this PR solve? Problem Summary: What changed and how does it work? The samples should append for the virtual generated column first for later use. +- Fix PR #62371: executor: fix the issue during analyze when first col is virtual col (#62333) + URL: https://github.com/pingcap/tidb/pull/62371 + State: closed + Merged At: 2025-07-16T13:16:00Z + Changed Files Count: 3 + Main Modules: pkg/executor, pkg/util + Sample Changed Files: + - pkg/executor/analyze_col_v2.go + - pkg/executor/test/analyzetest/analyze_test.go + - pkg/util/chunk/column.go + PR Summary: This is an automated cherry-pick of #62333 What problem does this PR solve? Problem Summary: What changed and how does it work? The samples should append for the virtual generated column first for later use. +- Fix PR #62372: executor: fix the issue during analyze when first col is virtual col (#62333) + URL: https://github.com/pingcap/tidb/pull/62372 + State: open + Merged At: not merged + Changed Files Count: 3 + Main Modules: executor/analyze_col_v2.go, executor/testdata, tests/integrationtest + Sample Changed Files: + - executor/analyze_col_v2.go + - executor/testdata/analyze_test_data.sql + - tests/integrationtest/r/executor/analyze.result + PR Summary: This is an automated cherry-pick of #62333 What problem does this PR solve? Problem Summary: What changed and how does it work? The samples should append for the virtual generated column first for later use. +- Fix PR #62373: executor: fix the issue during analyze when first col is virtual col (#62333) + URL: https://github.com/pingcap/tidb/pull/62373 + State: closed + Merged At: not merged + Changed Files Count: 3 + Main Modules: executor/analyze_col_v2.go, executor/testdata, tests/integrationtest + Sample Changed Files: + - executor/analyze_col_v2.go + - executor/testdata/analyze_test_data.sql + - tests/integrationtest/r/executor/analyze.result + PR Summary: This is an automated cherry-pick of #62333 What problem does this PR solve? Problem Summary: What changed and how does it work? The samples should append for the virtual generated column first for later use. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61669-planner-invalid-column-error-when-building-indexjoin-for-sub-query-with-multiple.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61669-planner-invalid-column-error-when-building-indexjoin-for-sub-query-with-multiple.md new file mode 100644 index 0000000..13ddcb1 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61669-planner-invalid-column-error-when-building-indexjoin-for-sub-query-with-multiple.md @@ -0,0 +1,37 @@ +# Issue #61669: planner: invalid column error when building IndexJoin for sub-query with multiple Agg + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61669 +- Status: closed +- Type: type/bug +- Created At: 2025-06-11T10:02:55Z +- Closed At: 2025-06-11T11:02:02Z +- Labels: affects-8.5, report/customer, severity/major, sig/planner, type/bug, type/regression + +## Customer-Facing Phenomenon +- Root Cause: 1. An optimizer bug, when constructing the inner operators under an IndexJoin in , we assume there is at most Agg. 2. In this SQL with 2 nested sub-queries, there are Agg under IndexJoin, which breaks our assumption, and causes some errors. + +## Linked PRs +- Fix PR #61672: planner: fix the issue that invalid column error when building IndexJoin for sub-query with multiple Agg + URL: https://github.com/pingcap/tidb/pull/61672 + State: closed + Merged At: 2025-06-11T11:02:01Z + Changed Files Count: 2 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/integration_test.go + PR Summary: What problem does this PR solve? Problem Summary: planner: fix the issue that invalid column error when building IndexJoin for sub-query with multiple Agg What changed and how does it work? planner: fix the issue that invalid column error when building IndexJoin for sub-query with multiple Agg +- Fix PR #61727: planner: fix the issue that invalid column error when building IndexJoin for sub-query with multiple Agg (#61672) + URL: https://github.com/pingcap/tidb/pull/61727 + State: closed + Merged At: 2025-06-26T06:57:39Z + Changed Files Count: 2 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/integration_test.go + PR Summary: This is an automated cherry-pick of #61672 What problem does this PR solve? Problem Summary: planner: fix the issue that invalid column error when building IndexJoin for sub-query with multiple Agg What changed and how does it work? planner: fix the issue that invalid column error when building IndexJoin for sub-query with multiple Agg + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61716-avoid-indexmerge-double-read-for-multi-valued-index.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61716-avoid-indexmerge-double-read-for-multi-valued-index.md new file mode 100644 index 0000000..479d6f5 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61716-avoid-indexmerge-double-read-for-multi-valued-index.md @@ -0,0 +1,47 @@ +# Issue #61716: Avoid IndexMerge double-read for multi-valued index + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61716 +- Status: open +- Type: type/enhancement +- Created At: 2025-06-13T03:00:51Z +- Labels: affects-7.1, affects-7.5, affects-8.1, affects-8.5, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- The multi-valued index uses operator to fetch data. However, if a composite index can cover needed columns, the operator should avoid reading data(TableRowIDScan) that introduces unnecessary overhead. + +## Linked PRs +- Fix PR #66952: executor,planner: add index-only index-merge path for MVIndex queries + URL: https://github.com/pingcap/tidb/pull/66952 + State: closed + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/executor, pkg/planner/core + Sample Changed Files: + - pkg/executor/builder.go + - pkg/executor/index_merge_reader.go + - pkg/executor/test/indexmergereadtest/BUILD.bazel + - pkg/executor/test/indexmergereadtest/index_merge_reader_test.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/flat_plan.go + - pkg/planner/core/operator/physicalop/physical_indexmerge_reader.go + - pkg/planner/core/operator/physicalop/task_base.go + PR Summary: What problem does this PR solve? Problem Summary: executor,planner: add index-only index-merge path for MVIndex queries What changed and how does it work? Summary Add an index-only fast path for when accessing MVIndex in eligible queries. +- Fix PR #66996: planner: enable index-only MV IndexMerge only for single covered partial path (optimizer part) + URL: https://github.com/pingcap/tidb/pull/66996 + State: open + Merged At: not merged + Changed Files Count: 7 + Main Modules: pkg/planner/core, pkg/executor + Sample Changed Files: + - pkg/executor/builder.go + - pkg/executor/test/indexmergereadtest/BUILD.bazel + - pkg/executor/test/indexmergereadtest/index_merge_reader_test.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/flat_plan.go + - pkg/planner/core/operator/physicalop/physical_indexmerge_reader.go + - pkg/planner/core/optimizer.go + PR Summary: What problem does this PR solve? Problem Summary: planner: enable index-only MV IndexMerge only for single covered partial path (optimizer part) What changed and how does it work? This PR improves optimizer behavior for MV IndexMerge by allowing index-only plans only when both conditions are met: 1. IndexMerge has exactly one partial MV index path. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61785-analyze-s-analyzecolumnspushdownv2-create-too-much-concurrency.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61785-analyze-s-analyzecolumnspushdownv2-create-too-much-concurrency.md new file mode 100644 index 0000000..5cce9fe --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61785-analyze-s-analyzecolumnspushdownv2-create-too-much-concurrency.md @@ -0,0 +1,53 @@ +# Issue #61785: Analyze's analyzeColumnsPushDownV2 create too much concurrency + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61785 +- Status: closed +- Type: type/bug +- Created At: 2025-06-17T20:15:49Z +- Closed At: 2025-06-18T14:59:31Z +- Labels: affects-7.5, affects-8.1, affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- The issue is that customers have observed higher I/O consumption when the analyze operation reaches the index, compared to when it analyzes regular tables. (The analyze status contains sensitive information, so it will not be included here.) ![Image]() The root cause of the issue lies in improper coding practices. When we perform the analyze operation, we create multiple concurrent tasks to execute it. However, within these concurrently spawned goroutines, we further create additional concurrency. This nested concurrency results in an actual level of parallelism that is significantly higher than we anticipated. You will see that it will create two task about . + +## Linked PRs +- Fix PR #61786: planner: avoid exceeding the configured concurrency limit + URL: https://github.com/pingcap/tidb/pull/61786 + State: closed + Merged At: 2025-06-18T14:59:30Z + Changed Files Count: 1 + Main Modules: pkg/executor + Sample Changed Files: + - pkg/executor/analyze_col_v2.go + PR Summary: What problem does this PR solve? Problem Summary: The issue is that customers have observed higher I/O consumption when the analyze operation reaches the index, compared to when it analyzes regular tables. (The analyze status contains sensitive information, so it will not be included here.) ![Image]() The root cause of the issue lies in improper coding practices. When we perform the analyze operation, we create multiple concurrent tasks to execute it. However, within these concurrently spawned goroutines, we further create additional concurrency. This nested concurrency results in an actual level of parallelism that is significantly higher than we anticipated. +- Fix PR #61813: planner: avoid exceeding the configured concurrency limit (#61786) + URL: https://github.com/pingcap/tidb/pull/61813 + State: closed + Merged At: 2025-07-09T00:43:15Z + Changed Files Count: 1 + Main Modules: pkg/executor + Sample Changed Files: + - pkg/executor/analyze_col_v2.go + PR Summary: This is an automated cherry-pick of #61786 What problem does this PR solve? Problem Summary: The issue is that customers have observed higher I/O consumption when the analyze operation reaches the index, compared to when it analyzes regular tables. (The analyze status contains sensitive information, so it will not be included here.) ![Image]() +- Fix PR #61814: planner: avoid exceeding the configured concurrency limit (#61786) + URL: https://github.com/pingcap/tidb/pull/61814 + State: open + Merged At: not merged + Changed Files Count: 1 + Main Modules: pkg/executor + Sample Changed Files: + - pkg/executor/analyze_col_v2.go + PR Summary: This is an automated cherry-pick of #61786 What problem does this PR solve? Problem Summary: The issue is that customers have observed higher I/O consumption when the analyze operation reaches the index, compared to when it analyzes regular tables. (The analyze status contains sensitive information, so it will not be included here.) ![Image]() +- Fix PR #61815: planner: avoid exceeding the configured concurrency limit (#61786) + URL: https://github.com/pingcap/tidb/pull/61815 + State: closed + Merged At: 2025-06-25T11:09:54Z + Changed Files Count: 1 + Main Modules: pkg/executor + Sample Changed Files: + - pkg/executor/analyze_col_v2.go + PR Summary: This is an automated cherry-pick of #61786 What problem does this PR solve? Problem Summary: The issue is that customers have observed higher I/O consumption when the analyze operation reaches the index, compared to when it analyzes regular tables. (The analyze status contains sensitive information, so it will not be included here.) ![Image]() + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61822-ttl-scan-cannot-trigger-sync-async-load-generateruntimefilter.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61822-ttl-scan-cannot-trigger-sync-async-load-generateruntimefilter.md new file mode 100644 index 0000000..45a5616 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61822-ttl-scan-cannot-trigger-sync-async-load-generateruntimefilter.md @@ -0,0 +1,72 @@ +# Issue #61822: TTL scan cannot trigger sync/async load/generateRuntimeFilter + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61822 +- Status: closed +- Type: type/bug +- Created At: 2025-06-19T01:18:13Z +- Closed At: 2025-07-30T17:18:17Z +- Labels: affects-7.1, affects-7.5, affects-8.1, affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- We previously thought that internal SQL was for system tables,and since system tables do not have statistics,we could skip the sync load.However,TTL is different because it generates SQL execution based on user tables.Therefore,we need to align its behavior with regular tables. + +## Linked PRs +- Fix PR #62616: planner: TTL scan can trigger sync/async load/generateRuntimeFilter + URL: https://github.com/pingcap/tidb/pull/62616 + State: closed + Merged At: 2025-07-30T17:18:16Z + Changed Files Count: 6 + Main Modules: pkg/planner/core, pkg/ttl, pkg/session + Sample Changed Files: + - pkg/planner/core/collect_column_stats_usage.go + - pkg/planner/core/optimizer.go + - pkg/planner/core/rule_collect_plan_stats.go + - pkg/sessionctx/variable/session.go + - pkg/ttl/ttlworker/session.go + - pkg/ttl/ttlworker/session_integration_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? The TTL scan is different from our other internal SQL, as it reads user tables. So, our previous approach of determining whether to perform sync load based on whether it is internal SQL has issues. +- Fix PR #62729: planner: TTL scan can trigger sync/async load/generateRuntimeFilter (#62616) + URL: https://github.com/pingcap/tidb/pull/62729 + State: open + Merged At: not merged + Changed Files Count: 6 + Main Modules: pkg/planner/core, pkg/ttl, pkg/session + Sample Changed Files: + - pkg/planner/core/collect_column_stats_usage.go + - pkg/planner/core/optimizer.go + - pkg/planner/core/plan_stats.go + - pkg/sessionctx/variable/session.go + - pkg/ttl/ttlworker/session.go + - pkg/ttl/ttlworker/session_integration_test.go + PR Summary: This is an automated cherry-pick of #62616 What problem does this PR solve? Problem Summary: What changed and how does it work? The TTL scan is different from our other internal SQL, as it reads user tables. So, our previous approach of determining whether to perform sync load based on whether it is internal SQL has issues. +- Fix PR #62730: planner: TTL scan can trigger sync/async load/generateRuntimeFilter (#62616) + URL: https://github.com/pingcap/tidb/pull/62730 + State: open + Merged At: not merged + Changed Files Count: 6 + Main Modules: pkg/planner/core, pkg/ttl, pkg/session + Sample Changed Files: + - pkg/planner/core/collect_column_stats_usage.go + - pkg/planner/core/optimizer.go + - pkg/planner/core/rule_collect_plan_stats.go + - pkg/sessionctx/variable/session.go + - pkg/ttl/ttlworker/session.go + - pkg/ttl/ttlworker/session_integration_test.go + PR Summary: This is an automated cherry-pick of #62616 What problem does this PR solve? Problem Summary: What changed and how does it work? The TTL scan is different from our other internal SQL, as it reads user tables. So, our previous approach of determining whether to perform sync load based on whether it is internal SQL has issues. +- Fix PR #62731: planner: TTL scan can trigger sync/async load (#62616) + URL: https://github.com/pingcap/tidb/pull/62731 + State: closed + Merged At: 2025-09-19T04:59:48Z + Changed Files Count: 5 + Main Modules: pkg/planner/core, pkg/session, pkg/ttl + Sample Changed Files: + - pkg/planner/core/collect_column_stats_usage.go + - pkg/planner/core/optimizer.go + - pkg/planner/core/rule_collect_plan_stats.go + - pkg/sessionctx/variable/session.go + - pkg/ttl/ttlworker/scan.go + PR Summary: This is an automated cherry-pick of #62616 What problem does this PR solve? Problem Summary: What changed and how does it work? The TTL scan is different from our other internal SQL, as it reads user tables. So, our previous approach of determining whether to perform sync load based on whether it is internal SQL has issues. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61889-tidb-load-binding-timeout-cannot-rightly-init-the-default-value.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61889-tidb-load-binding-timeout-cannot-rightly-init-the-default-value.md new file mode 100644 index 0000000..678c1ec --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61889-tidb-load-binding-timeout-cannot-rightly-init-the-default-value.md @@ -0,0 +1,50 @@ +# Issue #61889: tidb_load_binding_timeout cannot rightly init the default value + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61889 +- Status: closed +- Type: type/bug +- Created At: 2025-06-20T09:09:06Z +- Closed At: 2025-06-23T04:21:49Z +- Labels: affects-8.1, affects-8.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- As the log shows, The duration is . it is not expected. and the user doesn't update this value. so I add a assert into the problem place and CI report the panic at here. This indicates that there indeed is an issue here. + +## Linked PRs +- Fix PR #61891: planner: fix uninit timeout for loading bindings + URL: https://github.com/pingcap/tidb/pull/61891 + State: closed + Merged At: 2025-06-23T04:21:48Z + Changed Files Count: 2 + Main Modules: pkg/session + Sample Changed Files: + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? as the code show, the old code cannot init the right value for Currently,this switch is actually ineffective on the master branch.However,to comply with the process standards,it will still be merged into master first.But please refer to [this PR]() for the tests. +- Fix PR #61926: planner: fix uninit timeout for loading bindings (#61891) + URL: https://github.com/pingcap/tidb/pull/61926 + State: open + Merged At: not merged + Changed Files Count: 3 + Main Modules: pkg/session, pkg/bindinfo + Sample Changed Files: + - pkg/bindinfo/global_handle.go + - pkg/sessionctx/variable/sysvar.go + - pkg/sessionctx/variable/tidb_vars.go + PR Summary: This is an automated cherry-pick of #61891 What problem does this PR solve? Problem Summary: What changed and how does it work? as the code show, the old code cannot init the right value for +- Fix PR #61928: planner: fix uninit timeout for loading bindings (#61891) + URL: https://github.com/pingcap/tidb/pull/61928 + State: closed + Merged At: 2025-07-09T16:05:10Z + Changed Files Count: 4 + Main Modules: pkg/session, pkg/bindinfo + Sample Changed Files: + - pkg/bindinfo/global_handle.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + - pkg/sessionctx/variable/tidb_vars.go + PR Summary: This is an automated cherry-pick of #61891 What problem does this PR solve? Problem Summary: What changed and how does it work? as the code show, the old code cannot init the right value for + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61990-planner-index-predicates-based-on-column-prefix-are-not-pushed-down-to-index-sid.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61990-planner-index-predicates-based-on-column-prefix-are-not-pushed-down-to-index-sid.md new file mode 100644 index 0000000..5db00ea --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-61990-planner-index-predicates-based-on-column-prefix-are-not-pushed-down-to-index-sid.md @@ -0,0 +1,17 @@ +# Issue #61990: planner: index predicates based on column prefix are not pushed down to index side if they can't be used to construct scan ranges + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/61990 +- Status: open +- Type: type/question +- Created At: 2025-06-25T06:34:43Z +- Labels: report/customer, sig/planner, type/question + +## Customer-Facing Phenomenon +- The predicate is pushed down to table side. + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62050-cannot-simplify-predicate-isnull-not-null-column.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62050-cannot-simplify-predicate-isnull-not-null-column.md new file mode 100644 index 0000000..de19ae3 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62050-cannot-simplify-predicate-isnull-not-null-column.md @@ -0,0 +1,81 @@ +# Issue #62050: cannot simplify predicate ```isnull(not null column)``` + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62050 +- Status: closed +- Type: type/bug +- Created At: 2025-06-27T09:05:11Z +- Closed At: 2025-07-01T15:58:28Z +- Labels: affects-7.5, affects-8.1, affects-8.5, affects-9.0, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #62046: planner: constant folding to isnull(not null column) | tidb-test=pr/2544 + URL: https://github.com/pingcap/tidb/pull/62046 + State: closed + Merged At: 2025-07-01T15:58:27Z + Changed Files Count: 9 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/rule/BUILD.bazel + - pkg/planner/core/casetest/rule/main_test.go + - pkg/planner/core/casetest/rule/rule_predicate_simplification_test.go + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_in.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_out.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_xut.json + - pkg/planner/core/constraint/BUILD.bazel + - pkg/planner/core/constraint/exprs.go + - pkg/planner/core/operator/logicalop/logical_join.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? When we process the outer join, we remove the not-null flag. However, during predicate pushdown, this causes the flag to differ from the original one, and as a result, predicate simplification cannot be performed when handling the data source. +- Fix PR #62160: planner: constant folding to isnull(not null column) | tidb-test=pr/2560 + URL: https://github.com/pingcap/tidb/pull/62160 + State: closed + Merged At: 2025-07-16T01:17:18Z + Changed Files Count: 6 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/BUILD.bazel + - pkg/planner/core/casetest/rule/BUILD.bazel + - pkg/planner/core/casetest/rule/rule_predicate_simplification_test.go + - pkg/planner/core/constraint/BUILD.bazel + - pkg/planner/core/constraint/exprs.go + - pkg/planner/core/rule_predicate_push_down.go + PR Summary: This is an automated cherry-pick of #62046 What problem does this PR solve? Problem Summary: What changed and how does it work? When we process the outer join, we remove the not-null flag. However, during predicate pushdown, this causes the flag to differ from the original one, and as a result, predicate simplification cannot be performed when handling the data source. +- Fix PR #62161: planner: constant folding to isnull(not null column) | tidb-test=pr/2569 + URL: https://github.com/pingcap/tidb/pull/62161 + State: closed + Merged At: 2025-07-31T11:58:52Z + Changed Files Count: 9 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/BUILD.bazel + - pkg/planner/core/casetest/rule/BUILD.bazel + - pkg/planner/core/casetest/rule/main_test.go + - pkg/planner/core/casetest/rule/rule_predicate_simplification_test.go + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_in.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_out.json + - pkg/planner/core/constraint/BUILD.bazel + - pkg/planner/core/constraint/exprs.go + - pkg/planner/core/rule_predicate_push_down.go + PR Summary: This is an automated cherry-pick of #62046 What problem does this PR solve? Problem Summary: What changed and how does it work? When we process the outer join, we remove the not-null flag. However, during predicate pushdown, this causes the flag to differ from the original one, and as a result, predicate simplification cannot be performed when handling the data source. +- Fix PR #62163: planner: constant folding to isnull(not null column) | tidb-test=pr/2559 + URL: https://github.com/pingcap/tidb/pull/62163 + State: closed + Merged At: 2025-07-12T10:17:06Z + Changed Files Count: 8 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/rule/BUILD.bazel + - pkg/planner/core/casetest/rule/main_test.go + - pkg/planner/core/casetest/rule/rule_predicate_simplification_test.go + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_in.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_out.json + - pkg/planner/core/constraint/BUILD.bazel + - pkg/planner/core/constraint/exprs.go + - pkg/planner/core/operator/logicalop/logical_join.go + PR Summary: This is an automated cherry-pick of #62046 What problem does this PR solve? Problem Summary: What changed and how does it work? When we process the outer join, we remove the not-null flag. However, during predicate pushdown, this causes the flag to differ from the original one, and as a result, predicate simplification cannot be performed when handling the data source. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62216-stats-analyze-panic-with-error-index-out-of-range-10-with-length-10-include-virt.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62216-stats-analyze-panic-with-error-index-out-of-range-10-with-length-10-include-virt.md new file mode 100644 index 0000000..328d603 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62216-stats-analyze-panic-with-error-index-out-of-range-10-with-length-10-include-virt.md @@ -0,0 +1,27 @@ +# Issue #62216: stats: analyze panic with error "index out of range [10] with length 10" include virtual column + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62216 +- Status: closed +- Type: type/bug +- Created At: 2025-07-04T08:18:50Z +- Closed At: 2025-08-21T02:13:19Z +- Labels: affects-8.5, duplicate, may-affects-6.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. create table with virtual column as index first column 2. insert into some data 3. analyze table + +## Linked PRs +- Fix PR #62968: planner: filter generated columns that depend on skipped columns + URL: https://github.com/pingcap/tidb/pull/62968 + State: closed + Merged At: 2025-08-18T16:04:38Z + Changed Files Count: 2 + Main Modules: pkg/executor, pkg/planner/core + Sample Changed Files: + - pkg/executor/test/analyzetest/analyze_test.go + - pkg/planner/core/planbuilder.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? When filtering columns, also skip generated columns that have dependencies on any of the skipped columns to maintain consistency and avoid reference errors in query planning. Another issue is that the predicate should also consider the dependency columns. Otherwise, we will miss some column information when building the analyze executor. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62438-the-same-count-tablerowidscan-may-generate-different-cost-evaluation.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62438-the-same-count-tablerowidscan-may-generate-different-cost-evaluation.md new file mode 100644 index 0000000..872ed01 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62438-the-same-count-tablerowidscan-may-generate-different-cost-evaluation.md @@ -0,0 +1,121 @@ +# Issue #62438: the same count tableRowIDScan may generate different cost evaluation + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62438 +- Status: closed +- Type: type/bug +- Created At: 2025-07-16T03:55:36Z +- Closed At: 2025-08-01T09:49:55Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, component/statistics, epic/cardinality-estimation, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- And you could tell from those explain above, for same count in the tableScan, but the cost formula will compute into a different row size like: and which is confusing + +## Linked PRs +- Related PR #62537: planner: keep hist unchanged when deriving limit stats + URL: https://github.com/pingcap/tidb/pull/62537 + State: closed + Merged At: 2025-08-01T09:49:54Z + Changed Files Count: 19 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/planner + Sample Changed Files: + - pkg/planner/core/casetest/cbotest/BUILD.bazel + - pkg/planner/core/casetest/cbotest/cbo_test.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_in.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/cbotest/testdata/issue62438.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/vectorsearch/vector_index_test.go + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/integration_test.go + - pkg/planner/core/plan_cost_ver2_test.go + - pkg/planner/util/misc.go + - tests/integrationtest/r/planner/core/casetest/integration.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/plan_cache.result + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #62773: planner: keep hist unchanged when deriving limit stats (#62537) + URL: https://github.com/pingcap/tidb/pull/62773 + State: open + Merged At: not merged + Changed Files Count: 19 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/planner + Sample Changed Files: + - pkg/planner/core/casetest/cbotest/BUILD.bazel + - pkg/planner/core/casetest/cbotest/cbo_test.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_in.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/cbotest/testdata/issue62438.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/vectorsearch/vector_index_test.go + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/integration_test.go + - pkg/planner/core/plan_cost_ver2_test.go + - pkg/planner/util/misc.go + - tests/integrationtest/r/planner/core/casetest/integration.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/plan_cache.result + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + PR Summary: This is an automated cherry-pick of #62537 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #62774: planner: keep hist unchanged when deriving limit stats (#62537) + URL: https://github.com/pingcap/tidb/pull/62774 + State: open + Merged At: not merged + Changed Files Count: 19 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/planner + Sample Changed Files: + - pkg/planner/core/casetest/cbotest/BUILD.bazel + - pkg/planner/core/casetest/cbotest/cbo_test.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_in.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/cbotest/testdata/issue62438.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/vectorsearch/vector_index_test.go + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/integration_test.go + - pkg/planner/core/plan_cost_ver2_test.go + - pkg/planner/util/misc.go + - tests/integrationtest/r/planner/core/casetest/integration.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/plan_cache.result + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + PR Summary: This is an automated cherry-pick of #62537 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #63562: planner: keep hist unchanged when deriving limit stats (#62537) + URL: https://github.com/pingcap/tidb/pull/63562 + State: closed + Merged At: 2025-09-20T23:11:01Z + Changed Files Count: 15 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/planner + Sample Changed Files: + - pkg/planner/core/casetest/cbotest/BUILD.bazel + - pkg/planner/core/casetest/cbotest/cbo_test.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_in.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/issue62438.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_out.json + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/vectorsearch/vector_index_test.go + - pkg/planner/core/plan_cost_ver2_test.go + - pkg/planner/util/misc.go + - tests/integrationtest/r/planner/core/casetest/integration.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/plan_cache.result + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + PR Summary: This is an automated cherry-pick of #62537 What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62448-tidb-can-t-push-down-join-with-nulleq-key-condition-to-tiflash.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62448-tidb-can-t-push-down-join-with-nulleq-key-condition-to-tiflash.md new file mode 100644 index 0000000..2ac3542 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62448-tidb-can-t-push-down-join-with-nulleq-key-condition-to-tiflash.md @@ -0,0 +1,17 @@ +# Issue #62448: tidb can‘t push down join with nullEQ key condition to tiflash + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62448 +- Status: open +- Type: type/enhancement +- Created At: 2025-07-16T07:43:41Z +- Labels: report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- those join is generated from following sql + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62499-planner-consider-the-number-of-ranges-and-the-seek-operation-cost-for-indexjoin-.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62499-planner-consider-the-number-of-ranges-and-the-seek-operation-cost-for-indexjoin-.md new file mode 100644 index 0000000..88a7768 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62499-planner-consider-the-number-of-ranges-and-the-seek-operation-cost-for-indexjoin-.md @@ -0,0 +1,71 @@ +# Issue #62499: planner: consider the number of ranges and the seek operation cost for IndexJoin in cost model + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62499 +- Status: closed +- Type: type/enhancement +- Created At: 2025-07-18T09:16:33Z +- Closed At: 2025-09-23T03:45:09Z +- Labels: affects-8.5, epic/cost-model, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- IndexJoin's cost evaluation and even the index choosing should care about the number of ranges which could affect the seek operation in storage. + +## Linked PRs +- Fix PR #63488: planner: consider seek operation cost in cost model for Index and Table Range Scans + URL: https://github.com/pingcap/tidb/pull/63488 + State: closed + Merged At: not merged + Changed Files Count: 2 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/plan_cost_ver2_test.go + PR Summary: What problem does this PR solve? Problem Summary: planner: consider seek operation cost in cost model for Index and Table Range Scans What changed and how does it work? Please see #63487 for more information. +- Fix PR #63568: planner: consider magnified seeking cost in IndexJoin + URL: https://github.com/pingcap/tidb/pull/63568 + State: closed + Merged At: 2025-09-23T03:45:08Z + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + PR Summary: What problem does this PR solve? Problem Summary: planner: consider magnified seeking cost in IndexJoin What changed and how does it work? Our current cost model doesn't consider the cost of seeking operation, which is usually cased by IN predicate like . In most cases its cost could be negligible, but if its under , its cost could be magnified by the significantly (depending on the number of build rows), please see an simplified and concrete case here: +- Fix PR #64775: planner: consider magnified seeking cost in IndexJoin (#63568) + URL: https://github.com/pingcap/tidb/pull/64775 + State: closed + Merged At: 2025-12-15T02:59:33Z + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + PR Summary: This is an automated cherry-pick of #63568 What problem does this PR solve? Problem Summary: planner: consider magnified seeking cost in IndexJoin What changed and how does it work? Our current cost model doesn't consider the cost of seeking operation, which is usually cased by IN predicate like . +- Related PR #65465: planner: consider seek cost in range construction + URL: https://github.com/pingcap/tidb/pull/65465 + State: open + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/cost/factors_thresholds.go + - pkg/planner/core/integration_test.go + - pkg/planner/core/stats.go + - tests/integrationtest/r/planner/core/grouped_ranges_order_by.result + PR Summary: What problem does this PR solve? Problem Summary: The current optimizer cost model only takes seek cost into account for index joins. See #62499 for the fix for index join. However, the optimizer still selects the incorrect index on index range scan or table range scan operations. What changed and how does it work? It is not enough to simply account for the cost of seeks in the cost planning phase. It is sometimes better to only use a subset of the columns from the index and perform the selection afterwards (e.g. given an index abc, scan all entries under a = 1 and then perform a selection on b in (...) rather than doing a scan on every combination of (a = 1, b in (...)). To address this, when constructing the ranges for an index, we evaluate every possible set of ranges after the equality +- Related PR #66386: planner: consider seek cost in skyline pruning + URL: https://github.com/pingcap/tidb/pull/66386 + State: open + Merged At: not merged + Changed Files Count: 6 + Main Modules: pkg/planner/core, pkg/planner, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/cost/factors_thresholds.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/integration_test.go + - pkg/planner/core/stats.go + - pkg/planner/util/fixcontrol/get.go + - tests/integrationtest/r/planner/core/grouped_ranges_order_by.result + PR Summary: What problem does this PR solve? Problem Summary: The current optimizer cost model only takes seek cost into account for index joins. See for the fix for index join. However, the optimizer still selects the incorrect index on index range scan or table range scan operations. added support for IN-List match pruning to partially address this issue by allowing construction of ranges based on a prefix of an index. What changed and how does it work? This PR modifies the skyline pruning calculation to take into account seek cost when pruning indexes. This effectively allows comparison of seek costs between 2 paths to heuristically choose the better one prior to cost calculation. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62551-rollup-will-output-more-rows-than-expected.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62551-rollup-will-output-more-rows-than-expected.md new file mode 100644 index 0000000..06b25b2 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62551-rollup-will-output-more-rows-than-expected.md @@ -0,0 +1,104 @@ +# Issue #62551: rollup will output more rows than expected + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62551 +- Status: closed +- Type: type/bug +- Created At: 2025-07-22T08:18:03Z +- Closed At: 2025-07-23T14:42:15Z +- Labels: affects-8.1, affects-8.5, impact/wrong-result, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Related PR #62558: planner: fix expand operator shouldn't keep child keys && fix grouping function forget to encode their func meta + URL: https://github.com/pingcap/tidb/pull/62558 + State: closed + Merged At: 2025-07-23T14:42:14Z + Changed Files Count: 10 + Main Modules: pkg/planner/core, pkg/expression + Sample Changed Files: + - pkg/expression/builtin_grouping.go + - pkg/expression/scalar_function.go + - pkg/planner/core/explain.go + - pkg/planner/core/operator/logicalop/logical_expand.go + - pkg/planner/core/operator/logicalop/logicalop_test/BUILD.bazel + - pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go + - pkg/planner/core/operator/logicalop/logicalop_test/main_test.go + - pkg/planner/core/operator/logicalop/logicalop_test/testdata/cascades_suite_in.json + - pkg/planner/core/operator/logicalop/logicalop_test/testdata/cascades_suite_out.json + - pkg/planner/core/operator/logicalop/logicalop_test/testdata/cascades_suite_xut.json + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? * previously, tidb use buildKeyInfo to derive unique key and pk for each logical operator. While for logical expand, if we use baseLogicalPlan's buildKeyInfo logic, it will derive and keep child unique key info. Since expand op will duplicate rows, we just keep it sample as setting it nil. * the second problem is that: expression hashcode will simply encode func name and return type for normal scalar function, while for grouping function, its function meta should also be taken into consideration. Otherwise, two different grouping function will be seen as a same one in predicateSimplication logic. +- Related PR #62588: planner: fix expand operator shouldn't keep child keys && fix grouping function forget to encode their func meta (#62558) + URL: https://github.com/pingcap/tidb/pull/62588 + State: open + Merged At: not merged + Changed Files Count: 18 + Main Modules: pkg/planner/core, pkg/expression + Sample Changed Files: + - pkg/expression/builtin_grouping.go + - pkg/expression/scalar_function.go + - pkg/planner/core/explain.go + - pkg/planner/core/operator/logicalop/logical_expand.go + - pkg/planner/core/operator/logicalop/logicalop_test/BUILD.bazel + - pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go + - pkg/planner/core/operator/logicalop/logicalop_test/main_test.go + - pkg/planner/core/testdata/cascades_suite_in.json + - pkg/planner/core/testdata/cascades_suite_out.json + - pkg/planner/core/testdata/cascades_suite_xut.json + - pkg/planner/core/testdata/index_merge_suite_in.json + - pkg/planner/core/testdata/index_merge_suite_out.json + - pkg/planner/core/testdata/plan_cache_suite_in.json + - pkg/planner/core/testdata/plan_cache_suite_out.json + - pkg/planner/core/testdata/plan_suite_unexported_in.json + - pkg/planner/core/testdata/plan_suite_unexported_out.json + - pkg/planner/core/testdata/runtime_filter_generator_suite_in.json + - pkg/planner/core/testdata/runtime_filter_generator_suite_out.json + PR Summary: This is an automated cherry-pick of #62558 What problem does this PR solve? Problem Summary: What changed and how does it work? * previously, tidb use buildKeyInfo to derive unique key and pk for each logical operator. While for logical expand, if we use baseLogicalPlan's buildKeyInfo logic, it will derive and keep child unique key info. Since expand op will duplicate rows, we just keep it sample as setting it nil. +- Related PR #62589: planner: fix expand operator shouldn't keep child keys && fix grouping function forget to encode their func meta (#62558) + URL: https://github.com/pingcap/tidb/pull/62589 + State: closed + Merged At: 2025-07-29T07:11:16Z + Changed Files Count: 11 + Main Modules: pkg/planner/core, pkg/expression + Sample Changed Files: + - pkg/expression/builtin_grouping.go + - pkg/expression/expression.go + - pkg/expression/scalar_function.go + - pkg/expression/schema.go + - pkg/planner/core/explain.go + - pkg/planner/core/operator/logicalop/logical_expand.go + - pkg/planner/core/operator/logicalop/logicalop_test/BUILD.bazel + - pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go + - pkg/planner/core/operator/logicalop/logicalop_test/main_test.go + - pkg/planner/core/operator/logicalop/logicalop_test/testdata/cascades_suite_in.json + - pkg/planner/core/operator/logicalop/logicalop_test/testdata/cascades_suite_out.json + PR Summary: This is an automated cherry-pick of #62558 What problem does this PR solve? Problem Summary: What changed and how does it work? * previously, tidb use buildKeyInfo to derive unique key and pk for each logical operator. While for logical expand, if we use baseLogicalPlan's buildKeyInfo logic, it will derive and keep child unique key info. Since expand op will duplicate rows, we just keep it sample as setting it nil. +- Related PR #62595: planner: fix expand operator shouldn't keep child keys && fix grouping function forget to encode their func meta (#62558) + URL: https://github.com/pingcap/tidb/pull/62595 + State: open + Merged At: not merged + Changed Files Count: 16 + Main Modules: pkg/planner/core, pkg/expression + Sample Changed Files: + - pkg/expression/builtin_grouping.go + - pkg/expression/scalar_function.go + - pkg/planner/core/explain.go + - pkg/planner/core/operator/logicalop/logical_expand.go + - pkg/planner/core/operator/logicalop/logicalop_test/BUILD.bazel + - pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go + - pkg/planner/core/operator/logicalop/logicalop_test/main_test.go + - pkg/planner/core/testdata/cascades_suite_in.json + - pkg/planner/core/testdata/cascades_suite_out.json + - pkg/planner/core/testdata/cascades_suite_xut.json + - pkg/planner/core/testdata/index_merge_suite_in.json + - pkg/planner/core/testdata/index_merge_suite_out.json + - pkg/planner/core/testdata/plan_suite_unexported_in.json + - pkg/planner/core/testdata/plan_suite_unexported_out.json + - pkg/planner/core/testdata/runtime_filter_generator_suite_in.json + - pkg/planner/core/testdata/runtime_filter_generator_suite_out.json + PR Summary: This is an automated cherry-pick of #62558 What problem does this PR solve? Problem Summary: What changed and how does it work? * previously, tidb use buildKeyInfo to derive unique key and pk for each logical operator. While for logical expand, if we use baseLogicalPlan's buildKeyInfo logic, it will derive and keep child unique key info. Since expand op will duplicate rows, we just keep it sample as setting it nil. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62556-group-by-in-prepared-statement-causes-error-in-tidb-mysql-compatible.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62556-group-by-in-prepared-statement-causes-error-in-tidb-mysql-compatible.md new file mode 100644 index 0000000..800a7ca --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62556-group-by-in-prepared-statement-causes-error-in-tidb-mysql-compatible.md @@ -0,0 +1,38 @@ +# Issue #62556: GROUP BY ? in prepared statement causes error in TiDB (MySQL compatible) + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62556 +- Status: closed +- Type: type/bug +- Created At: 2025-07-22T09:28:57Z +- Closed At: 2026-03-09T15:55:30Z +- Labels: epic/plan-cache, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- In TiDB 8.5.2, when executing , the following error occurs: And when preparing the statement, a warning is shown: + +## Linked PRs +- Fix PR #66387: planner, executor: fix prepared string parameter compatibility in group/order by + URL: https://github.com/pingcap/tidb/pull/66387 + State: closed + Merged At: 2026-02-26T13:07:45Z + Changed Files Count: 2 + Main Modules: pkg/executor, pkg/planner/core + Sample Changed Files: + - pkg/executor/test/seqtest/prepared_test.go + - pkg/planner/core/logical_plan_builder.go + PR Summary: What problem does this PR solve? Problem Summary: Prepared statements with top-level (and aggregate ) could treat string parameters like as positional references at execute time, causing errors such as . This differs from MySQL behavior. What changed and how does it work? Added in planner resolver path. +- Fix PR #63338: planner: fix GROUP BY ? in prepared statement causes error + URL: https://github.com/pingcap/tidb/pull/63338 + State: closed + Merged At: not merged + Changed Files Count: 3 + Main Modules: pkg/planner/core, pkg/expression + Sample Changed Files: + - pkg/expression/util.go + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/plan_cache_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? In Spring ORM, sometimes such prepare statements are generated. Grouping by 0 is illegal, but here it should be of string type. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62665-equal-predicate-at-end-of-range-or-out-of-range-does-not-recognize-modifycount.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62665-equal-predicate-at-end-of-range-or-out-of-range-does-not-recognize-modifycount.md new file mode 100644 index 0000000..7d1ce85 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62665-equal-predicate-at-end-of-range-or-out-of-range-does-not-recognize-modifycount.md @@ -0,0 +1,45 @@ +# Issue #62665: Equal predicate at end of range or out of range does not recognize modifyCount + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62665 +- Status: closed +- Type: type/bug +- Created At: 2025-07-29T00:05:42Z +- Closed At: 2025-08-02T19:33:52Z +- Labels: affects-8.5, epic/cardinality-estimation, report/customer, severity/minor, sig/planner, type/bug + +## Customer-Facing Phenomenon +- The explain output is similar to before the rows were inserted + +## Linked PRs +- Fix PR #62695: planner: handle histogram last bucket end value underrepresented + URL: https://github.com/pingcap/tidb/pull/62695 + State: closed + Merged At: 2025-08-02T19:33:51Z + Changed Files Count: 6 + Main Modules: pkg/planner, pkg/statistics + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_column.go + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/statistics/histogram.go + PR Summary: What problem does this PR solve? Problem Summary: Like what the linked issue describes, the end value in the last histogram bucket could be underrepresented due to stale stats. This PR tries to detect such case precisely by looking at two conditions: 1. The value should be in last histogram bucket and it should be the upper bound 2. The last bucket end count is much less than the average count of other NDV, indicating this might be a tail write traffic +- Fix PR #62812: planner: handle histogram last bucket end value underrepresented (#62695) + URL: https://github.com/pingcap/tidb/pull/62812 + State: closed + Merged At: 2025-09-18T03:25:44Z + Changed Files Count: 6 + Main Modules: pkg/planner, pkg/statistics + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_column.go + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/statistics/histogram.go + PR Summary: This is an automated cherry-pick of #62695 What problem does this PR solve? Problem Summary: Like what the linked issue describes, the end value in the last histogram bucket could be underrepresented due to stale stats. This PR tries to detect such case precisely by looking at two conditions: 1. The value should be in last histogram bucket and it should be the upper bound + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62672-only-full-group-by-check-should-be-case-sensitive.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62672-only-full-group-by-check-should-be-case-sensitive.md new file mode 100644 index 0000000..8362415 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62672-only-full-group-by-check-should-be-case-sensitive.md @@ -0,0 +1,83 @@ +# Issue #62672: only full group by check should be case-sensitive + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62672 +- Status: closed +- Type: type/bug +- Created At: 2025-07-29T04:22:28Z +- Closed At: 2025-08-04T15:52:10Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- only full group by check error + +## Linked PRs +- Related PR #62751: planner: fix the old only full group check should be case-insensitive. + URL: https://github.com/pingcap/tidb/pull/62751 + State: closed + Merged At: 2025-08-04T15:52:09Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/parser + Sample Changed Files: + - pkg/parser/ast/expressions.go + - tests/integrationtest/r/executor/aggregate.result + - tests/integrationtest/t/executor/aggregate.test + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #62814: planner: fix the old only full group check should be case-insensitive. (#62751) + URL: https://github.com/pingcap/tidb/pull/62814 + State: open + Merged At: not merged + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/parser + Sample Changed Files: + - pkg/parser/ast/expressions.go + - tests/integrationtest/r/executor/aggregate.result + - tests/integrationtest/t/executor/aggregate.test + PR Summary: This is an automated cherry-pick of #62751 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #62815: planner: fix the old only full group check should be case-insensitive. (#62751) + URL: https://github.com/pingcap/tidb/pull/62815 + State: closed + Merged At: not merged + Changed Files Count: 3 + Main Modules: tests/integrationtest, parser/ast + Sample Changed Files: + - parser/ast/expressions.go + - tests/integrationtest/r/executor/aggregate.result + - tests/integrationtest/t/executor/aggregate.test + PR Summary: This is an automated cherry-pick of #62751 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #62816: planner: fix the old only full group check should be case-insensitive. (#62751) + URL: https://github.com/pingcap/tidb/pull/62816 + State: open + Merged At: not merged + Changed Files Count: 3 + Main Modules: tests/integrationtest, parser/ast + Sample Changed Files: + - parser/ast/expressions.go + - tests/integrationtest/r/executor/aggregate.result + - tests/integrationtest/t/executor/aggregate.test + PR Summary: This is an automated cherry-pick of #62751 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #63557: planner: fix the old only full group check should be case-insensitive. (#62751) + URL: https://github.com/pingcap/tidb/pull/63557 + State: closed + Merged At: 2025-09-21T18:41:07Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/parser + Sample Changed Files: + - pkg/parser/ast/expressions.go + - tests/integrationtest/r/executor/aggregate.result + - tests/integrationtest/t/executor/aggregate.test + PR Summary: This is an automated cherry-pick of #62751 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Related PR #63558: planner: fix the old only full group check should be case-insensitive. (#62751) + URL: https://github.com/pingcap/tidb/pull/63558 + State: open + Merged At: not merged + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/parser + Sample Changed Files: + - pkg/parser/ast/expressions.go + - tests/integrationtest/r/executor/aggregate.result + - tests/integrationtest/t/executor/aggregate.test + PR Summary: This is an automated cherry-pick of #62751 What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62722-enable-set-var-hint-support-for-tidb-allow-tiflash-cop.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62722-enable-set-var-hint-support-for-tidb-allow-tiflash-cop.md new file mode 100644 index 0000000..57be2cd --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62722-enable-set-var-hint-support-for-tidb-allow-tiflash-cop.md @@ -0,0 +1,28 @@ +# Issue #62722: Enable `set_var` Hint Support for `tidb_allow_tiflash_cop` + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62722 +- Status: closed +- Type: type/enhancement +- Created At: 2025-07-30T13:32:01Z +- Closed At: 2025-09-17T15:27:07Z +- Labels: epic/hint, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- Currently, has a default value of . This is because in most cases, we only want TiFlash to use the MPP protocol for execution, as MPP is almost always superior to the cop protocol. However, there is an exception: using the cop protocol allows for execution in "keep order" mode, meaning that queries like can be executed with the cop protocol and terminate early, similar to when using TiKV. + +## Linked PRs +- Fix PR #63570: sessionctx: disable warning when setting tidb_allow_tiflash_cop in set_var hint + URL: https://github.com/pingcap/tidb/pull/63570 + State: closed + Merged At: 2025-09-17T15:27:06Z + Changed Files Count: 3 + Main Modules: pkg/executor, pkg/session + Sample Changed Files: + - pkg/executor/test/tiflashtest/BUILD.bazel + - pkg/executor/test/tiflashtest/tiflash_test.go + - pkg/sessionctx/variable/setvar_affect.go + PR Summary: What problem does this PR solve? Problem Summary: check What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62766-same-predicate-estimation-of-two-approximate-stats-is-quite-different.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62766-same-predicate-estimation-of-two-approximate-stats-is-quite-different.md new file mode 100644 index 0000000..a57995c --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62766-same-predicate-estimation-of-two-approximate-stats-is-quite-different.md @@ -0,0 +1,30 @@ +# Issue #62766: same predicate estimation of two approximate stats is quite different + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62766 +- Status: open +- Type: type/enhancement +- Created At: 2025-08-01T07:22:37Z +- Labels: affects-7.5, component/statistics, epic/cardinality-estimation, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- we can tell from the pic above, it's 3M vs 0.3M actually, ref TCOC-3945 to get the replayer detail + +## Linked PRs +- Related PR #62695: planner: handle histogram last bucket end value underrepresented + URL: https://github.com/pingcap/tidb/pull/62695 + State: closed + Merged At: 2025-08-02T19:33:51Z + Changed Files Count: 6 + Main Modules: pkg/planner, pkg/statistics + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_column.go + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/statistics/histogram.go + PR Summary: What problem does this PR solve? Problem Summary: Like what the linked issue describes, the end value in the last histogram bucket could be underrepresented due to stale stats. This PR tries to detect such case precisely by looking at two conditions: 1. The value should be in last histogram bucket and it should be the upper bound 2. The last bucket end count is much less than the average count of other NDV, indicating this might be a tail write traffic + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62917-planner-mergejoin-cost-formula-should-consider-othercond.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62917-planner-mergejoin-cost-formula-should-consider-othercond.md new file mode 100644 index 0000000..332efdd --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62917-planner-mergejoin-cost-formula-should-consider-othercond.md @@ -0,0 +1,47 @@ +# Issue #62917: planner: MergeJoin cost formula should consider OtherCond + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62917 +- Status: closed +- Type: type/bug +- Created At: 2025-08-11T07:36:12Z +- Closed At: 2025-08-15T18:35:31Z +- Labels: affects-7.5, affects-8.5, epic/cost-model, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- They have the same cost. + +## Linked PRs +- Fix PR #63010: planner: consider OtherConds in MergeJoin cost formula + URL: https://github.com/pingcap/tidb/pull/63010 + State: closed + Merged At: 2025-08-15T18:35:30Z + Changed Files Count: 2 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/plan_cost_ver2_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Please see the case below, the first plan's cost should be larger than the second one, but now they are the same. The reason is that we forget to consider the cost of in our MergeJoin cost formula, this PR is going to fix this problem. +- Fix PR #63112: planner: consider OtherConds in MergeJoin cost formula (#63010) + URL: https://github.com/pingcap/tidb/pull/63112 + State: closed + Merged At: 2025-09-22T10:14:37Z + Changed Files Count: 2 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/plan_cost_ver2_test.go + PR Summary: This is an automated cherry-pick of #63010 What problem does this PR solve? Problem Summary: What changed and how does it work? Please see the case below, the first plan's cost should be larger than the second one, but now they are the same. +- Fix PR #63113: planner: consider OtherConds in MergeJoin cost formula (#63010) + URL: https://github.com/pingcap/tidb/pull/63113 + State: open + Merged At: not merged + Changed Files Count: 2 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/plan_cost_ver2_test.go + PR Summary: This is an automated cherry-pick of #63010 What problem does this PR solve? Problem Summary: What changed and how does it work? Please see the case below, the first plan's cost should be larger than the second one, but now they are the same. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62975-make-it-possible-to-push-agg-down-cross-unionscan.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62975-make-it-possible-to-push-agg-down-cross-unionscan.md new file mode 100644 index 0000000..0f750a4 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62975-make-it-possible-to-push-agg-down-cross-unionscan.md @@ -0,0 +1,17 @@ +# Issue #62975: Make it possible to push Agg down cross UnionScan + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62975 +- Status: open +- Type: type/enhancement +- Created At: 2025-08-13T12:03:37Z +- Labels: affects-8.5, report/customer, sig/planner, sig/transaction, type/enhancement + +## Customer-Facing Phenomenon +- As you can see, when there're dirty writes, the can not be pushed down to the coprocessor side. So the computation is left to the TiDB side. Compared with the normal execution without dirty writes, it's much slower. + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62978-unexpected-warning-when-using-read-from-storage-hint.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62978-unexpected-warning-when-using-read-from-storage-hint.md new file mode 100644 index 0000000..0dee08e --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-62978-unexpected-warning-when-using-read-from-storage-hint.md @@ -0,0 +1,18 @@ +# Issue #62978: Unexpected warning when using READ_FROM_STORAGE hint + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/62978 +- Status: closed +- Type: type/question +- Created At: 2025-08-13T15:26:28Z +- Closed At: 2025-08-14T13:40:32Z +- Labels: report/customer, sig/planner, type/question + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- The issue is closed, but no merged PR was resolved automatically from the timeline. It may have been fixed by an internal branch, a batch PR, or manual closure. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63075-select-1-from-dual-can-t-hit-instance-level-plan-cache.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63075-select-1-from-dual-can-t-hit-instance-level-plan-cache.md new file mode 100644 index 0000000..6f9a95d --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63075-select-1-from-dual-can-t-hit-instance-level-plan-cache.md @@ -0,0 +1,73 @@ +# Issue #63075: `select 1 from dual` can't hit instance level plan cache + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63075 +- Status: closed +- Type: type/bug +- Created At: 2025-08-19T17:12:42Z +- Closed At: 2025-08-28T01:51:37Z +- Labels: affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #63198: planner: implement func CloneForPlanCache() for PhysicalTableDual + URL: https://github.com/pingcap/tidb/pull/63198 + State: closed + Merged At: 2025-08-27T15:50:55Z + Changed Files Count: 3 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/instanceplancache/BUILD.bazel + - pkg/planner/core/casetest/instanceplancache/others_test.go + - pkg/planner/core/operator/physicalop/physical_table_dual.go + PR Summary: What problem does this PR solve? Problem Summary: “select 1 from dual” can't hit instance level plan cache because CloneForPlanCache() is not implemented for PhysicalTableDual. What changed and how does it work? I implemented func CloneForPlanCache() for PhysicalTableDual. +- Fix PR #63472: planner: use generator to implement the PhysicalTableDual's CloneForPlanCache + URL: https://github.com/pingcap/tidb/pull/63472 + State: closed + Merged At: 2025-09-11T13:12:13Z + Changed Files Count: 3 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/generator/plan_cache/plan_clone_generator.go + - pkg/planner/core/operator/physicalop/physical_table_dual.go + - pkg/planner/core/operator/physicalop/plan_clone_generated.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? use generator to implement the PhysicalTableDual's CloneForPlanCache +- Fix PR #63518: planner: implement func CloneForPlanCache() for PhysicalTableDual (#63198) + URL: https://github.com/pingcap/tidb/pull/63518 + State: closed + Merged At: 2025-09-17T19:45:59Z + Changed Files Count: 4 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/instanceplancache/BUILD.bazel + - pkg/planner/core/casetest/instanceplancache/others_test.go + - pkg/planner/core/generator/plan_cache/plan_clone_generator.go + - pkg/planner/core/plan_clone_generated.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #63547: planner: implement func CloneForPlanCache() for PhysicalTableDual (#63198) + URL: https://github.com/pingcap/tidb/pull/63547 + State: closed + Merged At: not merged + Changed Files Count: 3 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/instanceplancache/BUILD.bazel + - pkg/planner/core/casetest/instanceplancache/others_test.go + - pkg/planner/core/operator/physicalop/physical_table_dual.go + PR Summary: This is an automated cherry-pick of #63198 What problem does this PR solve? Problem Summary: “select 1 from dual” can't hit instance level plan cache because CloneForPlanCache() is not implemented for PhysicalTableDual. What changed and how does it work? +- Fix PR #63548: planner: use generator to implement the PhysicalTableDual's CloneForPlanCache (#63472) + URL: https://github.com/pingcap/tidb/pull/63548 + State: closed + Merged At: not merged + Changed Files Count: 3 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/generator/plan_cache/plan_clone_generator.go + - pkg/planner/core/operator/physicalop/physical_table_dual.go + - pkg/planner/core/plan_clone_generated.go + PR Summary: This is an automated cherry-pick of #63472 What problem does this PR solve? Problem Summary: What changed and how does it work? use generator to implement the PhysicalTableDual's CloneForPlanCache + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63235-with-the-introduction-of-issaferange-cpu-resources-are-consumed-more-heavily.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63235-with-the-introduction-of-issaferange-cpu-resources-are-consumed-more-heavily.md new file mode 100644 index 0000000..5ec4ff9 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63235-with-the-introduction-of-issaferange-cpu-resources-are-consumed-more-heavily.md @@ -0,0 +1,142 @@ +# Issue #63235: With the introduction of `isSafeRange`, CPU resources are consumed more heavily. + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63235 +- Status: closed +- Type: type/bug +- Created At: 2025-08-27T17:44:59Z +- Closed At: 2025-09-22T17:56:59Z +- Labels: affects-7.5, affects-8.1, affects-8.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #63566: planner: optimizer the performance of `IsFullRange` check + URL: https://github.com/pingcap/tidb/pull/63566 + State: closed + Merged At: 2025-09-22T17:56:58Z + Changed Files Count: 1 + Main Modules: pkg/util + Sample Changed Files: + - pkg/util/ranger/types.go + PR Summary: What problem does this PR solve? Problem Summary: planner: optimizer the performance of check What changed and how does it work? planner: optimizer the performance of check +- Fix PR #63812: planner: optimizer the performance of `IsFullRange` check (#63566) + URL: https://github.com/pingcap/tidb/pull/63812 + State: closed + Merged At: 2025-12-15T02:59:26Z + Changed Files Count: 1 + Main Modules: pkg/util + Sample Changed Files: + - pkg/util/ranger/types.go + PR Summary: This is an automated cherry-pick of #63566 What problem does this PR solve? Problem Summary: planner: optimizer the performance of check What changed and how does it work? planner: optimizer the performance of check +- Fix PR #63813: planner: optimizer the performance of `IsFullRange` check (#63566) + URL: https://github.com/pingcap/tidb/pull/63813 + State: closed + Merged At: 2025-10-31T15:06:36Z + Changed Files Count: 1 + Main Modules: pkg/util + Sample Changed Files: + - pkg/util/ranger/types.go + PR Summary: This is an automated cherry-pick of #63566 What problem does this PR solve? Problem Summary: planner: optimizer the performance of check What changed and how does it work? planner: optimizer the performance of check +- Fix PR #63828: planner: remove fullRange check from skyline pruning + URL: https://github.com/pingcap/tidb/pull/63828 + State: closed + Merged At: 2025-10-02T14:51:31Z + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/find_best_task.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? FullRange check was found to be expensive in some scenarios. This PR restructures the skylinePruning logic to reduce reliance on this check. It uses other checks where possible, and also will only call this after other criteria has been validated. +- Fix PR #64056: planner: optimizer the performance of `IsFullRange` check (#63566) + URL: https://github.com/pingcap/tidb/pull/64056 + State: open + Merged At: not merged + Changed Files Count: 1 + Main Modules: pkg/util + Sample Changed Files: + - pkg/util/ranger/types.go + PR Summary: This is an automated cherry-pick of #63566 What problem does this PR solve? Problem Summary: planner: optimizer the performance of check What changed and how does it work? planner: optimizer the performance of check +- Fix PR #64057: planner: remove fullRange check from skyline pruning (#63828) + URL: https://github.com/pingcap/tidb/pull/64057 + State: open + Merged At: not merged + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/find_best_task.go + PR Summary: This is an automated cherry-pick of #63828 What problem does this PR solve? Problem Summary: What changed and how does it work? FullRange check was found to be expensive in some scenarios. This PR will only call this after other criteria has been validated. +- Fix PR #64058: planner: remove fullRange check from skyline pruning (#63828) + URL: https://github.com/pingcap/tidb/pull/64058 + State: closed + Merged At: 2025-11-01T01:09:48Z + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/find_best_task.go + PR Summary: This is an automated cherry-pick of #63828 What problem does this PR solve? Problem Summary: What changed and how does it work? FullRange check was found to be expensive in some scenarios. This PR restructures the skylinePruning logic to reduce reliance on this check. It uses other checks where possible, and also will only call this after other criteria has been validated. +- Fix PR #64447: planner: remove fullRange check from skyline pruning (#63828) + URL: https://github.com/pingcap/tidb/pull/64447 + State: closed + Merged At: 2025-12-11T20:22:50Z + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/find_best_task.go + PR Summary: This is an automated cherry-pick of #63828 What problem does this PR solve? Problem Summary: What changed and how does it work? FullRange check was found to be expensive in some scenarios. This PR restructures the skylinePruning logic to reduce reliance on this check. It uses other checks where possible, and also will only call this after other criteria has been validated. +- Related PR #66304: planner: optimize for full range + URL: https://github.com/pingcap/tidb/pull/66304 + State: closed + Merged At: 2026-02-19T19:09:06Z + Changed Files Count: 5 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/stats.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? This PR provides 2 main optimizations: 1) It allows index row count estimation to be skipped if the predicate ranges cover the entire index range. +- Related PR #66313: planner: optimize for full range (#66304) + URL: https://github.com/pingcap/tidb/pull/66313 + State: closed + Merged At: not merged + Changed Files Count: 5 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/stats.go + PR Summary: This is an automated cherry-pick of #66304 What problem does this PR solve? Problem Summary: What changed and how does it work? This PR provides 2 main optimizations: +- Related PR #66314: planner: optimize for full range (#66304) + URL: https://github.com/pingcap/tidb/pull/66314 + State: closed + Merged At: not merged + Changed Files Count: 5 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/stats.go + PR Summary: This is an automated cherry-pick of #66304 What problem does this PR solve? Problem Summary: What changed and how does it work? This PR provides 2 main optimizations: +- Related PR #66695: planner: optimize for full range (#66304) + URL: https://github.com/pingcap/tidb/pull/66695 + State: open + Merged At: not merged + Changed Files Count: 6 + Main Modules: pkg/planner, pkg/planner/core, .cursor/commands + Sample Changed Files: + - .cursor/commands/new-question.md + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/stats.go + - pkg/planner/util/path.go + PR Summary: This is an automated cherry-pick of #66304 What problem does this PR solve? Problem Summary: What changed and how does it work? This PR provides 2 main optimizations: + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63280-planner-utilize-prefix-of-columns-in-order-limit-clause-to-avoid-full-table-scan.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63280-planner-utilize-prefix-of-columns-in-order-limit-clause-to-avoid-full-table-scan.md new file mode 100644 index 0000000..683da8c --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63280-planner-utilize-prefix-of-columns-in-order-limit-clause-to-avoid-full-table-scan.md @@ -0,0 +1,52 @@ +# Issue #63280: planner: utilize prefix of columns in order-limit clause to avoid full table scan in join queries + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63280 +- Status: closed +- Type: type/enhancement +- Created At: 2025-08-29T08:26:17Z +- Closed At: 2026-02-28T10:32:40Z +- Labels: affects-8.5, plan-rewrite, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- But actually, we could first use some prefix columns of order-limit clause to avoid full table scan, and then order the final results by 2 columns at the end, which is like: + +## Linked PRs +- Fix PR #65314: session: Add a new session variable to control the prefix index optimization for ORDER BY ... LIMIT queries + URL: https://github.com/pingcap/tidb/pull/65314 + State: closed + Merged At: 2025-12-31T12:48:45Z + Changed Files Count: 14 + Main Modules: pkg/session, build/nogo_config.json, pkg/planner/core + Sample Changed Files: + - build/nogo_config.json + - pkg/planner/core/hint_test.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/BUILD.bazel + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + - pkg/sessionctx/variable/sysvar_test.go + - pkg/sessionctx/variable/tests/BUILD.bazel + - pkg/sessionctx/variable/tests/main_test.go + - pkg/sessionctx/variable/tests/session_test.go + - pkg/sessionctx/variable/tests/slowlog/BUILD.bazel + - pkg/sessionctx/variable/tests/slowlog/main_test.go + - pkg/sessionctx/variable/tests/slowlog/slow_log_test.go + - pkg/sessionctx/variable/tests/variable_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? 1. Added a new session variable tidb_opt_partial_ordered_index_for_topn to control prefix index optimization for ORDER BY ... LIMIT queries. 2. This is a global | session variable +- Related PR #65533: planner: add the prefix index as candidate for topn optimization + URL: https://github.com/pingcap/tidb/pull/65533 + State: closed + Merged At: 2026-01-17T14:41:29Z + Changed Files Count: 5 + Main Modules: pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/core/find_best_task.go + - pkg/planner/core/operator/logicalop/logical_projection.go + - pkg/planner/core/operator/physicalop/physical_topn.go + - pkg/planner/core/util.go + - pkg/planner/property/physical_property.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? The planner part 1 of the partial order by enhancement 1. The new physical property + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63290-planner-the-greedy-join-order-algorithm-can-t-select-the-lowest-cost-plan-since-.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63290-planner-the-greedy-join-order-algorithm-can-t-select-the-lowest-cost-plan-since-.md new file mode 100644 index 0000000..7da6a58 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63290-planner-the-greedy-join-order-algorithm-can-t-select-the-lowest-cost-plan-since-.md @@ -0,0 +1,56 @@ +# Issue #63290: planner: the greedy join order algorithm can't select the lowest-cost plan since it skips cartesian join when comparing join cost + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63290 +- Status: closed +- Type: type/enhancement +- Created At: 2025-08-31T09:23:20Z +- Closed At: 2025-09-09T15:44:59Z +- Labels: plan-rewrite, planner/join-order, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- In the case above, the best join order should be , but since our current join order algorithm always use the minimal table as the leading table and skip cartesian joins when comparing join cost, it can't select the lowest cost join in this case. In this case, is the minimal table, and has no join key with , so our optimizer would use as the join order. + +## Linked PRs +- Fix PR #63309: planner: allow cartesian joins in greedy join order algo to explore better join orders + URL: https://github.com/pingcap/tidb/pull/63309 + State: closed + Merged At: 2025-09-09T15:44:58Z + Changed Files Count: 10 + Main Modules: pkg/planner/core, pkg/session, pkg/bindinfo + Sample Changed Files: + - pkg/bindinfo/binding_plan_generation.go + - pkg/bindinfo/binding_plan_generation_test.go + - pkg/planner/core/casetest/BUILD.bazel + - pkg/planner/core/casetest/integration_test.go + - pkg/planner/core/rule_join_reorder.go + - pkg/planner/core/rule_join_reorder_greedy.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/setvar_affect.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: What problem does this PR solve? Problem Summary: planner: allow cartesian joins in greedy join order algo to explore better join orders What changed and how does it work? In the current greedy join order algo implementation, we skip cartesian joins, which means we join tables having join keys with each others first. But this strategy might miss some optimal join orders (see #63290 as an example). This PR allows cartesian joins in the greedy join order algo, and use a to trade off the risk. +- Related PR #65705: planner: fix join reorder correctness with conflict detection algorithm + URL: https://github.com/pingcap/tidb/pull/65705 + State: closed + Merged At: 2026-02-13T04:11:59Z + Changed Files Count: 13 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/base/BUILD.bazel + - pkg/planner/core/base/plan_base.go + - pkg/planner/core/casetest/rule/BUILD.bazel + - pkg/planner/core/casetest/rule/main_test.go + - pkg/planner/core/casetest/rule/rule_cdc_join_reorder_test.go + - pkg/planner/core/casetest/rule/testdata/cdc_join_reorder_suite_in.json + - pkg/planner/core/casetest/rule/testdata/cdc_join_reorder_suite_out.json + - pkg/planner/core/casetest/rule/testdata/cdc_join_reorder_suite_xut.json + - pkg/planner/core/joinorder/BUILD.bazel + - pkg/planner/core/joinorder/bitset_bench_test.go + - pkg/planner/core/joinorder/conflict_detector.go + - pkg/planner/core/joinorder/join_order.go + - pkg/planner/core/rule_join_reorder.go + PR Summary: What problem does this PR solve? Problem Summary: While TiDB's Join Reorder module has supported both INNER JOIN and OUTER JOIN for a very long time, it continues to suffer from persistent correctness issues. There issues, analyzed over the last three years, basically fall into three categories: 1. Correctness problem: 1. case-1: Lack of effective validation logic for OUTER JOIN reordering: The current detection logic for OUTER JOINs is simple (it prevents reordering the null-producing side). This leads to implementations where the semantics after reordering deviate from the original SQL. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63314-planner-join-order-algo-can-t-select-the-lowest-cost-join-order-since-join-keys-.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63314-planner-join-order-algo-can-t-select-the-lowest-cost-join-order-since-join-keys-.md new file mode 100644 index 0000000..b7af63f --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63314-planner-join-order-algo-can-t-select-the-lowest-cost-join-order-since-join-keys-.md @@ -0,0 +1,164 @@ +# Issue #63314: planner: join order algo can't select the lowest cost join order since join keys are eliminated by constant propagation + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63314 +- Status: closed +- Type: type/enhancement +- Created At: 2025-09-01T10:46:40Z +- Closed At: 2025-09-10T02:07:49Z +- Labels: affects-8.5, planner/join-order, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- The best join order is , but the optimizer can't select it: + +## Linked PRs +- Fix PR #63375: planner: keep join keys for join optimization in constant propagation + URL: https://github.com/pingcap/tidb/pull/63375 + State: closed + Merged At: not merged + Changed Files Count: 3 + Main Modules: pkg/planner/core, pkg/expression + Sample Changed Files: + - pkg/expression/constant_propagation.go + - pkg/planner/core/casetest/join/BUILD.bazel + - pkg/planner/core/casetest/join/join_test.go + PR Summary: What problem does this PR solve? Problem Summary: planner: keep join keys for join optimization in constant propagation What changed and how does it work? +- Fix PR #63385: planner: refactoring, use the special function to propagate constant for Joins + URL: https://github.com/pingcap/tidb/pull/63385 + State: closed + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/rule/rule_init.go + - pkg/planner/core/rule/rule_predicate_simplification.go + - pkg/planner/core/rule/util/misc.go + PR Summary: What problem does this PR solve? Problem Summary: planner: refactoring, use the special function to propagate constant for Joins What changed and how does it work? planner: refactoring, use the special function to propagate constant for Joins +- Fix PR #63388: planner: refactor some code related to constant propagation for join + URL: https://github.com/pingcap/tidb/pull/63388 + State: closed + Merged At: 2025-09-05T12:55:36Z + Changed Files Count: 7 + Main Modules: pkg/planner/core, pkg/expression, pkg/planner + Sample Changed Files: + - pkg/expression/constant_propagation.go + - pkg/expression/constant_test.go + - pkg/planner/cascades/old/transformation_rules.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/rule/rule_init.go + - pkg/planner/core/rule/rule_predicate_simplification.go + - pkg/planner/core/rule/util/misc.go + PR Summary: What problem does this PR solve? Problem Summary: planner: refactor some code related to constant propagation for join What changed and how does it work? just refactor, no logical change +- Fix PR #63404: planner: keep join keys for join optimization in constant propagation | tidb-test=pr/2600 + URL: https://github.com/pingcap/tidb/pull/63404 + State: closed + Merged At: 2025-09-10T02:07:48Z + Changed Files Count: 12 + Main Modules: pkg/planner/core, pkg/session, pkg/expression, pkg/bindinfo, pkg/planner + Sample Changed Files: + - pkg/bindinfo/binding_plan_generation.go + - pkg/expression/constant_propagation.go + - pkg/expression/constant_test.go + - pkg/planner/cascades/old/transformation_rules.go + - pkg/planner/core/casetest/join/BUILD.bazel + - pkg/planner/core/casetest/join/join_test.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/rule/rule_predicate_simplification.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/setvar_affect.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: What problem does this PR solve? Problem Summary: planner: keep join keys for join optimization in constant propagation What changed and how does it work? +- Fix PR #63470: planner: always keep join keys by default | tidb-test=pr/2602 + URL: https://github.com/pingcap/tidb/pull/63470 + State: closed + Merged At: 2025-09-12T05:01:52Z + Changed Files Count: 19 + Main Modules: tests/integrationtest, pkg/planner/core, pkg/executor, pkg/session + Sample Changed Files: + - pkg/executor/explainfor_test.go + - pkg/executor/prepared_test.go + - pkg/executor/testdata/prepare_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_out.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_xut.json + - pkg/planner/core/issuetest/planner_issue_test.go + - pkg/planner/core/testdata/plan_suite_unexported_out.json + - pkg/sessionctx/vardef/tidb_vars.go + - tests/integrationtest/r/agg_predicate_pushdown.result + - tests/integrationtest/r/collation_misc_enabled.result + - tests/integrationtest/r/executor/explain.result + - tests/integrationtest/r/executor/explainfor.result + - tests/integrationtest/r/executor/merge_join.result + - tests/integrationtest/r/executor/point_get.result + - tests/integrationtest/r/explain_foreign_key.result + - tests/integrationtest/r/expression/issues.result + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + PR Summary: What problem does this PR solve? Problem Summary: planner: always keep join keys by default What changed and how does it work? I went through all changes and all seem excepted and have the same situation. Before we can only choose since there is no join key, while after this PR we can have more join choices like MergeJoin and IndexJoin because we can only choose them if we have join keys: +- Fix PR #64798: planner: refactor some code related to constant propagation for join (#63388) + URL: https://github.com/pingcap/tidb/pull/64798 + State: closed + Merged At: not merged + Changed Files Count: 7 + Main Modules: pkg/planner/core, pkg/expression, pkg/planner + Sample Changed Files: + - pkg/expression/constant_propagation.go + - pkg/expression/constant_test.go + - pkg/planner/cascades/transformation_rules.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/rule/rule_init.go + - pkg/planner/core/rule/util/misc.go + - pkg/planner/core/rule_predicate_simplification.go + PR Summary: This is an automated cherry-pick of #63388 What problem does this PR solve? Problem Summary: planner: refactor some code related to constant propagation for join What changed and how does it work? just refactor, no logical change +- Fix PR #64799: planner: keep join keys for join optimization in constant propagation | tidb-test=pr/2600 (#63404) + URL: https://github.com/pingcap/tidb/pull/64799 + State: open + Merged At: not merged + Changed Files Count: 12 + Main Modules: pkg/planner/core, pkg/session, pkg/expression, pkg/bindinfo, pkg/planner + Sample Changed Files: + - pkg/bindinfo/binding_plan_generation.go + - pkg/expression/constant_propagation.go + - pkg/expression/constant_test.go + - pkg/planner/cascades/transformation_rules.go + - pkg/planner/core/casetest/join/BUILD.bazel + - pkg/planner/core/casetest/join/join_test.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/rule/rule_predicate_simplification.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/setvar_affect.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: This is an automated cherry-pick of #63404 What problem does this PR solve? Problem Summary: planner: keep join keys for join optimization in constant propagation What changed and how does it work? +- Fix PR #64800: planner: always keep join keys by default | tidb-test=pr/2602 (#63470) + URL: https://github.com/pingcap/tidb/pull/64800 + State: open + Merged At: not merged + Changed Files Count: 19 + Main Modules: tests/integrationtest, pkg/planner/core, pkg/executor, pkg/session + Sample Changed Files: + - pkg/executor/explainfor_test.go + - pkg/executor/prepared_test.go + - pkg/executor/testdata/prepare_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_out.json + - pkg/planner/core/casetest/partition/testdata/partition_pruner_xut.json + - pkg/planner/core/issuetest/planner_issue_test.go + - pkg/planner/core/testdata/plan_suite_unexported_out.json + - pkg/sessionctx/vardef/tidb_vars.go + - tests/integrationtest/r/agg_predicate_pushdown.result + - tests/integrationtest/r/collation_misc_enabled.result + - tests/integrationtest/r/executor/explain.result + - tests/integrationtest/r/executor/explainfor.result + - tests/integrationtest/r/executor/merge_join.result + - tests/integrationtest/r/executor/point_get.result + - tests/integrationtest/r/explain_foreign_key.result + - tests/integrationtest/r/expression/issues.result + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + PR Summary: This is an automated cherry-pick of #63470 What problem does this PR solve? Problem Summary: planner: always keep join keys by default What changed and how does it work? I went through all changes and all seem excepted and have the same situation. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63336-planner-a-new-risk-aware-index-selection-heuristic-rule-for-indexlookup-in-skyli.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63336-planner-a-new-risk-aware-index-selection-heuristic-rule-for-indexlookup-in-skyli.md new file mode 100644 index 0000000..ca12624 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63336-planner-a-new-risk-aware-index-selection-heuristic-rule-for-indexlookup-in-skyli.md @@ -0,0 +1,43 @@ +# Issue #63336: planner: a new risk-aware index selection heuristic rule for IndexLookup in skyline pruning + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63336 +- Status: open +- Type: type/enhancement +- Created At: 2025-09-02T10:44:08Z +- Labels: affects-8.5, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- In the case above, the optimizer is prone to select since its estimated rows is slightly smaller than . But from the perspective of risk, especially avoiding large number of cop-tasks, the plan using seems lower-risky, since it contains all columns of , which means at least after index accessing / filtering, has less rows to trigger cop-tasks. + +## Linked PRs +- Fix PR #63349: planner: refactor compareCandidates logic | tidb-test=pr/2594 + URL: https://github.com/pingcap/tidb/pull/63349 + State: closed + Merged At: 2025-09-04T15:49:12Z + Changed Files Count: 5 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/testdata/plan_normalized_suite_out.json + - pkg/planner/core/casetest/testdata/plan_normalized_suite_xut.json + - pkg/planner/core/find_best_task.go + - pkg/planner/core/logical_plans_test.go + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? The following changes to the compareCandidates logic: 1. Factor out pseudo index comparison logic to simplify compareCandidates. +- Fix PR #63591: planner: refactor compareCandidates logic (#63349) | tidb-test=pr/2610 + URL: https://github.com/pingcap/tidb/pull/63591 + State: closed + Merged At: 2025-09-23T21:09:46Z + Changed Files Count: 6 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/testdata/plan_normalized_suite_out.json + - pkg/planner/core/casetest/testdata/plan_normalized_suite_xut.json + - pkg/planner/core/find_best_task.go + - pkg/planner/core/logical_plans_test.go + - tests/integrationtest/r/imdbload.result + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + PR Summary: This is an automated cherry-pick of #63349 What problem does this PR solve? Problem Summary: What changed and how does it work? The following changes to the compareCandidates logic: + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63353-dp-join-reorder-on-the-graph-that-is-not-totally-connect-may-fail-to-execute.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63353-dp-join-reorder-on-the-graph-that-is-not-totally-connect-may-fail-to-execute.md new file mode 100644 index 0000000..9b43ba9 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63353-dp-join-reorder-on-the-graph-that-is-not-totally-connect-may-fail-to-execute.md @@ -0,0 +1,39 @@ +# Issue #63353: DP Join reorder on the graph that is not totally connect may fail to execute + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63353 +- Status: closed +- Type: type/bug +- Created At: 2025-09-03T07:46:55Z +- Closed At: 2025-09-22T13:40:11Z +- Labels: affects-7.1, affects-7.5, affects-8.1, affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #63648: planner: fix the possble wrong plan for dp join reorder + URL: https://github.com/pingcap/tidb/pull/63648 + State: closed + Merged At: 2025-09-22T13:40:10Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/rule_join_reorder_dp.go + - tests/integrationtest/r/planner/core/issuetest/planner_issue.result + - tests/integrationtest/t/planner/core/issuetest/planner_issue.test + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? When the join graph is not totally connected, we may do BFS multiple times to get the connected subgraph. During each bfs, we'll assign the node in the subgraph with an ID that is its BFS order. +- Fix PR #63669: planner: fix the possble wrong plan for dp join reorder (#63648) + URL: https://github.com/pingcap/tidb/pull/63669 + State: closed + Merged At: 2025-09-23T05:57:25Z + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/rule_join_reorder_dp.go + - tests/integrationtest/r/planner/core/issuetest/planner_issue.result + - tests/integrationtest/t/planner/core/issuetest/planner_issue.test + PR Summary: This is an automated cherry-pick of #63648 What problem does this PR solve? Problem Summary: What changed and how does it work? When the join graph is not totally connected, we may do BFS multiple times to get the connected subgraph. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63369-tiflash-query-result-materialization-can-t-take-effect-after-setting-sql-mode-by.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63369-tiflash-query-result-materialization-can-t-take-effect-after-setting-sql-mode-by.md new file mode 100644 index 0000000..13517d0 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63369-tiflash-query-result-materialization-can-t-take-effect-after-setting-sql-mode-by.md @@ -0,0 +1,47 @@ +# Issue #63369: TiFlash query result materialization can't take effect after setting sql_mode by set_var + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63369 +- Status: closed +- Type: type/bug +- Created At: 2025-09-04T06:39:18Z +- Closed At: 2025-09-16T07:54:35Z +- Labels: affects-7.5, affects-8.1, affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- The [sql_mode check]() for result materialization occurs before [set_var action](), it cannot take effect. + +## Linked PRs +- Fix PR #63380: planner: fix `set_var` not working for write statements to read on TiFlash + URL: https://github.com/pingcap/tidb/pull/63380 + State: closed + Merged At: 2025-09-15T15:34:31Z + Changed Files Count: 2 + Main Modules: pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/core/integration_test.go + - pkg/planner/optimize.go + PR Summary: What problem does this PR solve? Problem Summary: See has fixed this issue, but it also made other changes, making it unsuitable for cherry-picking to other release branches. This PR fixes the bug on release-8.5 and will be cherry-picked to older release branches. +- Related PR #63531: planner: fix `set_var` not working for write statements to read on TiFlash (#63380) + URL: https://github.com/pingcap/tidb/pull/63531 + State: open + Merged At: not merged + Changed Files Count: 2 + Main Modules: pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/core/integration_test.go + - pkg/planner/optimize.go + PR Summary: This is an automated cherry-pick of #63380 What problem does this PR solve? Problem Summary: See has fixed this issue, but it also made other changes, making it unsuitable for cherry-picking to other release branches. +- Related PR #63532: planner: fix `set_var` not working for write statements to read on TiFlash (#63380) + URL: https://github.com/pingcap/tidb/pull/63532 + State: open + Merged At: not merged + Changed Files Count: 2 + Main Modules: pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/core/integration_test.go + - pkg/planner/optimize.go + PR Summary: This is an automated cherry-pick of #63380 What problem does this PR solve? Problem Summary: See has fixed this issue, but it also made other changes, making it unsuitable for cherry-picking to other release branches. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63370-limit-doesn-t-pushdown-to-tikv-automatically.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63370-limit-doesn-t-pushdown-to-tikv-automatically.md new file mode 100644 index 0000000..092332b --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63370-limit-doesn-t-pushdown-to-tikv-automatically.md @@ -0,0 +1,80 @@ +# Issue #63370: Limit doesn't pushdown to tikv automatically + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63370 +- Status: closed +- Type: type/bug +- Created At: 2025-09-04T07:04:04Z +- Closed At: 2025-09-10T16:29:20Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- limit didn't pushdown to tikv: + +## Linked PRs +- Fix PR #63399: planner: limit pushdown shouldn't be affected by tidb_opt_limit_push_down_threshold | tidb-test=pr/2601 + URL: https://github.com/pingcap/tidb/pull/63399 + State: closed + Merged At: 2025-09-10T16:29:19Z + Changed Files Count: 12 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/physicalplantest/BUILD.bazel + - pkg/planner/core/casetest/physicalplantest/physical_plan_test.go + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_in.json + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_out.json + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/exhaust_physical_plans.go + PR Summary: What problem does this PR solve? Problem Summary: Limit pushdown shouldn’t be affected by , so at first glance this description looks a bit strange. In practice, limit pushdown should always happen since it has no side effects. Therefore, only impacts TopN pushdown, not limit pushdown. +- Fix PR #63760: planner: limit pushdown shouldn't be affected by tidb_opt_limit_push_down_threshold | tidb-test=pr/2601 (#63399) + URL: https://github.com/pingcap/tidb/pull/63760 + State: closed + Merged At: not merged + Changed Files Count: 0 + PR Summary: This is an automated cherry-pick of #63399 What problem does this PR solve? Problem Summary: Limit pushdown shouldn’t be affected by , so at first glance this description looks a bit strange. In practice, limit pushdown should always happen since it has no side effects. +- Fix PR #63761: planner: limit pushdown shouldn't be affected by tidb_opt_limit_push_down_threshold | tidb-test=pr/2601 (#63399) + URL: https://github.com/pingcap/tidb/pull/63761 + State: open + Merged At: not merged + Changed Files Count: 12 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/physicalplantest/BUILD.bazel + - pkg/planner/core/casetest/physicalplantest/physical_plan_test.go + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_in.json + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_out.json + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/exhaust_physical_plans.go + PR Summary: This is an automated cherry-pick of #63399 What problem does this PR solve? Problem Summary: Limit pushdown shouldn’t be affected by , so at first glance this description looks a bit strange. In practice, limit pushdown should always happen since it has no side effects. +- Fix PR #63762: planner: limit pushdown shouldn't be affected by tidb_opt_limit_push_down_threshold | tidb-test=pr/2601 (#63399) + URL: https://github.com/pingcap/tidb/pull/63762 + State: closed + Merged At: 2025-09-29T08:32:16Z + Changed Files Count: 9 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/hint/testdata/integration_suite_out.json + - pkg/planner/core/casetest/physicalplantest/testdata/plan_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/plan_normalized_suite_out.json + - pkg/planner/core/exhaust_physical_plans.go + - tests/integrationtest/r/executor/issues.result + - tests/integrationtest/r/planner/core/rule_result_reorder.result + PR Summary: This is an automated cherry-pick of #63399 What problem does this PR solve? Problem Summary: Limit pushdown shouldn’t be affected by , so at first glance this description looks a bit strange. In practice, limit pushdown should always happen since it has no side effects. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63410-binding-create-binding-from-plan-digest-can-t-distinguish-different-plans-with-s.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63410-binding-create-binding-from-plan-digest-can-t-distinguish-different-plans-with-s.md new file mode 100644 index 0000000..dc9a11d --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63410-binding-create-binding-from-plan-digest-can-t-distinguish-different-plans-with-s.md @@ -0,0 +1,17 @@ +# Issue #63410: binding: `create binding from {plan_digest}` can't distinguish different plans with same digest but different schema + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63410 +- Status: open +- Type: type/bug +- Created At: 2025-09-08T09:23:09Z +- Labels: component/bindinfo, may-affects-6.5, may-affects-7.1, may-affects-7.5, may-affects-8.1, may-affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- They have the same plan_digest. + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63417-planner-expand-the-dnf-expression-to-construct-precise-scan-ranges.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63417-planner-expand-the-dnf-expression-to-construct-precise-scan-ranges.md new file mode 100644 index 0000000..fbe7e02 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63417-planner-expand-the-dnf-expression-to-construct-precise-scan-ranges.md @@ -0,0 +1,17 @@ +# Issue #63417: planner: expand the DNF expression to construct precise scan ranges + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63417 +- Status: open +- Type: type/enhancement +- Created At: 2025-09-09T03:21:38Z +- Labels: plan-rewrite, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- In the case above, we can only use the first 2 columns in the index to construct the scan ranges: . Actually we can expand and rewrite the DNF with this formula: . Then we'll get a precise and better key scan range: + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63492-hash-aggregation-may-not-be-pushed-down-to-tikv-during-admin-check-table.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63492-hash-aggregation-may-not-be-pushed-down-to-tikv-during-admin-check-table.md new file mode 100644 index 0000000..4f6580f --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63492-hash-aggregation-may-not-be-pushed-down-to-tikv-during-admin-check-table.md @@ -0,0 +1,41 @@ +# Issue #63492: Hash aggregation may not be pushed down to TiKV during `ADMIN CHECK TABLE` + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63492 +- Status: closed +- Type: type/bug +- Created At: 2025-09-12T09:19:44Z +- Closed At: 2025-10-11T15:41:46Z +- Labels: affects-7.5, affects-8.1, affects-8.5, component/ddl, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- Hash aggregation may not be pushed down to TiKV during + +## Linked PRs +- Fix PR #63665: executor: force pushdown aggregate to TiKV inside `admin check table` + URL: https://github.com/pingcap/tidb/pull/63665 + State: closed + Merged At: 2025-10-11T15:41:45Z + Changed Files Count: 4 + Main Modules: pkg/ddl, pkg/executor, pkg/server, pkg/session + Sample Changed Files: + - pkg/ddl/tests/partition/multi_domain_test.go + - pkg/executor/check_table_index.go + - pkg/server/tests/commontest/tidb_test.go + - pkg/sessionctx/variable/session.go + PR Summary: What problem does this PR solve? Problem Summary: The aggregate may not push down to TiKV What changed and how does it work? Force pushdown aggregate to TiKV +- Fix PR #64045: executor: force pushdown aggregate to TiKV inside `admin check table` (#63665) + URL: https://github.com/pingcap/tidb/pull/64045 + State: closed + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/ddl, pkg/executor, pkg/server, pkg/session + Sample Changed Files: + - pkg/ddl/tests/partition/multi_domain_test.go + - pkg/executor/check_table_index.go + - pkg/server/tests/commontest/tidb_test.go + - pkg/sessionctx/variable/session.go + PR Summary: This is an automated cherry-pick of #63665 What problem does this PR solve? Problem Summary: The aggregate may not push down to TiKV What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63573-true-or-expressions-are-not-always-short-circuit.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63573-true-or-expressions-are-not-always-short-circuit.md new file mode 100644 index 0000000..da425b2 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63573-true-or-expressions-are-not-always-short-circuit.md @@ -0,0 +1,17 @@ +# Issue #63573: TRUE OR ... expressions are not always short-circuit + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63573 +- Status: open +- Type: type/bug +- Created At: 2025-09-17T10:32:28Z +- Labels: report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- It executes the subquery, taking much longer time than needed. + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63602-statement-with-scalar-subquery-result-wrong.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63602-statement-with-scalar-subquery-result-wrong.md new file mode 100644 index 0000000..f7fabdc --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63602-statement-with-scalar-subquery-result-wrong.md @@ -0,0 +1,53 @@ +# Issue #63602: statement with scalar subquery result wrong + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63602 +- Status: closed +- Type: type/bug +- Created At: 2025-09-18T08:21:32Z +- Closed At: 2025-09-23T07:52:17Z +- Labels: affects-8.5, contribution, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #63637: planner: refactor outer to inner join | tidb-test=pr/2612 + URL: https://github.com/pingcap/tidb/pull/63637 + State: closed + Merged At: 2025-09-23T07:52:16Z + Changed Files Count: 9 + Main Modules: pkg/planner/core, pkg/executor + Sample Changed Files: + - pkg/executor/test/indexmergereadtest/index_merge_reader_test.go + - pkg/planner/core/casetest/parallelapply/parallel_apply_test.go + - pkg/planner/core/casetest/rule/BUILD.bazel + - pkg/planner/core/casetest/rule/rule_outer2inner_test.go + - pkg/planner/core/casetest/rule/testdata/outer2inner_out.json + - pkg/planner/core/casetest/rule/testdata/outer2inner_xut.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/operator/logicalop/BUILD.bazel + - pkg/planner/core/operator/logicalop/logical_join.go + PR Summary: What problem does this PR solve? close #61327 Problem Summary: What changed and how does it work? revert #52941 , Our has problems and it will get wrong transfer. +- Related PR #63688: planner: refactor outer to inner join | tidb-test=pr/2614 + URL: https://github.com/pingcap/tidb/pull/63688 + State: closed + Merged At: 2025-09-24T16:32:19Z + Changed Files Count: 11 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/executor, pkg/session + Sample Changed Files: + - pkg/executor/test/indexmergereadtest/index_merge_reader_test.go + - pkg/planner/core/casetest/parallelapply/parallel_apply_test.go + - pkg/planner/core/casetest/rule/BUILD.bazel + - pkg/planner/core/casetest/rule/rule_outer2inner_test.go + - pkg/planner/core/casetest/rule/testdata/outer2inner_out.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/operator/logicalop/BUILD.bazel + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/session/main_test.go + - tests/integrationtest/r/planner/core/rule_outer2inner.result + - tests/integrationtest/t/planner/core/rule_outer2inner.test + PR Summary: This is an automated cherry-pick of #63637 What problem does this PR solve? Problem Summary: What changed and how does it work? revert #52941 , Our has problems and it will get wrong transfer. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63805-prepare-query-with-order-by-clause-does-not-work-with-sql-binding.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63805-prepare-query-with-order-by-clause-does-not-work-with-sql-binding.md new file mode 100644 index 0000000..d7a6784 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63805-prepare-query-with-order-by-clause-does-not-work-with-sql-binding.md @@ -0,0 +1,27 @@ +# Issue #63805: Prepare query with ORDER BY clause does not work with sql binding + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63805 +- Status: open +- Type: type/bug +- Created At: 2025-09-29T10:47:13Z +- Labels: affects-8.5, good first issue, help wanted, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- The binding doesn't work. + +## Linked PRs +- Fix PR #63903: planner: fix binding usage for prepare query with order by clause + URL: https://github.com/pingcap/tidb/pull/63903 + State: open + Merged At: not merged + Changed Files Count: 3 + Main Modules: pkg/bindinfo, pkg/util + Sample Changed Files: + - pkg/bindinfo/binding_operator_test.go + - pkg/bindinfo/session_handle_test.go + - pkg/util/parser/ast.go + PR Summary: What problem does this PR solve? Problem Summary: Using prepare statement with ORDER BY doesn't use bindings What changed and how does it work? Possible bindings are selected by comparing the normalized statement for the binding with the current running statement. Since the prepared statements are stored based on the plan from , there is additional handling in which changes the in the AST. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63985-plan-changed-unexpectedly-because-expression-index-wasn-t-chosen-by-default.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63985-plan-changed-unexpectedly-because-expression-index-wasn-t-chosen-by-default.md new file mode 100644 index 0000000..2f9e626 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-63985-plan-changed-unexpectedly-because-expression-index-wasn-t-chosen-by-default.md @@ -0,0 +1,82 @@ +# Issue #63985: plan changed unexpectedly because expression index wasn't chosen by default + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/63985 +- Status: closed +- Type: type/bug +- Created At: 2025-10-16T01:56:43Z +- Closed At: 2025-11-05T17:02:54Z +- Labels: affects-8.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- you can reproduce this problem using dataset as And I think this is introduced by recent commit between be3ba74ef819842dae045121eec5f3f6f0aaf1a7 and 3e7f31765ebe78bea7e4fdbd762180e20ade300b + +## Linked PRs +- Related PR #63961: planner: fix uninitialized stats for expression index + URL: https://github.com/pingcap/tidb/pull/63961 + State: closed + Merged At: 2025-10-16T07:16:59Z + Changed Files Count: 3 + Main Modules: pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/selectivity.go + - pkg/planner/cardinality/selectivity_test.go + PR Summary: What problem does this PR solve? Problem Summary: For the following case, the explain analyze output shows for that expression index(), which is confusing. For expression index , there will be both original column stats and index stats when [calculating selectivity](). Check the following debug output, you can see the column stats is and the index stats is ready. So for expression index, should use index stats and its column stats. +- Fix PR #64151: planner: fix prefer task logic when enumerate physical limit/topn | tidb-test=pr/2622 + URL: https://github.com/pingcap/tidb/pull/64151 + State: closed + Merged At: 2025-11-05T17:02:53Z + Changed Files Count: 10 + Main Modules: pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/cascades/memo/group_expr.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/operator/physicalop/physical_topn.go + PR Summary: What problem does this PR solve? Problem Summary: check for detail. What changed and how does it work? there will be two kinds of tasks: 1. hintTask: this task was chosen because of hint +- Fix PR #64307: planner: fix prefer task logic when enumerate physical limit/topn + URL: https://github.com/pingcap/tidb/pull/64307 + State: closed + Merged At: 2025-11-06T06:41:44Z + Changed Files Count: 10 + Main Modules: pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/cascades/memo/group_expr.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/operator/physicalop/physical_topn.go + PR Summary: What problem does this PR solve? Problem Summary: this is for hotfix. cherry-pick manually What changed and how does it work? +- Fix PR #65134: planner: fix prefer task logic when enumerate physical limit/topn | tidb-test=pr/2622 (#64151) + URL: https://github.com/pingcap/tidb/pull/65134 + State: closed + Merged At: not merged + Changed Files Count: 10 + Main Modules: pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/cascades/memo/group_expr.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_out.json + - pkg/planner/core/casetest/dag/testdata/plan_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/operator/physicalop/physical_topn.go + PR Summary: This is an automated cherry-pick of #64151 What problem does this PR solve? Problem Summary: check for detail. What changed and how does it work? there will be two kinds of tasks: + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64137-planner-consider-empty-histogram-and-newly-emerging-values-in-out-of-range-estim.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64137-planner-consider-empty-histogram-and-newly-emerging-values-in-out-of-range-estim.md new file mode 100644 index 0000000..d5126f8 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64137-planner-consider-empty-histogram-and-newly-emerging-values-in-out-of-range-estim.md @@ -0,0 +1,57 @@ +# Issue #64137: planner: consider empty histogram and newly emerging values in out-of-range estimation for EQ conditions more properly + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64137 +- Status: closed +- Type: type/bug +- Created At: 2025-10-27T08:47:23Z +- Closed At: 2025-10-31T10:52:37Z +- Labels: affects-8.5, epic/cardinality-estimation, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- A large estimation error . + +## Linked PRs +- Fix PR #64139: planner: set an lower-bound for NDV used in out-of-range estimation for EQ conditions when the Histogram is empty | tidb-test=pr/2623 + URL: https://github.com/pingcap/tidb/pull/64139 + State: closed + Merged At: 2025-10-31T10:52:36Z + Changed Files Count: 6 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + PR Summary: What problem does this PR solve? Problem Summary: planner: consider empty histogram and newly emerging values in out-of-range estimation for EQ conditions more properly What changed and how does it work? See for more details. Hard to create a stable and graceful test case for this issue, so I tested it locally and here is the result: +- Fix PR #64220: planner: set an lower-bound for NDV used in out-of-range estimation for EQ conditions when the Histogram is empty | tidb-test=pr/2623 (#64139) + URL: https://github.com/pingcap/tidb/pull/64220 + State: closed + Merged At: not merged + Changed Files Count: 6 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + PR Summary: This is an automated cherry-pick of #64139 What problem does this PR solve? Problem Summary: planner: consider empty histogram and newly emerging values in out-of-range estimation for EQ conditions more properly What changed and how does it work? See for more details. +- Fix PR #64221: planner: set an lower-bound for NDV used in out-of-range estimation for EQ conditions when the Histogram is empty | tidb-test=pr/2623 (#64139) + URL: https://github.com/pingcap/tidb/pull/64221 + State: closed + Merged At: 2025-10-31T16:45:27Z + Changed Files Count: 4 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + PR Summary: This is an automated cherry-pick of #64139 What problem does this PR solve? Problem Summary: planner: consider empty histogram and newly emerging values in out-of-range estimation for EQ conditions more properly What changed and how does it work? See for more details. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64205-support-dump-plan-replayer-for-prepare-stmt.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64205-support-dump-plan-replayer-for-prepare-stmt.md new file mode 100644 index 0000000..0b0543f --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64205-support-dump-plan-replayer-for-prepare-stmt.md @@ -0,0 +1,30 @@ +# Issue #64205: support dump plan replayer for prepare stmt + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64205 +- Status: open +- Type: type/enhancement +- Created At: 2025-10-31T03:19:18Z +- Labels: report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- We can support like the following: + +## Linked PRs +- Fix PR #66129: executor: support dump plan replayer for prepare stmt + URL: https://github.com/pingcap/tidb/pull/66129 + State: open + Merged At: not merged + Changed Files Count: 6 + Main Modules: pkg/parser, pkg/executor + Sample Changed Files: + - pkg/executor/plan_replayer.go + - pkg/executor/test/planreplayer/plan_replayer_test.go + - pkg/parser/ast/misc_test.go + - pkg/parser/parser.go + - pkg/parser/parser.y + - pkg/parser/parser_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? support dump plan replayer for prepare stmt + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64217-planner-large-estimation-error-of-probe-side-of-indexjoin-when-table-filter-sele.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64217-planner-large-estimation-error-of-probe-side-of-indexjoin-when-table-filter-sele.md new file mode 100644 index 0000000..6c44b30 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64217-planner-large-estimation-error-of-probe-side-of-indexjoin-when-table-filter-sele.md @@ -0,0 +1,17 @@ +# Issue #64217: planner: large estimation error of probe-side of IndexJoin when table filter selectivity is extremely low + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64217 +- Status: open +- Type: type/enhancement +- Created At: 2025-10-31T08:23:17Z +- Labels: epic/cardinality-estimation, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- Large estimation error + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64250-global-binding-cache-may-be-incorrect-after-concurrent-binding-operations-on-mul.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64250-global-binding-cache-may-be-incorrect-after-concurrent-binding-operations-on-mul.md new file mode 100644 index 0000000..2c599c9 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64250-global-binding-cache-may-be-incorrect-after-concurrent-binding-operations-on-mul.md @@ -0,0 +1,66 @@ +# Issue #64250: global binding cache may be incorrect after concurrent binding operations on multiple tidb nodes + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64250 +- Status: closed +- Type: type/bug +- Created At: 2025-11-03T14:54:59Z +- Closed At: 2025-11-07T10:33:16Z +- Labels: affects-7.5, affects-8.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- There are fewer than 200 bindings after creating, and some remaining bindings at last. + +## Linked PRs +- Fix PR #64289: planner: tolerate reasonable time lag between different TiDB nodes when updating binding cache + URL: https://github.com/pingcap/tidb/pull/64289 + State: closed + Merged At: 2025-11-07T10:33:15Z + Changed Files Count: 4 + Main Modules: pkg/bindinfo + Sample Changed Files: + - pkg/bindinfo/BUILD.bazel + - pkg/bindinfo/binding_cache.go + - pkg/bindinfo/binding_operator.go + - pkg/bindinfo/binding_operator_test.go + PR Summary: What problem does this PR solve? Problem Summary: planner: tolerate reasonable time lag between different TiDB nodes when updating binding cache What changed and how does it work? When updating binding cache, TiDB doesn't consider time lag between different nodes, which could cause inconsistency on binding cache. See more info in #64250. Below is an simplified example, TiDB1's clock is consistent with the real time, while TiDB2's clock is 0.5s slower than the real time: +- Fix PR #64353: planner: tolerate reasonable time lag between different TiDB nodes when updating binding cache (#64289) + URL: https://github.com/pingcap/tidb/pull/64353 + State: open + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/bindinfo + Sample Changed Files: + - pkg/bindinfo/BUILD.bazel + - pkg/bindinfo/binding_cache.go + - pkg/bindinfo/binding_operator.go + - pkg/bindinfo/binding_operator_test.go + PR Summary: This is an automated cherry-pick of #64289 What problem does this PR solve? Problem Summary: planner: tolerate reasonable time lag between different TiDB nodes when updating binding cache What changed and how does it work? When updating binding cache, TiDB doesn't consider time lag between different nodes, which could cause inconsistency on binding cache. See more info in #64250. +- Fix PR #64354: planner: tolerate reasonable time lag between different TiDB nodes when updating binding cache (#64289) + URL: https://github.com/pingcap/tidb/pull/64354 + State: closed + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/bindinfo + Sample Changed Files: + - pkg/bindinfo/BUILD.bazel + - pkg/bindinfo/binding_cache.go + - pkg/bindinfo/binding_operator.go + - pkg/bindinfo/binding_operator_test.go + PR Summary: This is an automated cherry-pick of #64289 What problem does this PR solve? Problem Summary: planner: tolerate reasonable time lag between different TiDB nodes when updating binding cache What changed and how does it work? When updating binding cache, TiDB doesn't consider time lag between different nodes, which could cause inconsistency on binding cache. See more info in #64250. +- Fix PR #64355: planner: tolerate reasonable time lag between different TiDB nodes when updating binding cache (#64289) + URL: https://github.com/pingcap/tidb/pull/64355 + State: closed + Merged At: 2025-11-10T06:37:08Z + Changed Files Count: 5 + Main Modules: pkg/bindinfo, pkg/executor, build/nogo_config.json + Sample Changed Files: + - build/nogo_config.json + - pkg/bindinfo/global_handle.go + - pkg/bindinfo/global_handle_test.go + - pkg/executor/test/showtest/main_test.go + - pkg/executor/test/showtest/show_test.go + PR Summary: This is an automated cherry-pick of #64289 What problem does this PR solve? Problem Summary: planner: tolerate reasonable time lag between different TiDB nodes when updating binding cache What changed and how does it work? When updating binding cache, TiDB doesn't consider time lag between different nodes, which could cause inconsistency on binding cache. See more info in #64250. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64263-plan-cache-can-t-work-after-update-to-v8-5-3.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64263-plan-cache-can-t-work-after-update-to-v8-5-3.md new file mode 100644 index 0000000..c48d0ab --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64263-plan-cache-can-t-work-after-update-to-v8-5-3.md @@ -0,0 +1,44 @@ +# Issue #64263: plan cache can't work after update to v8.5.3 + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64263 +- Status: closed +- Type: type/bug +- Created At: 2025-11-04T10:13:10Z +- Closed At: 2025-11-10T03:15:51Z +- Labels: affects-8.5, impact/upgrade, report/customer, severity/moderate, sig/planner, type/bug, type/regression + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #64265: planner: fix wrongly identify the changes in the expression and skip the plan cache + URL: https://github.com/pingcap/tidb/pull/64265 + State: closed + Merged At: 2025-11-10T03:15:50Z + Changed Files Count: 6 + Main Modules: pkg/planner/core, pkg/executor + Sample Changed Files: + - pkg/executor/explainfor_test.go + - pkg/planner/core/casetest/rule/rule_predicate_simplification_test.go + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_in.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_out.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_xut.json + - pkg/planner/core/rule/rule_predicate_simplification.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? The issue was triggered by , The original logic would cause any two OR expressions and a scalar function to skip the plan cache, regardless of whether an expression transformation occurs. This PR introduces the correct logic to skip the plan cache only after a transformation has occurred. +- Fix PR #64376: planner: fix wrongly identify the changes in the expression and skip the plan cache (#64265) + URL: https://github.com/pingcap/tidb/pull/64376 + State: closed + Merged At: 2025-11-10T07:39:26Z + Changed Files Count: 5 + Main Modules: pkg/planner/core, pkg/executor + Sample Changed Files: + - pkg/executor/explainfor_test.go + - pkg/planner/core/casetest/rule/rule_predicate_simplification_test.go + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_in.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_out.json + - pkg/planner/core/rule_predicate_simplification.go + PR Summary: This is an automated cherry-pick of #64265 What problem does this PR solve? Problem Summary: What changed and how does it work? The issue was triggered by , The original logic would cause any two OR expressions and a scalar function to skip the plan cache, regardless of whether an expression transformation occurs. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64274-async-load-may-panic-when-processing-dropped-indexes.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64274-async-load-may-panic-when-processing-dropped-indexes.md new file mode 100644 index 0000000..ef07a64 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64274-async-load-may-panic-when-processing-dropped-indexes.md @@ -0,0 +1,30 @@ +# Issue #64274: Async load may panic when processing dropped indexes + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64274 +- Status: closed +- Type: type/bug +- Created At: 2025-11-04T21:20:24Z +- Closed At: 2025-11-08T22:42:47Z +- Labels: affects-6.5, affects-7.5, component/statistics, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- It directly accesses the IsLoadNeeded func. + +## Linked PRs +- Related PR #64292: statistics: remove the unnecessary log + URL: https://github.com/pingcap/tidb/pull/64292 + State: closed + Merged At: 2025-11-07T18:59:59Z + Changed Files Count: 5 + Main Modules: pkg/statistics, tests/realtikvtest + Sample Changed Files: + - pkg/statistics/handle/storage/BUILD.bazel + - pkg/statistics/handle/storage/read.go + - pkg/statistics/handle/storage/read_test.go + - tests/realtikvtest/statisticstest/BUILD.bazel + - tests/realtikvtest/statisticstest/statistics_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? IsLoadNeeded doesn’t check for nil internally. We should explicitly verify that it’s safe to use. The real use case the users may have is as follows: + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64276-planner-incorrect-estrows-on-projection-with-pointget.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64276-planner-incorrect-estrows-on-projection-with-pointget.md new file mode 100644 index 0000000..16f9c49 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64276-planner-incorrect-estrows-on-projection-with-pointget.md @@ -0,0 +1,17 @@ +# Issue #64276: Planner: Incorrect estRows on Projection with PointGet + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64276 +- Status: open +- Type: type/bug +- Created At: 2025-11-05T01:52:36Z +- Labels: affects-8.5, report/customer, sig/planner, type/bug + +## Customer-Facing Phenomenon +- When a parent LogicalProjection derives statistics, it copies the child’s RowCount in DeriveStats. Later in physical optimization, the plan may be rewritten by ConvertToPointGet, which updates the child’s stats.RowCount (e.g., to 1 for PointGet). However, the parent Projection’s previously copied RowCount is not refreshed, so its estRows becomes inconsistent with the child and remains significantly larger. This leads to wrong cost, sub-optimal plan choice, and misleading EXPLAIN output (parent shows thousands of rows while the child is Point_Get with 1 row). + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64278-planner-chooses-wrong-plan-due-to-the-overestimation-of-equal-condition-on-index.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64278-planner-chooses-wrong-plan-due-to-the-overestimation-of-equal-condition-on-index.md new file mode 100644 index 0000000..3786049 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64278-planner-chooses-wrong-plan-due-to-the-overestimation-of-equal-condition-on-index.md @@ -0,0 +1,43 @@ +# Issue #64278: planner chooses wrong plan due to the overestimation of equal condition on index + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64278 +- Status: closed +- Type: type/bug +- Created At: 2025-11-05T03:27:13Z +- Closed At: 2025-11-06T08:32:39Z +- Labels: affects-8.1, affects-8.5, epic/cardinality-estimation, report/customer, severity/major, sig/planner, type/bug, type/regression + +## Customer-Facing Phenomenon +- this is the plan of v8.5.3, use limit and use the primary key, which is also the order by key. you can see the result is empty, which means the IndexFullScan will scan the whole index, which is suboptimal + +## Linked PRs +- Fix PR #64139: planner: set an lower-bound for NDV used in out-of-range estimation for EQ conditions when the Histogram is empty | tidb-test=pr/2623 + URL: https://github.com/pingcap/tidb/pull/64139 + State: closed + Merged At: 2025-10-31T10:52:36Z + Changed Files Count: 6 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + PR Summary: What problem does this PR solve? Problem Summary: planner: consider empty histogram and newly emerging values in out-of-range estimation for EQ conditions more properly What changed and how does it work? See for more details. Hard to create a stable and graceful test case for this issue, so I tested it locally and here is the result: +- Related PR #64221: planner: set an lower-bound for NDV used in out-of-range estimation for EQ conditions when the Histogram is empty | tidb-test=pr/2623 (#64139) + URL: https://github.com/pingcap/tidb/pull/64221 + State: closed + Merged At: 2025-10-31T16:45:27Z + Changed Files Count: 4 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/BUILD.bazel + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + PR Summary: This is an automated cherry-pick of #64139 What problem does this PR solve? Problem Summary: planner: consider empty histogram and newly emerging values in out-of-range estimation for EQ conditions more properly What changed and how does it work? See for more details. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64329-xxxx-is-null-prevent-the-optimizer-from-choosing-the-best-join-order-and-indexjo.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64329-xxxx-is-null-prevent-the-optimizer-from-choosing-the-best-join-order-and-indexjo.md new file mode 100644 index 0000000..2ce9445 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64329-xxxx-is-null-prevent-the-optimizer-from-choosing-the-best-join-order-and-indexjo.md @@ -0,0 +1,37 @@ +# Issue #64329: "xxxx IS NULL" prevent the optimizer from choosing the best join order and IndexJoin + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64329 +- Status: closed +- Type: type/bug +- Created At: 2025-11-06T14:37:21Z +- Closed At: 2026-01-20T13:48:49Z +- Labels: plan-rewrite, planner/join-order, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- If there is no , the plan is: + +## Linked PRs +- Fix PR #64959: planner: support left outer join into anti semi join | tidb-test=pr/2649 + URL: https://github.com/pingcap/tidb/pull/64959 + State: closed + Merged At: 2026-01-20T13:48:47Z + Changed Files Count: 12 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/rule/BUILD.bazel + - pkg/planner/core/casetest/rule/main_test.go + - pkg/planner/core/casetest/rule/rule_outer_to_semi_join_test.go + - pkg/planner/core/casetest/rule/testdata/outer_to_semi_join_suite_in.json + - pkg/planner/core/casetest/rule/testdata/outer_to_semi_join_suite_out.json + - pkg/planner/core/casetest/rule/testdata/outer_to_semi_join_suite_xut.json + - pkg/planner/core/logical_plan_builder.go + - pkg/planner/core/optimizer.go + - pkg/planner/core/rule/BUILD.bazel + - pkg/planner/core/rule/logical_rules.go + - pkg/planner/core/rule/rule_outer_join_to_semi_join.go + - tests/integrationtest/r/topn_push_down.result + PR Summary: What problem does this PR solve? Problem Summary: planner: support left outer join into anti semi join What changed and how does it work? This PR introduces two optimization patterns for transforming LEFT JOIN queries into NOT EXISTS subqueries when checking for missing matches. Scenario 1: IS NULL on the Join Condition Column + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64345-recommend-index-statements-may-be-slow-and-can-t-be-killed.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64345-recommend-index-statements-may-be-slow-and-can-t-be-killed.md new file mode 100644 index 0000000..8548152 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64345-recommend-index-statements-may-be-slow-and-can-t-be-killed.md @@ -0,0 +1,17 @@ +# Issue #64345: `Recommend Index` statements may be slow and can't be killed + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64345 +- Status: open +- Type: type/enhancement +- Created At: 2025-11-07T08:39:42Z +- Labels: affects-8.5, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- What's worse, during this process, there is no checkpoint for the timeout option or signal, so the user is not able to it, and the timeout option is also ineffective. + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64375-planner-a-more-fine-grained-track-of-optimization-time.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64375-planner-a-more-fine-grained-track-of-optimization-time.md new file mode 100644 index 0000000..4a9be5f --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64375-planner-a-more-fine-grained-track-of-optimization-time.md @@ -0,0 +1,42 @@ +# Issue #64375: planner: a more fine-grained track of optimization time + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64375 +- Status: closed +- Type: type/enhancement +- Created At: 2025-11-10T03:14:36Z +- Closed At: 2025-12-29T06:13:12Z +- Labels: component/observability, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- We need a more fine-grained track of optimization time which should track all steps that could spend some time: 1. time used in sync_stats_load 2. time used in Plan Cache 3. time used in physical optimization 4. time used to get TiFlash logic core number: 5. derive stats for indexes or columns (if there are a large number of indexes / columns, this could take a long time like 100+ms) 6. ... + +## Linked PRs +- Fix PR #65096: planner: track more details about time spent on query optimization + URL: https://github.com/pingcap/tidb/pull/65096 + State: closed + Merged At: 2025-12-29T06:13:11Z + Changed Files Count: 8 + Main Modules: pkg/session, pkg/planner/core, pkg/bindinfo, pkg/executor, pkg/planner + Sample Changed Files: + - pkg/bindinfo/binding.go + - pkg/executor/adapter_test.go + - pkg/planner/core/optimizer.go + - pkg/planner/core/rule/rule_collect_plan_stats.go + - pkg/planner/optimize.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/session_test.go + - pkg/sessionctx/variable/slow_log.go + PR Summary: What problem does this PR solve? Problem Summary: planner: track more details about time spent on query optimization What changed and how does it work? Track more details about time spent on query optimization: Here is an example: +- Fix PR #65445: bindinfo: collect all the matchSQLBinding's time spent + URL: https://github.com/pingcap/tidb/pull/65445 + State: closed + Merged At: 2026-01-06T15:55:53Z + Changed Files Count: 1 + Main Modules: pkg/bindinfo + Sample Changed Files: + - pkg/bindinfo/binding.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? The core of MatchSQLBinding is matchSQLBinding, but matchSQLBinding itself is used in multiple places, so it's best to add the timing for MatchSQLBinding inside matchSQLBinding. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64423-analyze-a-small-table-costs-a-lot-of-memory.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64423-analyze-a-small-table-costs-a-lot-of-memory.md new file mode 100644 index 0000000..2cfce7b --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64423-analyze-a-small-table-costs-a-lot-of-memory.md @@ -0,0 +1,63 @@ +# Issue #64423: Analyze a small table costs a lot of memory + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64423 +- Status: closed +- Type: type/bug +- Created At: 2025-11-11T07:52:42Z +- Closed At: 2026-02-28T10:32:42Z +- Labels: affects-7.1, affects-7.5, affects-8.1, affects-8.5, affects-9.0, component/statistics, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- The table is small, it only has 80MB data. But the analyze statement costs more than 2GB memory. check the memory cost in the dashboard: + +## Linked PRs +- Fix PR #65446: executor: optimize analyze column memory usage + URL: https://github.com/pingcap/tidb/pull/65446 + State: closed + Merged At: 2026-01-30T06:16:17Z + Changed Files Count: 9 + Main Modules: pkg/statistics, pkg/executor + Sample Changed Files: + - pkg/executor/analyze_col.go + - pkg/executor/analyze_col_v2.go + - pkg/statistics/builder.go + - pkg/statistics/builder_test.go + - pkg/statistics/fmsketch_test.go + - pkg/statistics/main_test.go + - pkg/statistics/sample.go + - pkg/statistics/sample_test.go + - pkg/statistics/statistics_test.go + PR Summary: What problem does this PR solve? Problem Summary: As said, the memory reduced from 2.7G to 2.1G. It's not a big improvement. But for an order-of-magnitude improvement in memory footprint, we’d need to optmize the Datum structure, which involves significant effort and risk. What changed and how does it work? +- Fix PR #66166: executor: optimize analyze column memory usage (#65446) + URL: https://github.com/pingcap/tidb/pull/66166 + State: open + Merged At: not merged + Changed Files Count: 9 + Main Modules: pkg/statistics, pkg/executor + Sample Changed Files: + - pkg/executor/analyze_col.go + - pkg/executor/analyze_col_v2.go + - pkg/statistics/builder.go + - pkg/statistics/builder_test.go + - pkg/statistics/fmsketch_test.go + - pkg/statistics/main_test.go + - pkg/statistics/sample.go + - pkg/statistics/sample_test.go + - pkg/statistics/statistics_test.go + PR Summary: This is an automated cherry-pick of #65446 What problem does this PR solve? Problem Summary: As said, the memory reduced from 2.7G to 2.1G. It's not a big improvement. But for an order-of-magnitude improvement in memory footprint, we’d need to optmize the Datum structure, which involves significant effort and risk. What changed and how does it work? +- Fix PR #66747: executor, statistics: release analyze collector memory eagerly + URL: https://github.com/pingcap/tidb/pull/66747 + State: closed + Merged At: 2026-03-10T22:49:38Z + Changed Files Count: 4 + Main Modules: pkg/executor, pkg/statistics + Sample Changed Files: + - pkg/executor/analyze_col.go + - pkg/executor/analyze_col_v2.go + - pkg/executor/analyze_test.go + - pkg/statistics/sample.go + PR Summary: What problem does this PR solve? Problem Summary: After extended stats was removed, analyze no longer needs to retain column s until the end of the build flow. The obsolete retention path keeps sampled values alive longer than necessary and inflates analyze memory usage. What changed and how does it work? Release column collector memory immediately after histogram / TopN materialization in analyze v1/v2, and add a regression test for the v2 release timing. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64463-tidb-report-error-some-columns-of-limit-69-cannot-find-the-reference-from-its-ch.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64463-tidb-report-error-some-columns-of-limit-69-cannot-find-the-reference-from-its-ch.md new file mode 100644 index 0000000..2f0f002 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64463-tidb-report-error-some-columns-of-limit-69-cannot-find-the-reference-from-its-ch.md @@ -0,0 +1,17 @@ +# Issue #64463: tidb report error 'Some columns of Limit_69 cannot find the reference from its child(ren)' + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64463 +- Status: open +- Type: type/bug +- Created At: 2025-11-13T08:34:47Z +- Labels: affects-7.5, report/customer, sig/planner, type/bug + +## Customer-Facing Phenomenon +- (1105, 'Some columns of Limit_69 cannot find the reference from its child(ren)') + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64572-planner-unexpected-low-out-of-range-estimation-on-index-stats.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64572-planner-unexpected-low-out-of-range-estimation-on-index-stats.md new file mode 100644 index 0000000..3d2e30f --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64572-planner-unexpected-low-out-of-range-estimation-on-index-stats.md @@ -0,0 +1,50 @@ +# Issue #64572: planner: unexpected low out-of-range estimation on index stats + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64572 +- Status: closed +- Type: type/bug +- Created At: 2025-11-19T14:21:22Z +- Closed At: 2025-11-20T08:20:51Z +- Labels: affects-7.5, affects-8.1, affects-8.5, epic/cardinality-estimation, report/customer, severity/major, sig/planner, type/bug, type/regression + +## Customer-Facing Phenomenon +- Extremely low estimation. + +## Linked PRs +- Fix PR #64582: planner: out of range col stats only if loaded | tidb-test=pr/2631 + URL: https://github.com/pingcap/tidb/pull/64582 + State: closed + Merged At: 2025-11-20T08:20:50Z + Changed Files Count: 4 + Main Modules: pkg/planner/core, pkg/planner, tests/integrationtest + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - tests/integrationtest/r/imdbload.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #64591: planner: out of range col stats only if loaded | tidb-test=pr/2631 (#64582) + URL: https://github.com/pingcap/tidb/pull/64591 + State: closed + Merged At: not merged + Changed Files Count: 4 + Main Modules: pkg/planner/core, pkg/planner, tests/integrationtest + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - tests/integrationtest/r/imdbload.result + PR Summary: This is an automated cherry-pick of #64582 What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #64593: planner: out of range col stats only if loaded | tidb-test=pr/2631 (#64582) + URL: https://github.com/pingcap/tidb/pull/64593 + State: closed + Merged At: 2025-11-20T16:32:14Z + Changed Files Count: 1 + Main Modules: pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + PR Summary: This is an automated cherry-pick of #64582 What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64643-planner-read-from-storage-hint-table-alias-matching-problem-when-using-crossdb-b.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64643-planner-read-from-storage-hint-table-alias-matching-problem-when-using-crossdb-b.md new file mode 100644 index 0000000..f1889b0 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64643-planner-read-from-storage-hint-table-alias-matching-problem-when-using-crossdb-b.md @@ -0,0 +1,59 @@ +# Issue #64643: planner: `READ_FROM_STORAGE` hint table alias matching problem when using crossdb bindings + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64643 +- Status: closed +- Type: type/bug +- Created At: 2025-11-24T07:25:48Z +- Closed At: 2025-12-08T06:23:27Z +- Labels: affects-8.5, component/spm, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #64644: planner: fix the issue that `READ_FROM_STORAGE` hint doesn't consider cross-db binding + URL: https://github.com/pingcap/tidb/pull/64644 + State: closed + Merged At: 2025-11-24T12:45:26Z + Changed Files Count: 3 + Main Modules: pkg/bindinfo, pkg/util + Sample Changed Files: + - pkg/bindinfo/tests/BUILD.bazel + - pkg/bindinfo/tests/cross_db_binding_test.go + - pkg/util/hint/hint.go + PR Summary: What problem does this PR solve? Problem Summary: planner: fix the issue that hint doesn't consider cross-db binding What changed and how does it work? The database name might be "*" for cross-db bindings, we need to consider this case when matching tables for hint. +- Fix PR #64653: planner: fix the issue that `READ_FROM_STORAGE` hint doesn't consider cross-db binding (#64644) + URL: https://github.com/pingcap/tidb/pull/64653 + State: closed + Merged At: 2025-12-15T14:05:56Z + Changed Files Count: 2 + Main Modules: pkg/bindinfo, pkg/util + Sample Changed Files: + - pkg/bindinfo/fuzzy_binding_test.go + - pkg/util/hint/hint.go + PR Summary: This is an automated cherry-pick of #64644 What problem does this PR solve? Problem Summary: planner: fix the issue that hint doesn't consider cross-db binding What changed and how does it work? The database name might be "*" for cross-db bindings, we need to consider this case when matching tables for hint. +- Related PR #64689: planner: pick #64622 #63519 #63962 #64644 #64501 | tidb-test=pr/2634 + URL: https://github.com/pingcap/tidb/pull/64689 + State: closed + Merged At: 2025-11-27T05:26:14Z + Changed Files Count: 13 + Main Modules: pkg/statistics, pkg/planner/core, pkg/bindinfo, pkg/domain, pkg/util + Sample Changed Files: + - pkg/bindinfo/fuzzy_binding_test.go + - pkg/domain/domain.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/index_join_path.go + - pkg/planner/core/integration_test.go + - pkg/statistics/handle/bootstrap.go + - pkg/statistics/handle/storage/read.go + - pkg/statistics/handle/syncload/stats_syncload.go + - pkg/statistics/handle/usage/BUILD.bazel + - pkg/statistics/handle/usage/session_stats_collect.go + - pkg/statistics/handle/usage/session_stats_collect_test.go + - pkg/statistics/table.go + - pkg/util/hint/hint.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64648-planner-optimizer-uses-total-table-size-for-partition-table-scan-cost-estimation.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64648-planner-optimizer-uses-total-table-size-for-partition-table-scan-cost-estimation.md new file mode 100644 index 0000000..072560f --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64648-planner-optimizer-uses-total-table-size-for-partition-table-scan-cost-estimation.md @@ -0,0 +1,17 @@ +# Issue #64648: planner: Optimizer uses total table size for partition table scan cost estimation even when query can be pruned + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64648 +- Status: open +- Type: type/enhancement +- Created At: 2025-11-24T10:44:26Z +- Labels: epic/cardinality-estimation, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- I encountered an issue regarding the cost estimation logic for partitioned tables. When querying a partitioned table with a specific partition key, the estCost for the IndexFullScan operator is calculated based on the global table statistics (Total Rows) rather than the statistics of the pruned partition(s), resulting in an extremely inflated cost. + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64649-planner-index-join-cardinality-estimation-issue-because-of-filter-re-estimates-c.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64649-planner-index-join-cardinality-estimation-issue-because-of-filter-re-estimates-c.md new file mode 100644 index 0000000..d049435 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64649-planner-index-join-cardinality-estimation-issue-because-of-filter-re-estimates-c.md @@ -0,0 +1,17 @@ +# Issue #64649: planner: Index Join cardinality estimation Issue because of filter re-estimates conditions already covered by range + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64649 +- Status: open +- Type: type/enhancement +- Created At: 2025-11-24T10:53:36Z +- Labels: epic/cardinality-estimation, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- **Table schema (simplified):** + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64707-planner-extract-common-leading-predicates-in-dnf-and-push-them-to-indexscan-to-f.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64707-planner-extract-common-leading-predicates-in-dnf-and-push-them-to-indexscan-to-f.md new file mode 100644 index 0000000..8034b21 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64707-planner-extract-common-leading-predicates-in-dnf-and-push-them-to-indexscan-to-f.md @@ -0,0 +1,17 @@ +# Issue #64707: planner: extract common leading predicates in DNF and push them to IndexScan to filter out unnecessary data earlier + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64707 +- Status: open +- Type: type/enhancement +- Created At: 2025-11-27T06:51:48Z +- Labels: plan-rewrite, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- Actually, we could extract some common leading predicates from that DNF, for example and then push it into to filter our some unnecessary data earlier: + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64712-build-better-range-with-not-expression.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64712-build-better-range-with-not-expression.md new file mode 100644 index 0000000..af39349 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64712-build-better-range-with-not-expression.md @@ -0,0 +1,17 @@ +# Issue #64712: build better range with NOT expression + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64712 +- Status: open +- Type: type/enhancement +- Created At: 2025-11-27T08:30:40Z +- Labels: report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- There is the following case: + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64809-planner-out-of-range-estimation-enhancement.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64809-planner-out-of-range-estimation-enhancement.md new file mode 100644 index 0000000..8c40362 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-64809-planner-out-of-range-estimation-enhancement.md @@ -0,0 +1,17 @@ +# Issue #64809: planner: out-of-range estimation enhancement + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/64809 +- Status: open +- Type: type/enhancement +- Created At: 2025-12-02T06:45:34Z +- Labels: epic/cardinality-estimation, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- - out-of-range estimation for EQ/IN (point-query) - - - - - out-of-range estimation for CMP (range-query) - - - - + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65032-planner-set-the-default-value-of-tidb-opt-ordering-index-selectivity-threshold-t.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65032-planner-set-the-default-value-of-tidb-opt-ordering-index-selectivity-threshold-t.md new file mode 100644 index 0000000..60d4954 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65032-planner-set-the-default-value-of-tidb-opt-ordering-index-selectivity-threshold-t.md @@ -0,0 +1,18 @@ +# Issue #65032: planner: set the default value of `tidb_opt_ordering_index_selectivity_threshold` to 0.01 + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65032 +- Status: closed +- Type: type/enhancement +- Created At: 2025-12-15T03:30:59Z +- Closed At: 2025-12-15T06:09:00Z +- Labels: duplicate, epic/cardinality-estimation, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- The ordering-index is more risky because its estimation formula contains more assumptions and uncertainty; in contrast, the formula of filtering-index is more stable and natural. + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- The issue is closed, but no merged PR was resolved automatically from the timeline. It may have been fixed by an internal branch, a batch PR, or manual closure. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65059-tiflash-plan-returns-no-access-path-error-in-rc-isolation-level.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65059-tiflash-plan-returns-no-access-path-error-in-rc-isolation-level.md new file mode 100644 index 0000000..c8c0ee8 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65059-tiflash-plan-returns-no-access-path-error-in-rc-isolation-level.md @@ -0,0 +1,55 @@ +# Issue #65059: TiFlash plan returns `No access path` error in RC isolation level + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65059 +- Status: closed +- Type: type/question +- Created At: 2025-12-16T03:37:35Z +- Closed At: 2025-12-29T07:23:42Z +- Labels: affects-8.5, component/tiflash, report/customer, sig/planner, type/question + +## Customer-Facing Phenomenon +- ERROR 1815 (HY000): Internal : No access path for table 't1' is found with 'tidb_isolation_read_engines' = 'tidb,tiflash', valid values can be 'tikv'. + +## Linked PRs +- Fix PR #65127: planner: fix no access path when TiKV read is disabled under RC isolation + URL: https://github.com/pingcap/tidb/pull/65127 + State: closed + Merged At: 2025-12-29T07:23:41Z + Changed Files Count: 11 + Main Modules: pkg/planner/core, pkg/planner, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/enforcempp/BUILD.bazel + - pkg/planner/core/casetest/enforcempp/enforce_mpp_test.go + - pkg/planner/core/casetest/tpch/BUILD.bazel + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_in.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_out.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_xut.json + - pkg/planner/core/casetest/tpch/tpch_test.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/integration_test.go + - pkg/planner/util/misc.go + - tests/integrationtest/r/planner/core/integration.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Because we use RC isolation, so is true. Here will not remove the tikv path. +- Fix PR #66951: planner: fix no access path when TiKV read is disabled under RC isolation (#65127) + URL: https://github.com/pingcap/tidb/pull/66951 + State: open + Merged At: not merged + Changed Files Count: 11 + Main Modules: pkg/planner/core, pkg/planner, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/enforcempp/BUILD.bazel + - pkg/planner/core/casetest/enforcempp/enforce_mpp_test.go + - pkg/planner/core/casetest/tpch/BUILD.bazel + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_in.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_out.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_xut.json + - pkg/planner/core/casetest/tpch/tpch_test.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/integration_test.go + - pkg/planner/util/misc.go + - tests/integrationtest/r/planner/core/integration.result + PR Summary: This is an automated cherry-pick of #65127 What problem does this PR solve? Problem Summary: What changed and how does it work? Because we use RC isolation, so is true. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65100-planner-unnecessary-overhead-caused-by-the-optimizer-matching-bindings-twice-whe.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65100-planner-unnecessary-overhead-caused-by-the-optimizer-matching-bindings-twice-whe.md new file mode 100644 index 0000000..764bc35 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65100-planner-unnecessary-overhead-caused-by-the-optimizer-matching-bindings-twice-whe.md @@ -0,0 +1,60 @@ +# Issue #65100: planner: unnecessary overhead caused by the optimizer matching bindings twice when plan cache miss + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65100 +- Status: closed +- Type: type/enhancement +- Created At: 2025-12-17T12:46:23Z +- Closed At: 2026-01-13T19:55:09Z +- Labels: affects-8.5, epic/plan-cache, report/customer, sig/planner, type/enhancement, type/performance + +## Customer-Facing Phenomenon +- See the picture below, the optimizer calls twice when getting the cached plan, which could bring us some unnecessary overhead: + +## Linked PRs +- Fix PR #65104: planner: plan cache performance optimization | tidb-test=86a1cff755f507d127f0bca7ff18e2f3b9537124 + URL: https://github.com/pingcap/tidb/pull/65104 + State: closed + Merged At: 2025-12-18T01:46:34Z + Changed Files Count: 7 + Main Modules: pkg/session, pkg/planner/core, pkg/bindinfo, pkg/executor + Sample Changed Files: + - pkg/bindinfo/binding_match.go + - pkg/executor/select.go + - pkg/planner/core/plan_cache_test.go + - pkg/planner/core/plan_cache_utils.go + - pkg/sessionctx/stmtctx/stmtctx.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? +- Fix PR #65484: bindinfo: save bindinfo into stmtctx to avoid calling NormalizeStmtForBinding and getbindcache repeatedly + URL: https://github.com/pingcap/tidb/pull/65484 + State: closed + Merged At: 2026-01-13T19:55:08Z + Changed Files Count: 11 + Main Modules: pkg/bindinfo, pkg/planner/core, .bazelrc, pkg/executor, pkg/session + Sample Changed Files: + - .bazelrc + - pkg/bindinfo/binding.go + - pkg/bindinfo/binding_cache_test.go + - pkg/bindinfo/session_handle_test.go + - pkg/bindinfo/tests/cross_db_binding_test.go + - pkg/executor/select.go + - pkg/planner/core/casetest/plancache/BUILD.bazel + - pkg/planner/core/casetest/plancache/plan_cache_test.go + - pkg/planner/core/integration_partition_test.go + - pkg/planner/core/plan_cache_utils.go + - pkg/sessionctx/stmtctx/stmtctx.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? save bindinfo into stmtctx to avoid calling NormalizeStmtForBinding and getbindcache repeatedly. This PR is an improvement for +- Related PR #65524: planner: add some plan cache benchmark into bench daily + URL: https://github.com/pingcap/tidb/pull/65524 + State: closed + Merged At: 2026-01-12T05:58:56Z + Changed Files Count: 1 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/integration_partition_test.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? planner: add some plan cache benchmark into bench daily + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65165-planner-outer-join-should-be-eliminated-when-join-keys-do-not-use-nulleq-and-can.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65165-planner-outer-join-should-be-eliminated-when-join-keys-do-not-use-nulleq-and-can.md new file mode 100644 index 0000000..4221402 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65165-planner-outer-join-should-be-eliminated-when-join-keys-do-not-use-nulleq-and-can.md @@ -0,0 +1,28 @@ +# Issue #65165: planner: outer join should be eliminated when join keys do not use NULLEQ and can hit a nullable unique index + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65165 +- Status: closed +- Type: type/enhancement +- Created At: 2025-12-22T09:27:30Z +- Closed At: 2026-02-06T04:22:22Z +- Labels: plan-rewrite, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- We should eliminate the Join for both queries above. Although could be , it doesn't affect the final result. + +## Linked PRs +- Fix PR #65843: planner: eliminate outer join on nullable unique keys under EQ + URL: https://github.com/pingcap/tidb/pull/65843 + State: closed + Merged At: 2026-02-06T04:22:21Z + Changed Files Count: 3 + Main Modules: pkg/planner/core, pkg/planner + Sample Changed Files: + - pkg/planner/cascades/old/transformation_rules.go + - pkg/planner/core/casetest/plan_test.go + - pkg/planner/core/rule_join_elimination.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Outer join elimination relies on the inner side join keys guaranteeing at most one match. Previously we only considered Schema.PKOrUK and unique secondary indexes, which effectively excluded nullable unique keys (Schema.NullableUK). This missed the safe case where join keys use normal equality '=': when the inner unique key is nullable, rows with NULL do not match under '=' so the join still cannot introduce extra matches and the outer join can be removed when the parent doesn't reference inner columns. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65166-planner-outer-join-cannot-be-eliminated-due-to-the-impact-of-the-expression-inde.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65166-planner-outer-join-cannot-be-eliminated-due-to-the-impact-of-the-expression-inde.md new file mode 100644 index 0000000..cc99c09 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65166-planner-outer-join-cannot-be-eliminated-due-to-the-impact-of-the-expression-inde.md @@ -0,0 +1,115 @@ +# Issue #65166: planner: outer join cannot be eliminated due to the impact of the expression index + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65166 +- Status: closed +- Type: type/bug +- Created At: 2025-12-22T10:05:56Z +- Closed At: 2026-01-05T18:27:15Z +- Labels: affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.5, plan-rewrite, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- See the example below: See the case above, the outer join is not eliminated at first place, but after putting the force index hint on , we can eliminate it. The problem is that the expression index affects the when eliminating outer join and causes us to stop eliminating early (see the picture below): If is a normal index instead of an expression index, the outer join elimination can also work: + +## Linked PRs +- Fix PR #65187: planner: outer join cannot be eliminated due to the impact of the expression index | tidb-test=pr/2657 + URL: https://github.com/pingcap/tidb/pull/65187 + State: closed + Merged At: 2026-01-05T18:27:14Z + Changed Files Count: 19 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/binaryplan/testdata/binary_plan_suite_out.json + - pkg/planner/core/casetest/binaryplan/testdata/binary_plan_suite_xut.json + - pkg/planner/core/casetest/cascades/testdata/cascades_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/mpp/testdata/integration_suite_out.json + - pkg/planner/core/casetest/mpp/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_out.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_xut.json + - pkg/planner/core/integration_test.go + - pkg/planner/core/rule_eliminate_projection.go + - pkg/planner/core/testdata/plan_suite_unexported_out.json + - tests/integrationtest/r/planner/cascades/integration.result + - tests/integrationtest/r/planner/core/casetest/rule/rule_result_reorder.result + - tests/integrationtest/r/planner/core/indexjoin.result + - tests/integrationtest/r/planner/core/plan_cache.result + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + PR Summary: What problem does this PR solve? Problem Summary: [65166](planner: outer join cannot be eliminated due to the impact of the expression index) What changed and how does it work? Please see for more details. +- Fix PR #65425: planner: outer join cannot be eliminated due to the impact of the expression index | tidb-test=pr/2657 (#65187) + URL: https://github.com/pingcap/tidb/pull/65425 + State: closed + Merged At: not merged + Changed Files Count: 19 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/binaryplan/testdata/binary_plan_suite_out.json + - pkg/planner/core/casetest/binaryplan/testdata/binary_plan_suite_xut.json + - pkg/planner/core/casetest/cascades/testdata/cascades_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/mpp/testdata/integration_suite_out.json + - pkg/planner/core/casetest/mpp/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_xut.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_out.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_xut.json + - pkg/planner/core/integration_test.go + - pkg/planner/core/rule_eliminate_projection.go + - pkg/planner/core/testdata/plan_suite_unexported_out.json + - tests/integrationtest/r/planner/cascades/integration.result + - tests/integrationtest/r/planner/core/casetest/rule/rule_result_reorder.result + - tests/integrationtest/r/planner/core/indexjoin.result + - tests/integrationtest/r/planner/core/plan_cache.result + - tests/integrationtest/r/planner/core/rule_constant_propagation.result + PR Summary: This is an automated cherry-pick of #65187 What problem does this PR solve? Problem Summary: [65166](planner: outer join cannot be eliminated due to the impact of the expression index) What changed and how does it work? Please see for more details. +- Fix PR #65449: planner: outer join cannot be eliminated due to the impact of the expression index + URL: https://github.com/pingcap/tidb/pull/65449 + State: closed + Merged At: not merged + Changed Files Count: 1825 + Main Modules: br/pkg, pkg/executor, pkg/planner/core, pkg/ddl, tests/integrationtest, pkg/statistics + Sample Changed Files: + - .bazelrc + - .bazelversion + - DEPS.bzl + - Makefile + - Makefile.common + - OWNERS + - OWNERS_ALIASES + - WORKSPACE + - br/cmd/br/BUILD.bazel + - br/cmd/br/abort.go + - br/cmd/br/backup.go + - br/cmd/br/cmd.go + - br/cmd/br/debug.go + - br/cmd/br/main.go + - br/cmd/br/operator.go + - br/cmd/br/restore.go + - br/metrics/grafana/br.json + - br/pkg/backup/BUILD.bazel + - br/pkg/backup/check.go + - br/pkg/backup/client.go + PR Summary: This is an automated cherry-pick of #65187 What problem does this PR solve? Problem Summary: [65166](planner: outer join cannot be eliminated due to the impact of the expression index) What changed and how does it work? Please see for more details. +- Fix PR #65450: planner: outer join cannot be eliminated due to the impact of the expression index + URL: https://github.com/pingcap/tidb/pull/65450 + State: closed + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/binaryplan/testdata/binary_plan_suite_out.json + - pkg/planner/core/casetest/mpp/testdata/integration_suite_out.json + - pkg/planner/core/casetest/testdata/integration_suite_out.json + - pkg/planner/core/integration_test.go + - pkg/planner/core/rule_eliminate_projection.go + - pkg/planner/core/testdata/plan_suite_unexported_out.json + - tests/integrationtest/r/explain_complex.result + - tests/integrationtest/r/index_merge.result + PR Summary: This is an automated cherry-pick of #65187 What problem does this PR solve? Problem Summary: [65166](planner: outer join cannot be eliminated due to the impact of the expression index) What changed and how does it work? Please see for more details. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65183-planner-row-count-incorrectly-forced-to-1-for-indexjoin-when-matching-only-a-pre.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65183-planner-row-count-incorrectly-forced-to-1-for-indexjoin-when-matching-only-a-pre.md new file mode 100644 index 0000000..87bc962 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65183-planner-row-count-incorrectly-forced-to-1-for-indexjoin-when-matching-only-a-pre.md @@ -0,0 +1,49 @@ +# Issue #65183: planner: Row count incorrectly forced to 1 for IndexJoin when matching only a prefix of a composite Primary Key + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65183 +- Status: closed +- Type: type/bug +- Created At: 2025-12-23T07:22:57Z +- Closed At: 2026-01-16T03:17:03Z +- Labels: report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- **Root Cause Analysis**: The investigation reveals that the optimizer incorrectly handles the row count estimation for the inner (probe) side of an IndexJoin when a Primary Key is involved. The issue is located in pkg/planner/core/exhaust_physical_plans.go within the function constructDS2TableScanTask. The code explicitly forces the row count using: RowCount: math.Min(1.0, countAfterAccess), This logic assumes that any access utilizing Primary Key columns will return at most one row. However, it fails to account for cases where the join condition only covers a prefix of a composite Primary Key. In such scenarios, the operation is actually a range scan that could involve a large number of rows. By forcing the RowCount to 1, the optimizer severely underestimates the scan cost, leading to the selection of an inefficient IndexJoin instead of a more suitable HashJoin. + +## Linked PRs +- Related PR #65190: planner: fix the row count for index join use the PK in inner side + URL: https://github.com/pingcap/tidb/pull/65190 + State: closed + Merged At: 2026-01-16T03:17:02Z + Changed Files Count: 8 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_out.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_xut.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_out.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_xut.json + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/index_join_path.go + - tests/integrationtest/r/planner/core/indexjoin.result + - tests/integrationtest/t/planner/core/indexjoin.test + PR Summary: What problem does this PR solve? Problem Summary: fix the row count for index join use the PK in inner side What changed and how does it work? Use the same way in the function() which use to handle the unique index. It is very similar here. Check whether we use the full columns in the PK +- Related PR #65745: planner: fix the row count for index join use the PK in inner side (#65190) + URL: https://github.com/pingcap/tidb/pull/65745 + State: open + Merged At: not merged + Changed Files Count: 8 + Main Modules: pkg/planner/core, tests/integrationtest + Sample Changed Files: + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_out.json + - pkg/planner/core/casetest/rule/testdata/predicate_simplification_xut.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_out.json + - pkg/planner/core/casetest/tpch/testdata/tpch_suite_xut.json + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/index_join_path.go + - tests/integrationtest/r/planner/core/indexjoin.result + - tests/integrationtest/t/planner/core/indexjoin.test + PR Summary: This is an automated cherry-pick of #65190 What problem does this PR solve? Problem Summary: fix the row count for index join use the PK in inner side What changed and how does it work? + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65208-support-using-ordering-property-to-get-better-join-order.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65208-support-using-ordering-property-to-get-better-join-order.md new file mode 100644 index 0000000..1ae5223 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65208-support-using-ordering-property-to-get-better-join-order.md @@ -0,0 +1,35 @@ +# Issue #65208: support using ordering property to get better join order + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65208 +- Status: open +- Type: type/enhancement +- Created At: 2025-12-24T02:53:56Z +- Labels: report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- The better plan: choose join order of and use , so we can leverage the ordering property and only need to get 1000 rows from Join output. + +## Linked PRs +- Fix PR #63522: planner: Allow leading ordered table to survive (WIP) + URL: https://github.com/pingcap/tidb/pull/63522 + State: open + Merged At: not merged + Changed Files Count: 11 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/operator/physicalop/physical_index_join.go + - pkg/planner/core/operator/physicalop/physical_topn.go + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/rule_join_reorder.go + - pkg/planner/core/rule_join_reorder_greedy.go + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/session.go + - pkg/sessionctx/variable/sysvar.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65381-planner-the-optimizer-can-t-use-the-latest-tidb-mem-quota-binding-cache-value-to.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65381-planner-the-optimizer-can-t-use-the-latest-tidb-mem-quota-binding-cache-value-to.md new file mode 100644 index 0000000..2c8190a --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65381-planner-the-optimizer-can-t-use-the-latest-tidb-mem-quota-binding-cache-value-to.md @@ -0,0 +1,44 @@ +# Issue #65381: planner: the optimizer can't use the latest `tidb_mem_quota_binding_cache` value to initialize binding cache + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65381 +- Status: closed +- Type: type/bug +- Created At: 2026-01-02T03:34:13Z +- Closed At: 2026-01-04T07:26:42Z +- Labels: affects-7.5, affects-8.5, component/spm, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- Step 1: create massive bindings, and their total memory usage is larger than the default binding cache size 64MB Step 2: restart TiDB and we'll see errors like Step 3: increase the binding cache size through Step 4: restart TiDB again, there should be no binding cache errors. + +## Linked PRs +- Fix PR #65382: planner: fix the issue that the optimizer can't use the latest system value to initialize binding cache | tidb-test=86a1cff755f507d127f0bca7ff18e2f3b9537124 + URL: https://github.com/pingcap/tidb/pull/65382 + State: closed + Merged At: 2026-01-02T05:12:47Z + Changed Files Count: 1 + Main Modules: pkg/session + Sample Changed Files: + - pkg/session/session.go + PR Summary: What problem does this PR solve? Problem Summary: planner: fix the issue that the optimizer can't use the latest tidb_mem_quota_binding_cache value to initialize binding cache What changed and how does it work? Just move the initialization of binding cache after the initialization of system variables. This case is hard to add unit tests since it's about Bootstrap. +- Fix PR #65392: planner: fix the issue that the optimizer can't use the latest system value to initialize binding cache + URL: https://github.com/pingcap/tidb/pull/65392 + State: closed + Merged At: 2026-01-04T07:26:41Z + Changed Files Count: 1 + Main Modules: pkg/session + Sample Changed Files: + - pkg/session/session.go + PR Summary: What problem does this PR solve? Problem Summary: planner: fix the issue that the optimizer can't use the latest system value to initialize binding cache What changed and how does it work? We have to initialize Binding Handle after initializing Sysvars, since the optimizer need to use the SQL variable to set up the binding cache size. In our prior implementation, the optimizer can't get the latest value of this variable when initializing binding cache. This is hard to add unit tests since it's related to TiDB bootstrap. But I tested it locally and this PR can work. +- Fix PR #65397: planner: fix the issue that the optimizer can't use the latest system value to initialize binding cache (#65392) + URL: https://github.com/pingcap/tidb/pull/65397 + State: closed + Merged At: 2026-01-06T15:58:32Z + Changed Files Count: 1 + Main Modules: pkg/session + Sample Changed Files: + - pkg/session/session.go + PR Summary: This is an automated cherry-pick of #65392 What problem does this PR solve? Problem Summary: planner: fix the issue that the optimizer can't use the latest system value to initialize binding cache What changed and how does it work? We have to initialize Binding Handle after initializing Sysvars, since the optimizer need to use the SQL variable to set up the binding cache size. In our prior implementation, the optimizer can't get the latest value of this variable when initializing binding cache. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65489-memory-leak-when-analyze-table-is-the-last-statement-in-a-session.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65489-memory-leak-when-analyze-table-is-the-last-statement-in-a-session.md new file mode 100644 index 0000000..dcc3852 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65489-memory-leak-when-analyze-table-is-the-last-statement-in-a-session.md @@ -0,0 +1,50 @@ +# Issue #65489: Memory leak when `analyze table` is the last statement in a session + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65489 +- Status: closed +- Type: type/bug +- Created At: 2026-01-08T12:17:48Z +- Closed At: 2026-01-09T22:24:42Z +- Labels: affects-7.5, affects-8.1, affects-8.5, component/statistics, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- About 1GiB memory in . + +## Linked PRs +- Fix PR #65492: session: fix memory leak when sessions close after ANALYZE statements + URL: https://github.com/pingcap/tidb/pull/65492 + State: closed + Merged At: 2026-01-09T22:24:41Z + Changed Files Count: 3 + Main Modules: pkg/executor, pkg/session + Sample Changed Files: + - pkg/executor/test/analyzetest/memorycontrol/BUILD.bazel + - pkg/executor/test/analyzetest/memorycontrol/memory_control_test.go + - pkg/session/session.go + PR Summary: What problem does this PR solve? Problem Summary: Short-lived sessions executing ANALYZE statements leak ~1GiB memory in after 20k iterations. When ANALYZE statements are executed, the session's is attached to . For sessions that close immediately after ANALYZE (common in connection pooling), the tracker remains attached, preventing garbage collection. What changed and how does it work? ANALYZE has a special memory tracking path where is called. keeps strong references to its children. When a session closes immediately after ANALYZE, the session-level is not detached, so the global tracker retains the session, keeping alive. **Changes:** +- Fix PR #65516: session: fix memory leak when sessions close after ANALYZE statements (#65492) + URL: https://github.com/pingcap/tidb/pull/65516 + State: closed + Merged At: 2026-01-13T01:50:14Z + Changed Files Count: 3 + Main Modules: pkg/executor, pkg/session + Sample Changed Files: + - pkg/executor/test/analyzetest/memorycontrol/BUILD.bazel + - pkg/executor/test/analyzetest/memorycontrol/memory_control_test.go + - pkg/session/session.go + PR Summary: This is an automated cherry-pick of #65492 What problem does this PR solve? Problem Summary: Short-lived sessions executing ANALYZE statements leak ~1GiB memory in after 20k iterations. When ANALYZE statements are executed, the session's is attached to . For sessions that close immediately after ANALYZE (common in connection pooling), the tracker remains attached, preventing garbage collection. What changed and how does it work? ANALYZE has a special memory tracking path where is called. keeps strong references to its children. When a session closes immediately after ANALYZE, the session-level is not detached, so the global tracker retains the session, keeping alive. +- Fix PR #65539: session: fix memory leak when sessions close after ANALYZE statements (#65492) | tidb-test=86a1cff755f507d127f0bca7ff18e2f3b9537124 + URL: https://github.com/pingcap/tidb/pull/65539 + State: closed + Merged At: 2026-01-13T05:25:26Z + Changed Files Count: 3 + Main Modules: pkg/executor, pkg/session + Sample Changed Files: + - pkg/executor/test/analyzetest/memorycontrol/BUILD.bazel + - pkg/executor/test/analyzetest/memorycontrol/memory_control_test.go + - pkg/session/session.go + PR Summary: This is an automated cherry-pick of #65492 What problem does this PR solve? Problem Summary: Short-lived sessions executing ANALYZE statements leak ~1GiB memory in after 20k iterations. When ANALYZE statements are executed, the session's is attached to . For sessions that close immediately after ANALYZE (common in connection pooling), the tracker remains attached, preventing garbage collection. What changed and how does it work? ANALYZE has a special memory tracking path where is called. keeps strong references to its children. When a session closes immediately after ANALYZE, the session-level is not detached, so the global tracker retains the session, keeping alive. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65556-cost-model-v2-indexhashjoin-cost-underestimation.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65556-cost-model-v2-indexhashjoin-cost-underestimation.md new file mode 100644 index 0000000..fabae66 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65556-cost-model-v2-indexhashjoin-cost-underestimation.md @@ -0,0 +1,37 @@ +# Issue #65556: Cost Model v2: `IndexHashJoin` cost underestimation + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65556 +- Status: open +- Type: type/bug +- Created At: 2026-01-13T09:20:03Z +- Labels: epic/cost-model, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- IndexHashJoin cost formula misses hashprobe(...) and shows hashkey(...*0*...) in . + +## Linked PRs +- Related PR #65557: planner: fix indexHashJoin cost formula + URL: https://github.com/pingcap/tidb/pull/65557 + State: open + Merged At: not merged + Changed Files Count: 13 + Main Modules: tests/integrationtest, pkg/bindinfo, pkg/planner/core + Sample Changed Files: + - pkg/bindinfo/binding_auto_test.go + - pkg/bindinfo/testdata/binding_auto_suite_out.json + - pkg/planner/core/plan_cost_ver2.go + - pkg/planner/core/plan_cost_ver2_test.go + - tests/integrationtest/r/executor/explain.result + - tests/integrationtest/r/globalindex/index_join.result + - tests/integrationtest/r/planner/core/casetest/integration.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/casetest/rule/rule_join_reorder.result + - tests/integrationtest/r/planner/core/physical_plan.result + - tests/integrationtest/r/planner/core/plan_cost_ver2.result + - tests/integrationtest/r/planner/core/rule_join_reorder.result + - tests/integrationtest/r/tpch.result + PR Summary: What problem does this PR solve? Problem Summary: Cost Model v2 underestimates : the cost formula missed probe-side hash cost and could use incorrect join key count, leading to wrong plan preference. What changed and how does it work? Fix cost model v2 for index-join family in : + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65639-sql-binding-fails-silently-when-sql-is-too-long-binding-created-but-not-visible-.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65639-sql-binding-fails-silently-when-sql-is-too-long-binding-created-but-not-visible-.md new file mode 100644 index 0000000..96037af --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65639-sql-binding-fails-silently-when-sql-is-too-long-binding-created-but-not-visible-.md @@ -0,0 +1,17 @@ +# Issue #65639: SQL Binding fails silently when SQL is too long - binding created but not visible in SHOW GLOBAL BINDINGS + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65639 +- Status: open +- Type: type/bug +- Created At: 2026-01-19T09:00:50Z +- Labels: contribution, may-affects-8.5, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- Actual behavior observed: 1. **Silent failure**: executed **without any error**, appearing to succeed 2. **Data inconsistency**: **does NOT show** the binding + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65712-better-plan-for-limit-n-order-by-on-indexmerge-when-not-all-partial-paths-satisf.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65712-better-plan-for-limit-n-order-by-on-indexmerge-when-not-all-partial-paths-satisf.md new file mode 100644 index 0000000..669eed8 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65712-better-plan-for-limit-n-order-by-on-indexmerge-when-not-all-partial-paths-satisf.md @@ -0,0 +1,34 @@ +# Issue #65712: Better plan for `LIMIT n ORDER BY` on `IndexMerge` when not all partial paths satisfy the `ORDER BY` + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65712 +- Status: open +- Type: type/enhancement +- Created At: 2026-01-22T01:28:23Z +- Labels: affects-7.5, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- This is a simplified case from a customer scenario. + +## Linked PRs +- Fix PR #66097: planner, executor: support merge sort for IN conditions in IndexMerge… + URL: https://github.com/pingcap/tidb/pull/66097 + State: open + Merged At: not merged + Changed Files Count: 10 + Main Modules: tests/integrationtest, pkg/executor, pkg/planner/core + Sample Changed Files: + - pkg/executor/index_merge_reader.go + - pkg/executor/test/indexmergereadtest/BUILD.bazel + - pkg/executor/test/indexmergereadtest/index_merge_reader_test.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/operator/physicalop/physical_index_scan.go + - pkg/planner/core/operator/physicalop/physical_table_scan.go + - tests/integrationtest/r/index_merge.result + - tests/integrationtest/r/planner/core/casetest/physicalplantest/physical_plan.result + - tests/integrationtest/r/planner/core/grouped_ranges_order_by.result + - tests/integrationtest/t/index_merge.test + PR Summary: What problem does this PR solve? Problem Summary: For queries like : The partial path can keep order → returns The partial path needs merge sort → returns + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65818-analyze-cannot-be-cancelled-promptly.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65818-analyze-cannot-be-cancelled-promptly.md new file mode 100644 index 0000000..058b977 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65818-analyze-cannot-be-cancelled-promptly.md @@ -0,0 +1,36 @@ +# Issue #65818: Analyze cannot be cancelled promptly + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65818 +- Status: open +- Type: type/bug +- Created At: 2026-01-26T11:27:23Z +- Labels: affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- The analyze worker blocks on RPC/NextRaw; the job does not exit promptly after kill, and the analyze execution can hang. + +## Linked PRs +- Fix PR #65249: executor: fix analyze cannot be killed + URL: https://github.com/pingcap/tidb/pull/65249 + State: open + Merged At: not merged + Changed Files Count: 12 + Main Modules: pkg/executor, pkg/distsql, pkg/statistics + Sample Changed Files: + - pkg/distsql/distsql.go + - pkg/executor/analyze.go + - pkg/executor/analyze_col.go + - pkg/executor/analyze_col_v2.go + - pkg/executor/analyze_idx.go + - pkg/executor/analyze_test.go + - pkg/executor/analyze_utils.go + - pkg/executor/table_reader.go + - pkg/executor/test/analyzetest/BUILD.bazel + - pkg/executor/test/analyzetest/analyze_test.go + - pkg/executor/test/analyzetest/memorycontrol/memory_control_test.go + - pkg/statistics/handle/autoanalyze/autoanalyze.go + PR Summary: What problem does this PR solve? Problem Summary: After #63067 removed the coprocessor-side periodic kill polling fallback in , cancellation for and RPC paths effectively relies on the caller-provided context being canceled promptly. Analyze still had several V1 paths that entered or RPC with or otherwise inconsistent contexts. As a result, a kill signal could be observed before entering the RPC, but once a worker was blocked in or RPC, some paths could not exit proactively. What changed and how does it work? + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65822-support-building-indexmerge-path-and-pushing-down-limit-for-in-expression-in-nes.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65822-support-building-indexmerge-path-and-pushing-down-limit-for-in-expression-in-nes.md new file mode 100644 index 0000000..20c8bbe --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65822-support-building-indexmerge-path-and-pushing-down-limit-for-in-expression-in-nes.md @@ -0,0 +1,17 @@ +# Issue #65822: Support building IndexMerge path and pushing down `Limit` for `IN` expression in nested OR list like `a = 1 and (b in (1,3) or c = 2)` + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65822 +- Status: open +- Type: type/enhancement +- Created At: 2026-01-26T12:15:56Z +- Labels: affects-7.5, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- What we already have: + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65838-inl-join-hint-shows-warning-when-tidb-opt-advanced-join-hint-off-but-silently-fa.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65838-inl-join-hint-shows-warning-when-tidb-opt-advanced-join-hint-off-but-silently-fa.md new file mode 100644 index 0000000..4a403a4 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65838-inl-join-hint-shows-warning-when-tidb-opt-advanced-join-hint-off-but-silently-fa.md @@ -0,0 +1,17 @@ +# Issue #65838: `inl_join` hint shows warning when `tidb_opt_advanced_join_hint=OFF` but silently fails when `ON` + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65838 +- Status: open +- Type: type/bug +- Created At: 2026-01-27T07:41:56Z +- Labels: affects-8.5, contribution, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- Executed a query with hint on TiDB v8.5.5 with different settings of : + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65867-tidb-runtime-error-invalid-memory-address-or-nil-pointer-dereference.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65867-tidb-runtime-error-invalid-memory-address-or-nil-pointer-dereference.md new file mode 100644 index 0000000..f14cef6 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65867-tidb-runtime-error-invalid-memory-address-or-nil-pointer-dereference.md @@ -0,0 +1,27 @@ +# Issue #65867: TiDB：runtime error: invalid memory address or nil pointer dereference + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65867 +- Status: closed +- Type: type/bug +- Created At: 2026-01-28T07:02:48Z +- Closed At: 2026-02-25T10:52:59Z +- Labels: affects-8.5, contribution, first-time-contributor, impact/panic, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- This table always being ‘inset’and 'delete' , and suddenly the following error popped up: It recovers after doing the steps below, but the issue keeps coming back frequently. table info and field : + +## Linked PRs +- Fix PR #66251: planner: fix a nil pointer bug caused by nil TopN + URL: https://github.com/pingcap/tidb/pull/66251 + State: closed + Merged At: 2026-02-14T04:46:41Z + Changed Files Count: 2 + Main Modules: pkg/planner, pkg/planner/core + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/core/integration_test.go + PR Summary: What problem does this PR solve? Problem Summary: planner: fix a nil pointer bug caused by nil TopN What changed and how does it work? planner: fix a nil pointer bug caused by nil TopN + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65876-planner-non-prep-plan-cache-support-is-null.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65876-planner-non-prep-plan-cache-support-is-null.md new file mode 100644 index 0000000..67f54cf --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65876-planner-non-prep-plan-cache-support-is-null.md @@ -0,0 +1,28 @@ +# Issue #65876: planner: non-prep plan cache support "is null" + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65876 +- Status: closed +- Type: type/enhancement +- Created At: 2026-01-28T09:41:21Z +- Closed At: 2026-02-11T21:46:48Z +- Labels: epic/plan-cache, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- According to [our official document]() , if a query contains predicates, it can't use non-prepared plan cache: + +## Linked PRs +- Fix PR #66134: planner: Support cacheing queries with IS NULL expressions. + URL: https://github.com/pingcap/tidb/pull/66134 + State: closed + Merged At: 2026-02-11T21:46:47Z + Changed Files Count: 3 + Main Modules: pkg/planner/core + Sample Changed Files: + - pkg/planner/core/casetest/plancache/plan_cache_suite_test.go + - pkg/planner/core/casetest/plancache/plan_cacheable_checker_test.go + - pkg/planner/core/plan_cacheable_checker.go + PR Summary: What problem does this PR solve? Problem Summary: Non-prepared queries with IS NULL expressions cannot be cached. What changed and how does it work? Added to the allowlist of AST node types in . Also sorted the type list alphabetically. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65878-planner-plan-cache-support-queries-with-int-col-str-val.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65878-planner-plan-cache-support-queries-with-int-col-str-val.md new file mode 100644 index 0000000..e75bead --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65878-planner-plan-cache-support-queries-with-int-col-str-val.md @@ -0,0 +1,26 @@ +# Issue #65878: planner: plan-cache support queries with "int_col = str_val" + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65878 +- Status: open +- Type: type/enhancement +- Created At: 2026-01-28T09:51:56Z +- Labels: epic/plan-cache, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- According to [our official document](), if a query contains a predicate like "int_col = str_val", it can't hit the plan cache, please see the case below: + +## Linked PRs +- Fix PR #66198: expression: skip refineArgs for comparisons when plan cache is active + URL: https://github.com/pingcap/tidb/pull/66198 + State: open + Merged At: not merged + Changed Files Count: 2 + Main Modules: pkg/expression, pkg/planner/core + Sample Changed Files: + - pkg/expression/builtin_compare.go + - pkg/planner/core/casetest/plancache/plan_cache_suite_test.go + PR Summary: What problem does this PR solve? Problem Summary: The function in tried to permit certain comparison argument refinements (e.g. -> ) while marking the plan as uncacheable via . This unnecessarily prevented plan caching for queries with comparisons involving int, year, decimal, double, and datetime columns. What changed and how does it work? Removed the function. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65915-analyze-statement-freezes-if-saveanalyzeresulttostorage-meets-error-when-analyze.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65915-analyze-statement-freezes-if-saveanalyzeresulttostorage-meets-error-when-analyze.md new file mode 100644 index 0000000..75e5ac3 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65915-analyze-statement-freezes-if-saveanalyzeresulttostorage-meets-error-when-analyze.md @@ -0,0 +1,29 @@ +# Issue #65915: `ANALYZE` statement freezes if `SaveAnalyzeResultToStorage` meets error when analyzeing partitioned table + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65915 +- Status: closed +- Type: type/bug +- Created At: 2026-01-29T12:15:28Z +- Closed At: 2026-02-27T12:15:49Z +- Labels: affects-8.5, component/statistics, report/customer, severity/major, sig/planner, type/bug + +## Customer-Facing Phenomenon +- Analyze statement freezes forever, and can't be ed. + +## Linked PRs +- Fix PR #66169: executor, statistics: avoid analyze hang on save error + URL: https://github.com/pingcap/tidb/pull/66169 + State: closed + Merged At: 2026-02-27T12:15:48Z + Changed Files Count: 4 + Main Modules: pkg/executor, pkg/statistics + Sample Changed Files: + - pkg/executor/analyze.go + - pkg/executor/analyze_test.go + - pkg/executor/analyze_worker.go + - pkg/statistics/handle/storage/save.go + PR Summary: What problem does this PR solve? Problem Summary: ANALYZE on heavily partitioned tables can hang when saving stats fails (for example, lock wait timeout), because save workers exit early and analyze workers block on result channels. What changed and how does it work? Add a failpoint to simulate save errors and a unit test that asserts ANALYZE returns an error instead of hanging. + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65916-explain-for-connection-fails-with-non-prepared-plan-cache-parameterization.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65916-explain-for-connection-fails-with-non-prepared-plan-cache-parameterization.md new file mode 100644 index 0000000..5536d79 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65916-explain-for-connection-fails-with-non-prepared-plan-cache-parameterization.md @@ -0,0 +1,17 @@ +# Issue #65916: EXPLAIN FOR CONNECTION fails with non‑prepared plan cache parameterization + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65916 +- Status: open +- Type: type/bug +- Created At: 2026-01-29T13:34:26Z +- Labels: affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- ERROR 1815 (HY000): expression eq(test.status, ?) cannot be pushed down + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65924-the-estrows-of-equal-condition-is-overestimated-when-the-modify-count-is-zero.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65924-the-estrows-of-equal-condition-is-overestimated-when-the-modify-count-is-zero.md new file mode 100644 index 0000000..cb6da4d --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-65924-the-estrows-of-equal-condition-is-overestimated-when-the-modify-count-is-zero.md @@ -0,0 +1,54 @@ +# Issue #65924: The estRows of equal condition is overestimated when the modify count is zero + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/65924 +- Status: open +- Type: type/bug +- Created At: 2026-01-30T06:41:56Z +- Labels: affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- 1. Minimal reproduce step (Required) + +## Linked PRs +- Fix PR #66145: planner: Consolidate out-of-range equal to use common func | tidb-test=pr/2684 + URL: https://github.com/pingcap/tidb/pull/66145 + State: open + Merged At: not merged + Changed Files Count: 21 + Main Modules: pkg/planner/core, tests/integrationtest, pkg/planner, pkg/statistics, pkg/executor + Sample Changed Files: + - pkg/executor/test/analyzetest/analyze_test.go + - pkg/planner/cardinality/row_count_column.go + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/selectivity.go + - pkg/planner/cardinality/selectivity_test.go + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + - pkg/planner/core/casetest/cascades/testdata/cascades_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_out.json + - pkg/planner/core/casetest/cbotest/testdata/analyze_suite_xut.json + - pkg/planner/core/casetest/partition/testdata/integration_partition_suite_out.json + - pkg/planner/core/casetest/partition/testdata/integration_partition_suite_xut.json + - pkg/planner/core/casetest/testdata/stats_suite_out.json + - pkg/planner/core/casetest/testdata/stats_suite_xut.json + - pkg/statistics/handle/globalstats/global_stats_test.go + - pkg/statistics/histogram.go + - tests/integrationtest/r/executor/issues.result + - tests/integrationtest/r/executor/partition/partition_with_expression.result + - tests/integrationtest/r/explain_generate_column_substitute.result + - tests/integrationtest/r/imdbload.result + - tests/integrationtest/r/planner/core/cbo.result + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? While this affects 8.5 - it will be very difficult to cherry pick. The goal here is to consolidate all "out-of-range" estimation to use the same logic. It shouldn't matter if you are out-of-range for an equals or a range predicate. This is only targeting stats V2 use. +- Fix PR #66310: planner: out of Range full NDV + URL: https://github.com/pingcap/tidb/pull/66310 + State: open + Merged At: not merged + Changed Files Count: 2 + Main Modules: pkg/planner + Sample Changed Files: + - pkg/planner/cardinality/row_count_index.go + - pkg/planner/cardinality/testdata/cardinality_suite_out.json + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66203-planner-decrease-the-default-value-64mb-of-tidb-opt-range-max-size.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66203-planner-decrease-the-default-value-64mb-of-tidb-opt-range-max-size.md new file mode 100644 index 0000000..88810ed --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66203-planner-decrease-the-default-value-64mb-of-tidb-opt-range-max-size.md @@ -0,0 +1,17 @@ +# Issue #66203: planner: decrease the default value (64MB) of `tidb_opt_range_max_size` + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/66203 +- Status: open +- Type: type/enhancement +- Created At: 2026-02-11T03:44:01Z +- Labels: report/customer, sig/planner, type/enhancement, type/performance + +## Customer-Facing Phenomenon +- The second explain-stmt spent more than to finish: + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66320-planner-correlate-a-non-correlated-in-subquery.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66320-planner-correlate-a-non-correlated-in-subquery.md new file mode 100644 index 0000000..c7beb39 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66320-planner-correlate-a-non-correlated-in-subquery.md @@ -0,0 +1,44 @@ +# Issue #66320: Planner: correlate a non-correlated IN subquery + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/66320 +- Status: open +- Type: type/bug +- Created At: 2026-02-21T00:19:21Z +- Labels: report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- TiDB can "decorrelate" a correlated subquery today, but cannot correlate a subquery that was originally non-correlated. + +## Linked PRs +- Fix PR #66206: planner: correlate subquery rule + URL: https://github.com/pingcap/tidb/pull/66206 + State: open + Merged At: not merged + Changed Files Count: 22 + Main Modules: pkg/planner/core, pkg/session, pkg/bindinfo, pkg/planner + Sample Changed Files: + - pkg/bindinfo/binding_auto_test.go + - pkg/planner/core/BUILD.bazel + - pkg/planner/core/casetest/rule/BUILD.bazel + - pkg/planner/core/casetest/rule/main_test.go + - pkg/planner/core/casetest/rule/rule_correlate_test.go + - pkg/planner/core/casetest/rule/testdata/correlate_suite_in.json + - pkg/planner/core/casetest/rule/testdata/correlate_suite_out.json + - pkg/planner/core/casetest/rule/testdata/correlate_suite_xut.json + - pkg/planner/core/core_init.go + - pkg/planner/core/exhaust_physical_plans.go + - pkg/planner/core/expression_rewriter.go + - pkg/planner/core/find_best_task.go + - pkg/planner/core/operator/logicalop/logical_join.go + - pkg/planner/core/operator/physicalop/base_physical_plan.go + - pkg/planner/core/optimizer.go + - pkg/planner/core/optimizer_test.go + - pkg/planner/core/rule/logical_rules.go + - pkg/planner/core/rule_correlate.go + - pkg/planner/util/utilfuncp/func_pointer_misc.go + - pkg/sessionctx/vardef/tidb_vars.go + PR Summary: What problem does this PR solve? Problem Summary: What changed and how does it work? Claude code review: Detailed PR Review: Cost-Based Selection Between Join and Apply for Non-Correlated IN Subqueries + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66623-planner-same-plans-with-different-in-list-should-have-the-same-plan-digests.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66623-planner-same-plans-with-different-in-list-should-have-the-same-plan-digests.md new file mode 100644 index 0000000..f263795 --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66623-planner-same-plans-with-different-in-list-should-have-the-same-plan-digests.md @@ -0,0 +1,56 @@ +# Issue #66623: planner: same plans with different in-list should have the same plan digests + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/66623 +- Status: closed +- Type: type/bug +- Created At: 2026-03-02T04:09:13Z +- Closed At: 2026-03-04T15:52:21Z +- Labels: affects-8.5, report/customer, severity/moderate, sig/planner, type/bug + +## Customer-Facing Phenomenon +- These queries and plans are the same, but the is flooded with different plans, which significantly affect the user experience: The root cause is that they have different plan digest, we need to update our plan digest algorithm to ignore different length of in-list: + +## Linked PRs +- Fix PR #66624: planner: enable tidb_ignore_inlist_plan_digest by default to improve user experience and add more test cases + URL: https://github.com/pingcap/tidb/pull/66624 + State: closed + Merged At: 2026-03-04T15:52:18Z + Changed Files Count: 5 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/casetest/plan_test.go + - pkg/planner/core/casetest/testdata/plan_normalized_suite_out.json + - pkg/planner/core/casetest/testdata/plan_normalized_suite_xut.json + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/sysvar_test.go + PR Summary: What problem does this PR solve? Problem Summary: Queries with the same plan but different IN-list lengths (e.g., vs ) produce different plan digests. This causes the Dashboard to show many similar plans and harms usability. What changed and how does it work? See 1. **Default value change:** Set the default of from to in . When enabled, plan digests ignore IN-list length so queries that only differ in IN-list size share the same plan digest. +- Fix PR #66683: planner: enable tidb_ignore_inlist_plan_digest by default to improve user experience and add more test cases (#66624) + URL: https://github.com/pingcap/tidb/pull/66683 + State: closed + Merged At: not merged + Changed Files Count: 5 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/casetest/plan_test.go + - pkg/planner/core/casetest/testdata/plan_normalized_suite_out.json + - pkg/planner/core/casetest/testdata/plan_normalized_suite_xut.json + - pkg/sessionctx/vardef/tidb_vars.go + - pkg/sessionctx/variable/sysvar_test.go + PR Summary: This is an automated cherry-pick of #66624 What problem does this PR solve? Problem Summary: Queries with the same plan but different IN-list lengths (e.g., vs ) produce different plan digests. This causes the Dashboard to show many similar plans and harms usability. What changed and how does it work? See +- Fix PR #66698: planner: enable tidb_ignore_inlist_plan_digest by default to improve user experience and add more test cases (#66624) + URL: https://github.com/pingcap/tidb/pull/66698 + State: open + Merged At: not merged + Changed Files Count: 5 + Main Modules: pkg/planner/core, pkg/session + Sample Changed Files: + - pkg/planner/core/casetest/plan_test.go + - pkg/planner/core/casetest/testdata/plan_normalized_suite_out.json + - pkg/planner/core/casetest/testdata/plan_normalized_suite_xut.json + - pkg/sessionctx/variable/sysvar_test.go + - pkg/sessionctx/variable/tidb_vars.go + PR Summary: This is an automated cherry-pick of #66624 What problem does this PR solve? Problem Summary: Queries with the same plan but different IN-list lengths (e.g., vs ) produce different plan digests. This causes the Dashboard to show many similar plans and harms usability. What changed and how does it work? See + +## Notes +- At least one merged PR was found. The merge timestamp above can be used as the fix landing time in the main branch. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66658-planner-decrease-concurrency-batch-size-automatically-for-streaming-plans-with-l.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66658-planner-decrease-concurrency-batch-size-automatically-for-streaming-plans-with-l.md new file mode 100644 index 0000000..eafc0ac --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66658-planner-decrease-concurrency-batch-size-automatically-for-streaming-plans-with-l.md @@ -0,0 +1,17 @@ +# Issue #66658: planner: decrease concurrency / batch_size automatically for streaming plans with limit-clause + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/66658 +- Status: open +- Type: type/enhancement +- Created At: 2026-03-03T10:48:48Z +- Labels: report/customer, sig/execution, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- We have a clause, but TiDB still scanned more than 390000+ rows. The root cause is that our default concurrency (5, 15) and batch-size (25000) is too large. We have to scan and process more rows before the plan is stopped by the limit-clause. This could waste some system resources, like CPU. + +## Linked PRs +- No linked PR was found from the issue timeline. + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66668-planner-can-t-use-primary-as-an-index-in-indexmerge-for-predicate-id-and-a-or-b.md b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66668-planner-can-t-use-primary-as-an-index-in-indexmerge-for-predicate-id-and-a-or-b.md new file mode 100644 index 0000000..ce2016e --- /dev/null +++ b/skills/tidb-query-tuning/references/tidb-customer-planner-issues/issue-66668-planner-can-t-use-primary-as-an-index-in-indexmerge-for-predicate-id-and-a-or-b.md @@ -0,0 +1,27 @@ +# Issue #66668: planner: can't use primary as an index in IndexMerge for predicate `id=? and (a=? or b=?)` + +## Metadata +- Issue: https://github.com/pingcap/tidb/issues/66668 +- Status: open +- Type: type/enhancement +- Created At: 2026-03-04T02:53:02Z +- Labels: plan-rewrite, report/customer, sig/planner, type/enhancement + +## Customer-Facing Phenomenon +- Enhancement See the example below, we can use IndexMerge for the second plan, but can't for the first plan: + +## Linked PRs +- Fix PR #66670: pkg/planner: allow primary key as IndexMerge partial path for `id=? and (a=? or b=?)` + URL: https://github.com/pingcap/tidb/pull/66670 + State: open + Merged At: not merged + Changed Files Count: 3 + Main Modules: tests/integrationtest, pkg/planner/core + Sample Changed Files: + - pkg/planner/core/indexmerge_unfinished_path.go + - tests/integrationtest/r/planner/core/indexmerge_path.result + - tests/integrationtest/t/planner/core/indexmerge_path.test + PR Summary: What problem does this PR solve? Problem Summary: IndexMerge cannot use the primary key as a partial path for predicates like . For example, falls back to TableRangeScan instead of IndexMerge, while works correctly. What changed and how does it work? In , the table path (primary key for clustered tables) was skipped whenever returned nil. For conditions such as , each DNF branch (, ) is first handled separately, and the top-level is merged later. Because alone cannot build a valid range on primary key , failed and the table path was always skipped, so it never got the merged filters. **Fix:** Only skip the table path when (int-handle tables). For common-handle/clustered tables, holds the primary key index, so we keep the gradual filter collection and allow filters like to be merge + +## Notes +- This issue is still open. Use this file as a reminder list for customer-driven gaps that still need a fix or a completed rollout. diff --git a/skills/tidb-query-tuning/scripts/generate_tidb_issue_experiences.py b/skills/tidb-query-tuning/scripts/generate_tidb_issue_experiences.py new file mode 100755 index 0000000..d00bf6c --- /dev/null +++ b/skills/tidb-query-tuning/scripts/generate_tidb_issue_experiences.py @@ -0,0 +1,486 @@ +#!/usr/bin/env python3 +import argparse +import json +import re +import subprocess +import sys +import time +from collections import Counter +from datetime import datetime, timedelta +from pathlib import Path + + +REPO = "pingcap/tidb" +DEFAULT_QUERY = 'repo:pingcap/tidb is:issue label:"report/customer" label:"sig/planner" created:>=2024-01-01' +DEFAULT_OUT_DIR = Path("outputs/tidb-customer-planner-issues") +MAX_RETRIES = 3 +SLEEP_SECS = 0.15 + + +def parse_args(): + parser = argparse.ArgumentParser( + description="Generate a local markdown corpus from GitHub issue and PR history." + ) + parser.add_argument( + "--repo", + default=REPO, + help="GitHub repository in owner/name form. Default: pingcap/tidb", + ) + parser.add_argument( + "--query", + default=DEFAULT_QUERY, + help="GitHub issue search query used by gh api search/issues.", + ) + parser.add_argument( + "--out-dir", + default=str(DEFAULT_OUT_DIR), + help="Directory where issue markdown files and README.md are written.", + ) + return parser.parse_args() + + +def run(cmd): + proc = subprocess.run(cmd, capture_output=True, text=True) + if proc.returncode != 0: + raise RuntimeError( + f"command failed: {' '.join(cmd)}\nstdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return proc.stdout + + +def gh_json(args): + last_err = None + for _ in range(MAX_RETRIES): + try: + out = run(["gh", "api", *args]) + return json.loads(out) + except Exception as exc: + last_err = exc + time.sleep(0.5) + raise last_err + + +def fetch_search_issues(query): + issues = [] + page = 1 + while True: + data = gh_json( + [ + "search/issues", + "-X", + "GET", + "-f", + f"q={query}", + "-f", + "per_page=100", + "-f", + f"page={page}", + ] + ) + items = data.get("items", []) + if not items: + break + issues.extend(items) + if len(items) < 100: + break + page += 1 + time.sleep(SLEEP_SECS) + return issues + + +def fetch_timeline(repo, issue_number): + page = 1 + items = [] + while True: + batch = gh_json( + [ + f"repos/{repo}/issues/{issue_number}/timeline", + "-X", + "GET", + "-H", + "Accept: application/vnd.github+json", + "-f", + "per_page=100", + "-f", + f"page={page}", + ] + ) + if not batch: + break + items.extend(batch) + if len(batch) < 100: + break + page += 1 + time.sleep(SLEEP_SECS) + return items + + +def fetch_pr(repo, pr_number): + return gh_json([f"repos/{repo}/pulls/{pr_number}"]) + + +def fetch_pr_files(repo, pr_number): + files = [] + page = 1 + while True: + batch = gh_json( + [ + f"repos/{repo}/pulls/{pr_number}/files", + "-X", + "GET", + "-f", + "per_page=100", + "-f", + f"page={page}", + ] + ) + if not batch: + break + files.extend(batch) + if len(batch) < 100: + break + page += 1 + time.sleep(SLEEP_SECS) + return files + + +def slugify(text): + text = text.lower() + text = re.sub(r"[^a-z0-9]+", "-", text) + text = text.strip("-") + return text[:80] or "issue" + + +def issue_type(labels): + for label in labels: + name = label["name"] + if name.startswith("type/"): + return name + return "type/unknown" + + +def label_names(labels): + return sorted(label["name"] for label in labels) + + +def sanitize_text(text): + text = text.replace("\r\n", "\n") + text = re.sub(r"", "", text) + text = re.sub(r"https://github\.com/[^\s)]+", "", text) + text = re.sub(r"https?://[^\s)]+", "", text) + text = re.sub(r"!\[[^\]]*\]\([^)]+\)", "", text) + text = re.sub(r"]*>", "", text) + text = re.sub(r"`{3}[\s\S]*?`{3}", "", text) + text = re.sub(r"`[^`]*`", "", text) + text = re.sub(r"^#+\s*", "", text, flags=re.M) + text = re.sub(r"\n{3,}", "\n\n", text) + return text.strip() + + +def parse_iso8601(ts): + if not ts: + return None + return datetime.fromisoformat(ts.replace("Z", "+00:00")) + + +def extract_section(body, heading): + pattern = re.compile(rf"{re.escape(heading)}\s*(.*?)(?=\n### |\n## |\Z)", re.S) + match = pattern.search(body) + if match: + return sanitize_text(match.group(1)) + return "" + + +def extract_phenomenon(body): + body = body or "" + for heading in [ + "### 3. What did you see instead (Required)", + "### 1. Minimal reproduce step (Required)", + "### What did you see instead (Required)", + "### Problem Summary", + "## Problem Summary", + ]: + section = extract_section(body, heading) + if section: + lines = [ln.strip("- ").strip() for ln in section.splitlines() if ln.strip()] + lines = [ln for ln in lines if not ln.startswith("