Close ort correctness gaps vs C Git: recursive merge, dir-rename relevance, submodules, attributes

Bring the pure-Python ort engine to byte-for-byte parity with C Git (v2.44)
across the result-affecting behaviors that the single-base implementation
previously diverged on.

Recursive merge (virtual merge base):
- Port merge_ort_internal/merge_incore_recursive: compute all merge bases and
  recursively merge them into a virtual ancestor (mergeort.merge_recursive),
  exposed via ort.merge_commits and used by `git merge` (porcelain_merge).
  Matches `git merge-tree <a> <b>` (no --merge-base) on criss-cross histories.
- Implement every call_depth>0 behavior: modify/delete uses the base stage,
  distinct-types/symlink use the merge-base version, binary merge takes the
  ancestor (virtual_ancestor), and content-merge marker size grows with depth.
- Rewrite merge.merge_bases as a faithful paint_down_to_common
  (date-ordered priority walk with insertion-order tie-breaking) plus
  remove_redundant, so merge bases come back in C Git's exact order — which
  the virtual-ancestor construction (and its stage-1 blob) depends on.

Directory rename detection:
- Track dir_rename_mask through collect_merge_info with the per-directory 0x07
  flip, computing rename-source relevance (content_relevant || location_relevant)
  and removed-dir relevance (RELEVANT_FOR_SELF/ANCESTOR/NOT_RELEVANT).
- Cull non-relevant sources from inexact/basename rename matching in diffcore
  (matching diffcore_rename_extended) so rename *pairings* match git, and only
  count dir renames from relevant sources. Fixes nested, split, transitive, and
  "rename whose source the other side left untouched" cases.

Submodules: port merge_submodule fast-forward (descendant wins when the
submodule object store is available; conflict otherwise, like git).

Config/attributes: honor merge.conflictStyle (merge/diff3/zdiff3) and the
working-tree .gitattributes merge / conflict-marker-size attributes (incl.
merge=union), matching how git's attr stack reads them for merge-tree.

Validated byte-for-byte vs git 2.44 over thousands of randomized cases
(blob merges, criss-cross/recursive merges, mixed and nested directory
renames) plus new pytest parity tests; full suite 127 passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: blhsing

README.md

@@ -266,31 +266,38 @@ mechanism.
 
 ### Merge
 
-`merge.merge_bases` mirrors `commit-reach.c`'s `paint_down_to_common`: BFS
-from both tips with PARENT1/PARENT2 flags, marking double-flagged commits as
-results and pushing STALE to their ancestors.
+`merge.merge_bases` is a faithful port of `commit-reach.c`'s
+`paint_down_to_common`: a date-ordered priority walk with PARENT1/PARENT2/STALE
+flags and insertion-order tie-breaking, followed by `remove_redundant`, so the
+merge bases come back in the **same order** C Git returns them (which the
+recursive merge below depends on).
 
 High-level three-way merges run a pure-Python port of Git's own `ort` engine —
-no `git` binary and no fallback engine. The port lives in three modules and
+no `git` binary and no fallback engine. The port lives in four modules and
 reproduces `git merge-tree --write-tree` byte-for-byte (result tree oid,
 conflicted blobs with markers, and conflicted index stages):
 
 * `xdiff.py` — Git's xdiff library: record classification, the **histogram**
   diff that `ort` hardcodes for content merges (with the classic Myers
   algorithm as its documented fallback), change compaction, and the zealous
-  three-way `xdl_merge` that emits `<<<<<<<` / `=======` / `>>>>>>>` markers.
+  three-way `xdl_merge` that emits `<<<<<<<` / `=======` / `>>>>>>>` markers
+  (merge / diff3 / zdiff3 styles, configurable marker size).
 * `diffcore.py` — rename detection: the `diffcore-delta` spanhash similarity
   estimator plus exact, basename-driven, and inexact NxM matrix matching from
-  `diffcore-rename.c`.
+  `diffcore-rename.c`, with `relevant_sources` source-culling.
 * `mergeort.py` — the `merge-ort.c` tree engine: the recursive three-way tree
-  walk (`collect_merge_info`), file and **directory** rename detection and
-  resolution (`process_renames`), per-path resolution (`process_entry`), and
-  streamed result-tree assembly with conflicted index stages.
-
-`ort.py` is a thin adapter exposing `merge_tree(repo, merge_base, ours,
-theirs)`; the `merge_base`/`ours`/`theirs` arguments double as the
-conflict-marker labels, exactly as the corresponding `git merge-tree
---merge-base` arguments do.
+  walk (`collect_merge_info`, tracking `dir_rename_mask` and rename-source
+  relevance), file and **directory** rename detection/resolution
+  (`process_renames`, dir-rename counting with RELEVANT_FOR_SELF/ANCESTOR
+  gating), per-path resolution (`process_entry`, including the `call_depth`
+  virtual-ancestor behaviors), submodule fast-forward, `.gitattributes`
+  `merge`/`conflict-marker-size` handling, and streamed result-tree assembly.
+* `ort.py` — adapter exposing `merge_tree(repo, merge_base, ours, theirs)`
+  (explicit base, like `git merge-tree --merge-base`) and `merge_commits(repo,
+  ours, theirs)` (computes all merge bases and **recursively** merges them into
+  a virtual ancestor, like `git merge-tree <a> <b>`). The tree-ish arguments
+  double as conflict-marker labels, exactly as the matching `git merge-tree`
+  arguments do; `merge.conflictStyle` is honored.
 
 ### Rerere
 
@@ -405,13 +412,15 @@ randomized cases.
   remaining scale-sensitive cases are commands whose output inherently requires
   inspecting every path or blob.
 * The `ort` merge engine is a pure-Python reimplementation (no `git` binary,
-  no fallback) and is validated for byte-for-byte parity against
-  `git merge-tree --write-tree` across content merges, rename detection
-  (file and directory), and conflict presentation. It targets a single merge
-  base (as `git merge-tree --merge-base` provides); recursive merge of multiple
-  merge bases (a virtual ancestor) and full submodule fast-forward resolution
-  are not modelled, and `merge.conflictStyle`/whitespace merge drivers default
-  to Git's standard behavior.
+  no fallback), validated for byte-for-byte parity against
+  `git merge-tree --write-tree` across content merges, file and directory
+  renames, recursive merges (criss-cross histories with a virtual ancestor),
+  submodule fast-forwards, conflict styles (merge/diff3/zdiff3), `merge=union`
+  attributes, and conflict presentation. Two areas are not fully modelled:
+  custom external `.gitattributes` merge drivers (treated as the built-in text
+  driver), and a small number of pathological deeply-nested simultaneous
+  directory-rename cases where Git's `merge-ort` deferred two-pass traversal
+  computes rename-source relevance slightly differently.
 * `fsmonitor-daemon run` uses native filesystem notifications on Windows and
   Linux (`ReadDirectoryChangesW` / inotify). One-shot `fsmonitor` calls and
   unsupported platforms fall back to configurable polling.

pythongit/diffcore.py

@@ -212,12 +212,17 @@ def detect_renames(
     rename_limit: int = 7000,
     minimum_score: int = 0,
     rename_empty: bool = False,
+    relevant_sources: Optional[set] = None,
 ) -> list[RenamePair]:
     """Detect file renames between two trees represented as path->(mode, oid).
 
     Sources are paths present in base but absent in side (deletions);
     destinations are paths present in side but absent in base (additions).
     Returns the list of detected rename pairs.
+
+    ``relevant_sources`` (if given) limits inexact/basename rename detection to
+    those source paths (exact renames still consider all sources), mirroring
+    merge-ort's relevant_sources culling in diffcore_rename_extended.
     """
     if minimum_score == 0:
         minimum_score = DEFAULT_RENAME_SCORE
@@ -304,6 +309,8 @@ def remaining_srcs() -> list[int]:
     for i in range(len(srcs)):
         if srcs[i].rename_used:
             continue
+        if relevant_sources is not None and srcs[i].path not in relevant_sources:
+            continue
         base = _basename(srcs[i].path)
         src_index = src_base.get(base, -1)
         if base in dst_base:
@@ -321,7 +328,9 @@ def remaining_srcs() -> list[int]:
             record(dst_index, src_index, score)
 
     # --- inexact matrix (NxM similarity) ---
-    src_idx = remaining_srcs()
+    # cull sources not in relevant_sources (remove_unneeded_paths_from_src)
+    src_idx = [i for i in remaining_srcs()
+               if relevant_sources is None or srcs[i].path in relevant_sources]
     num_sources = len(src_idx)
     num_destinations = sum(1 for i in range(len(dsts)) if not dst_is_rename[i])
     if not num_sources or not num_destinations:

pythongit/merge.py

@@ -40,43 +40,92 @@ def _parents(repo: Repository, sha: str) -> list[str]:
     return objs.parse_commit(data).parents
 
 
-def merge_bases(repo: Repository, a: str, b: str) -> list[str]:
-    if a == b:
-        return [a]
-    flags: dict[str, int] = {a: PARENT1, b: PARENT2}
-    # max-heap by commit time via negative
-    pq: list[tuple[int, str]] = []
-    heapq.heappush(pq, (-_commit_time(repo, a), a))
-    heapq.heappush(pq, (-_commit_time(repo, b), b))
-    result: list[str] = []
-    while pq:
-        # check if any non-stale remain with both flags possible
-        if all((flags[s] & STALE) for _, s in pq):
-            break
-        _, sha = heapq.heappop(pq)
-        f = flags.get(sha, 0)
-        if (f & (PARENT1 | PARENT2)) == (PARENT1 | PARENT2):
-            if not (f & RESULT):
-                f |= RESULT
-                result.append(sha)
+def _insert_by_date(lst: list, item: str, date: int) -> None:
+    """Insert into a date-descending list; equal dates keep insertion (FIFO),
+    mirroring commit_list_insert_by_date."""
+    i = 0
+    while i < len(lst) and lst[i][1] >= date:
+        i += 1
+    lst.insert(i, (item, date))
+
+
+def _paint_down_to_common(repo: Repository, one: str, twos: list[str]):
+    """Faithful port of commit-reach.c:paint_down_to_common (no commit-graph
+    generation numbers, so ordering is by commit date). Returns
+    (result_shas_in_date_order, flags)."""
+    flags: dict[str, int] = {one: PARENT1}
+    # min-heap on (-date, insertion_ctr) → pops newest first, FIFO for ties
+    heap: list[tuple[int, int, str]] = []
+    ctr = 0
+    heapq.heappush(heap, (-_commit_time(repo, one), ctr, one))
+    ctr += 1
+    for t in twos:
+        flags[t] = flags.get(t, 0) | PARENT2
+        heapq.heappush(heap, (-_commit_time(repo, t), ctr, t))
+        ctr += 1
+
+    result: list[tuple[str, int]] = []
+    while any(not (flags.get(s, 0) & STALE) for _, _, s in heap):
+        negd, _c, commit = heapq.heappop(heap)
+        f = flags.get(commit, 0) & (PARENT1 | PARENT2 | STALE)
+        if f == (PARENT1 | PARENT2):
+            if not (flags.get(commit, 0) & RESULT):
+                flags[commit] = flags.get(commit, 0) | RESULT
+                _insert_by_date(result, commit, -negd)
             f |= STALE
-        flags[sha] = f
-        carry = f & (PARENT1 | PARENT2 | STALE)
-        for p in _parents(repo, sha):
+        for p in _parents(repo, commit):
             pf = flags.get(p, 0)
-            if (pf & carry) == carry:
+            if (pf & f) == f:
                 continue
-            flags[p] = pf | carry
-            heapq.heappush(pq, (-_commit_time(repo, p), p))
-    # filter out stale results
-    return [s for s in result if not (flags.get(s, 0) & STALE) or (flags.get(s, 0) & RESULT)]
+            flags[p] = pf | f
+            heapq.heappush(heap, (-_commit_time(repo, p), ctr, p))
+            ctr += 1
+    return [s for s, _ in result], flags
+
+
+def _remove_redundant(repo: Repository, array: list[str]) -> list[str]:
+    """Port of remove_redundant_no_gen: drop merge bases that are ancestors of
+    other merge bases, preserving order."""
+    cnt = len(array)
+    redundant = [False] * cnt
+    for i in range(cnt):
+        if redundant[i]:
+            continue
+        work = []
+        filled_index = []
+        for j in range(cnt):
+            if i == j or redundant[j]:
+                continue
+            filled_index.append(j)
+            work.append(array[j])
+        if not work:
+            continue
+        _res, flags = _paint_down_to_common(repo, array[i], work)
+        if flags.get(array[i], 0) & PARENT2:
+            redundant[i] = True
+        for k, wj in enumerate(work):
+            if flags.get(wj, 0) & PARENT1:
+                redundant[filled_index[k]] = True
+    return [array[i] for i in range(cnt) if not redundant[i]]
+
+
+def merge_bases(repo: Repository, a: str, b: str) -> list[str]:
+    """Return the merge bases of two commits, in git's order
+    (repo_get_merge_bases): date-descending with FIFO tie-breaking, redundant
+    bases removed."""
+    if a == b:
+        return [a]
+    res_shas, flags = _paint_down_to_common(repo, a, [b])
+    result = [s for s in res_shas if not (flags.get(s, 0) & STALE)]
+    if len(result) <= 1:
+        return result
+    return _remove_redundant(repo, result)
 
 
 def is_ancestor(repo: Repository, ancestor: str, descendant: str) -> bool:
     if ancestor == descendant:
         return True
-    bases = merge_bases(repo, ancestor, descendant)
-    return ancestor in bases
+    return ancestor in merge_bases(repo, ancestor, descendant)
 
 
 # ---------------------------------------------------------------------------