Add validation + compatibility tests for DeepLabCut-live (#56)

* dlc_processor: add lightweight validation of shape and dtype (+tests)

* Add basic compatibility tests for deeplabcut-live API

* Add optional smoke test for exported dlclive model via env vars

* Update dlclivegui/services/dlc_processor.py

Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>

* Update dlclivegui/services/dlc_processor.py

Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>

* Update dlclivegui/services/dlc_processor.py

Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>

* Remove duplicate commands block from rebase

Remove the earlier pytest commands block and update the tox 'commands' to run pytest with the marker excluding both 'hardware' and 'dlclive_compat'. Adjust coverage invocation to use the installed package path (--cov={envsitepackagesdir}/dlclivegui) and emit per-env XML coverage files (.coverage.{envname}.xml). This aligns tox behavior with the GitHub Actions job and removes the prior posargs-based command duplication.

* Use Python 3.12 in CI testing matrix

Update the GitHub Actions testing matrix to run on Python 3.12 instead of 3.11. This moves CI to test against the newer Python runtime while keeping existing matrix include entries unchanged.

* Run pre-commit

* Use PoseBackends enum for backend field

Add a PoseBackends Enum and switch PoseSource.backend from a string to that enum for stronger typing and clarity. Update default PosePacket/PoseSource instances and the validate_pose_array signature to use PoseBackends.DLC_LIVE. Import Enum/auto and remove an unused sentinel variable. These are small refactors to improve type-safety and consistency around backend identification.

* Update & fix DLCLive compatibility tests

Relax and correct DLCLive compatibility checks: only consider keyword/positional params when inspecting __init__, drop the assertion that __init__ must accept **kwargs, and only require 'frame' for init_inference/get_pose (frame_time is passed as a kwarg to processors). Also update FakeDLCLive.get_pose to return an array of shape (2, 3) to match the expected pose output shape.

* Fix hard-coded str to use enum

* Ensure tox failure fails CI job rather than silently passing

Set the test step shell to `bash -eo pipefail` and redirect `tox` stderr into `tee` (changed `tox -q` to `tox -q 2>&1 | tee tox-output.log`). This ensures pipeline failures are detected (pipefail) and that tox's stderr is recorded in `tox-output.log` for improved debugging in the CI workflow.

* Use tox for DLCLive compatibility in CI

Replace ad-hoc DLCLive installs in the GitHub Actions job with tox-based testenvs. The workflow fixes Python to 3.12, installs required Qt/OpenGL runtime libs on Ubuntu, installs tox and tox-gh-actions, and runs tox -e matrix.tox_env. tox.ini was extended with dlclive-pypi and dlclive-github testenvs (pypi pinned and GitHub main respectively) to run the compatibility pytest, and the new envs were added to env_list to allow local and CI execution.

* tests(compat): stub torch module in conftest

Add tests/compat/conftest.py to inject a stub torch module into sys.modules when torch is not installed. This prevents ImportError during DLCLive compatibility tests so the API can be validated without requiring torch to be installed. This is a pragmatic workaround and includes a note to remove or replace it once imports are properly guarded in the package.

* Make camera validation timer cancellable

Replace QTimer.singleShot with a cancellable QTimer instance (self._camera_validation_timer) that is single-shot, connected to _validate_configured_cameras, and started with a 100ms delay. The closeEvent now stops the timer if it's still active to prevent validation from firing while the window is closing, avoiding modal QMessageBox races/crashes during tests/CI teardown.

* Add finite check and improve compatibility tests

dlc_processor: broaden validate_pose_array signature to accept source_backend as PoseBackends|str and add a check_finite option (default True) that raises on non-finite values to catch invalid pose outputs early.

tests: make torch stubbing more robust by using importlib.util.find_spec in tests/compat/conftest.py; update tests/compat/test_dlclive_package_compat.py to refine _get_signature_params typing, import Parameter, and handle constructors that accept **kwargs more gracefully (provide clearer diagnostic when expected GUI kwargs may be passed through).

tox.ini: add a [testenv:dlclive] passenv block for DLCLIVE test env vars and propagate that passenv to dlclive-pypi and dlclive-github envs so DLCLive compatibility tests pick up required environment variables.

* Refactor DLCLive signature checks in tests

Add helper functions to inspect method signatures (_signature_accepts_positional_after_self and _signature_accepts_keyword) and replace previous name-based parameter checks for DLCLive.init_inference and DLCLive.get_pose. Tests now verify that these methods accept a frame-like positional argument after self and that get_pose accepts frame_time as a keyword (or **kwargs). Also update an example in the docstring to match the new signature helper usage and improve assertion messages.

* tox.ini: skip install for dlclive-github env

Update dlclive-github testenv to run compatibility tests against GitHub main without installing the local package. Adds a comment explaining the rationale, sets package = skip, adds pytest to deps, and inserts a small diagnostic python command to print the imported dlclive file and deeplabcut-live version before running the compatibility test.

* Set pytest --confcutdir for compat tests

Add --confcutdir=tests/compat to the pytest command in tox.ini for the dlclive compatibility test. This forces pytest to stop config discovery at the tests/compat directory so the compat tests run with their intended configuration and aren't affected by higher-level pytest configs.

---------

Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>

Author: deruyter92

.github/workflows/testing-ci.yml

@@ -61,8 +61,9 @@ jobs:
             libxcb-cursor0
 
       - name: Run tests (exclude hardware) with coverage via tox
+        shell: bash -eo pipefail {0}
         run: |
-          tox -q | tee tox-output.log
+          tox -q 2>&1 | tee tox-output.log
 
 
       - name: Append Coverage Summary to Job
@@ -85,3 +86,43 @@ jobs:
           token: ${{ secrets.CODECOV_TOKEN }}
           files: ./.coverage.py312.xml
           fail_ci_if_error: false
+
+  dlclive-compat:
+    name: DLCLive Compatibility • ${{ matrix.label }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - label: pypi-1.1
+            tox_env: dlclive-pypi
+          - label: github-main
+            tox_env: dlclive-github
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - uses: actions/setup-python@v6
+        with:
+          python-version: '3.12'
+          cache: 'pip'
+
+      - name: Install Qt/OpenGL runtime deps (Ubuntu)
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y \
+            libegl1 \
+            libgl1 \
+            libopengl0 \
+            libxkbcommon-x11-0 \
+            libxcb-cursor0
+
+      - name: Install tox
+        run: |
+          python -m pip install -U pip wheel
+          python -m pip install -U tox tox-gh-actions
+
+      - name: Run DLCLive compatibility tests via tox
+        shell: bash -eo pipefail {0}
+        run: |
+          tox -e ${{ matrix.tox_env }} -q

dlclivegui/gui/main_window.py

@@ -196,14 +196,17 @@ def __init__(self, config: ApplicationSettings | None = None):
 
         # Validate cameras from loaded config (deferred to allow window to show first)
         # NOTE IMPORTANT (tests/CI): This is scheduled via a QTimer and may fire during pytest-qt teardown.
-        QTimer.singleShot(100, self._validate_configured_cameras)
+        # NOTE @C-Achard 2026-03-02: Handling this in closeEvent should help
+        self._camera_validation_timer = QTimer(self)
+        self._camera_validation_timer.setSingleShot(True)
+        self._camera_validation_timer.timeout.connect(self._validate_configured_cameras)
+        self._camera_validation_timer.start(100)
         # If validation triggers a modal QMessageBox (warning/error) while the parent window is closing,
         # it can cause errors with unpredictable timing (heap corruption / access violations).
         #
         # Mitigations for tests/CI:
         #   - Disable this timer by monkeypatching _validate_configured_cameras in GUI tests
         #   - OR monkeypatch/override _show_warning/_show_error to no-op in GUI tests (easiest)
-        #   - OR use a cancellable QTimer attribute and stop() it in closeEvent
 
     def resizeEvent(self, event):
         super().resizeEvent(event)
@@ -2021,6 +2024,8 @@ def closeEvent(self, event: QCloseEvent) -> None:  # pragma: no cover - GUI beha
         if self.multi_camera_controller.is_running():
             self.multi_camera_controller.stop(wait=True)
 
+        if hasattr(self, "_camera_validation_timer") and self._camera_validation_timer.isActive():
+            self._camera_validation_timer.stop()
         # Stop all multi-camera recorders
         self._rec_manager.stop_all()
 

dlclivegui/services/dlc_processor.py

@@ -10,12 +10,13 @@
 from collections import deque
 from contextlib import contextmanager
 from dataclasses import dataclass
+from enum import Enum, auto
 from typing import Any
 
 import numpy as np
 from PySide6.QtCore import QObject, Signal
 
-from dlclivegui.config import DLCProcessorSettings
+from dlclivegui.config import DLCProcessorSettings, ModelType
 from dlclivegui.processors.processor_utils import instantiate_from_scan
 from dlclivegui.temp import Engine  # type: ignore # TODO use main package enum when released
 
@@ -33,10 +34,79 @@
     DLCLive = None  # type: ignore[assignment]
 
 
+class PoseBackends(Enum):
+    DLC_LIVE = auto()
+
+
 @dataclass
 class PoseResult:
     pose: np.ndarray | None
     timestamp: float
+    packet: PosePacket | None = None
+
+
+@dataclass(slots=True, frozen=True)
+class PoseSource:
+    backend: PoseBackends  # e.g. "DLCLive"
+    model_type: ModelType | None = None
+
+
+@dataclass(slots=True, frozen=True)
+class PosePacket:
+    schema_version: int = 0
+    keypoints: np.ndarray | None = None
+    keypoint_names: list[str] | None = None
+    individual_ids: list[str] | None = None
+    source: PoseSource = PoseSource(backend=PoseBackends.DLC_LIVE)
+    raw: Any | None = None
+
+
+def validate_pose_array(
+    pose: Any, *, source_backend: PoseBackends | str = PoseBackends.DLC_LIVE, check_finite: bool = True
+) -> np.ndarray:
+    """
+    Validate pose output shape and dtype.
+
+    Accepted runner output shapes:
+    - (K, 3): single-animal
+    - (N, K, 3): multi-animal
+    """
+    try:
+        arr = np.asarray(pose)
+    except Exception as exc:
+        raise ValueError(
+            f"{source_backend} returned an invalid pose output format: could not convert to array ({exc})"
+        ) from exc
+
+    if arr.ndim not in (2, 3):
+        raise ValueError(
+            f"{source_backend} returned an invalid pose output format:"
+            f" expected a 2D or 3D array, got ndim={arr.ndim}, shape={arr.shape!r}"
+        )
+
+    if arr.shape[-1] != 3:
+        raise ValueError(
+            f"{source_backend} returned an invalid pose output format:"
+            f" expected last dimension size 3 (x, y, likelihood), got shape={arr.shape!r}"
+        )
+
+    if arr.ndim == 2 and arr.shape[0] <= 0:
+        raise ValueError(f"{source_backend} returned an invalid pose output format: expected at least one keypoint")
+    if arr.ndim == 3 and (arr.shape[0] <= 0 or arr.shape[1] <= 0):
+        raise ValueError(
+            f"{source_backend} returned an invalid pose output format:"
+            f" expected at least one individual and one keypoint, got shape={arr.shape!r}"
+        )
+
+    if not np.issubdtype(arr.dtype, np.number):
+        raise ValueError(
+            f"{source_backend} returned an invalid pose output format: expected numeric values, got dtype={arr.dtype}"
+        )
+
+    if check_finite and not np.isfinite(arr).all():
+        raise ValueError(f"{source_backend} returned an invalid pose output format: contains non-finite values")
+
+    return arr
 
 
 @dataclass
@@ -60,9 +130,6 @@ class ProcessorStats:
     avg_processor_overhead: float = 0.0  # Socket processor overhead
 
 
-# _SENTINEL = object()
-
-
 class DLCLiveProcessor(QObject):
     """Background pose estimation using DLCLive with queue-based threading."""
 
@@ -269,8 +336,17 @@ def _process_frame(
         # Time GPU inference (and processor overhead when present)
         with self._timed_processor() as proc_holder:
             inference_start = time.perf_counter()
-            pose = self._dlc.get_pose(frame, frame_time=timestamp)
+            raw_pose: Any = self._dlc.get_pose(frame, frame_time=timestamp)
             inference_time = time.perf_counter() - inference_start
+        pose_arr: np.ndarray = validate_pose_array(raw_pose, source_backend=PoseBackends.DLC_LIVE)
+        pose_packet = PosePacket(
+            schema_version=0,
+            keypoints=pose_arr,
+            keypoint_names=None,
+            individual_ids=None,
+            source=PoseSource(backend=PoseBackends.DLC_LIVE, model_type=self._settings.model_type),
+            raw=raw_pose,
+        )
 
         processor_overhead = 0.0
         gpu_inference_time = inference_time
@@ -280,7 +356,7 @@ def _process_frame(
 
         # Emit pose (measure signal overhead)
         signal_start = time.perf_counter()
-        self.pose_ready.emit(PoseResult(pose=pose, timestamp=timestamp))
+        self.pose_ready.emit(PoseResult(pose=pose_packet.keypoints, timestamp=timestamp, packet=pose_packet))
         signal_time = time.perf_counter() - signal_start
 
         end_ts = time.perf_counter()

pyproject.toml

@@ -115,6 +115,7 @@ markers = [
   "unit: Unit tests for individual components",
   "integration: Integration tests for component interaction",
   "functional: Functional tests for end-to-end workflows",
+  "dlclive_compat: Package/API compatibility tests against supported dlclive versions",
   "hardware: Tests that require specific hardware, notable camera backends",
   # "slow: Tests that take a long time to run",
   "gui: Tests that require GUI interaction",

tests/compat/conftest.py

@@ -0,0 +1,11 @@
+# tests/compat/conftest.py
+import importlib.util
+import sys
+import types
+
+# Stub out torch imports to avoid ImportError when torch is not installed in DLCLive package.
+# This allows testing of DLCLive API compatibility without requiring torch.
+# Ideally imports should be guarded in the package itself, but this is a pragmatic solution for now.
+# IMPORTANT NOTE: This should ideally be removed and replaced whenever possible.
+if importlib.util.find_spec("torch") is None:
+    sys.modules["torch"] = types.ModuleType("torch")