diff --git a/README.md b/README.md index 1b16d5b..eb77740 100644 --- a/README.md +++ b/README.md @@ -1,319 +1,178 @@ - -# DeepLabCut Live GUI - -A modern PySide6 GUI for running [DeepLabCut-live](https://github.com/DeepLabCut/DeepLabCut-live) experiments with real-time pose estimation. The application streams frames from industrial or consumer cameras, performs DLCLive inference, and records high-quality video with synchronized pose data. - -## Features - -### Core Functionality -- **Modern Python Stack**: Python 3.10+ compatible codebase with PySide6 interface -- **Multi-Backend Camera Support**: OpenCV, GenTL (Harvesters), Aravis, and Basler (pypylon) -- **Real-Time Pose Estimation**: Live DLCLive inference with configurable models (TensorFlow, PyTorch) -- **High-Performance Recording**: Hardware-accelerated video encoding via FFmpeg -- **Flexible Configuration**: Single JSON file for all settings with GUI editing - -### Camera Features -- **Multiple Backends**: - - OpenCV - Universal webcam support - - GenTL - Industrial cameras via Harvesters (Windows/Linux) - - Aravis - GenICam/GigE cameras (Linux/macOS) - - Basler - Basler cameras via pypylon -- **Smart Device Detection**: Automatic camera enumeration without unnecessary probing -- **Camera Controls**: Exposure time, gain, frame rate, and ROI cropping -- **Live Preview**: Real-time camera feed with rotation support (0°, 90°, 180°, 270°) - -### DLCLive Features -- **Model Support**: Only PyTorch models! (in theory also tensorflow models work) -- **Processor System**: Plugin architecture for custom pose processing -- **Auto-Recording**: Automatic video recording triggered by processor commands -- **Performance Metrics**: Real-time FPS, latency, and queue monitoring -- **Pose Visualization**: Optional overlay of detected keypoints on live feed - -### Recording Features -- **Hardware Encoding**: NVENC (NVIDIA GPU) and software codecs (libx264, libx265) -- **Configurable Quality**: CRF-based quality control -- **Multiple Formats**: MP4, AVI, MOV containers -- **Timestamp Support**: Frame-accurate timestamps for synchronization -- **Performance Monitoring**: Write FPS, buffer status, and dropped frame tracking - -### User Interface -- **Intuitive Layout**: Organized control panels with clear separation of concerns -- **Configuration Management**: Load/save settings, support for multiple configurations -- **Status Indicators**: Real-time feedback on camera, inference, and recording status -- **Bounding Box Tool**: Visual overlay for ROI definition +# DeepLabCut-Live-GUI DLC LIVE! GUI -## Installation -### Basic Installation +[![Code style:Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) +![PyPI - Package Version](https://img.shields.io/pypi/v/deeplabcut-live-gui) +![PyPI - Downloads](https://img.shields.io/pypi/dm/deeplabcut-live-gui?color=purple) +![Python versions](https://img.shields.io/pypi/pyversions/deeplabcut-live-gui) + -```bash -pip install deeplabcut-live-gui -``` +[![License](https://img.shields.io/github/license/DeepLabCut/DeepLabCut-live-GUI?label=license)](https://github.com/DeepLabCut/DeepLabCut-live-GUI/blob/main/LICENSE) +[![Image.sc forum](https://img.shields.io/badge/dynamic/json.svg?label=forum&url=https%3A%2F%2Fforum.image.sc%2Ftags%2Fdeeplabcut.json&query=%24.topic_list.tags.0.topic_count&colorB=brightgreen&&suffix=%20topics&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAAOCAYAAAAfSC3RAAABPklEQVR42m3SyyqFURTA8Y2BER0TDyExZ+aSPIKUlPIITFzKeQWXwhBlQrmFgUzMMFLKZeguBu5y+//17dP3nc5vuPdee6299gohUYYaDGOyyACq4JmQVoFujOMR77hNfOAGM+hBOQqB9TjHD36xhAa04RCuuXeKOvwHVWIKL9jCK2bRiV284QgL8MwEjAneeo9VNOEaBhzALGtoRy02cIcWhE34jj5YxgW+E5Z4iTPkMYpPLCNY3hdOYEfNbKYdmNngZ1jyEzw7h7AIb3fRTQ95OAZ6yQpGYHMMtOTgouktYwxuXsHgWLLl+4x++Kx1FJrjLTagA77bTPvYgw1rRqY56e+w7GNYsqX6JfPwi7aR+Y5SA+BXtKIRfkfJAYgj14tpOF6+I46c4/cAM3UhM3JxyKsxiOIhH0IO6SH/A1Kb1WBeUjbkAAAAAElFTkSuQmCC)](https://forum.image.sc/tags/deeplabcut) +[![Gitter](https://badges.gitter.im/DeepLabCut/community.svg)](https://gitter.im/DeepLabCut/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) +[![Twitter Follow](https://img.shields.io/twitter/follow/DeepLabCut.svg?label=DeepLabCut&style=social)](https://twitter.com/DeepLabCut) -This installs the core package with OpenCV camera support. +GUI to run [DeepLabCut-Live](https://github.com/DeepLabCut/DeepLabCut-live) on a video feed, **preview and record from one or multiple cameras**, and optionally record external timestamps and processor outputs. -### Full Installation with Optional Dependencies +## Update -```bash -# Install with gentl support -pip install deeplabcut-live-gui[gentl] -``` +> The GUI has been modernized and is now built with **PySide6 (Qt)** (replacing tkinter). The new interface supports **multi-camera preview** with a **tiled display**, PyTorch models, and features improved interactive workflows for experimental use. -### Platform-Specific Camera Backend Setup +--- -#### Windows (GenTL for Industrial Cameras) -1. Install camera vendor drivers and SDK -2. Ensure GenTL producer (.cti) files are accessible -3. Common locations: - - `C:\Program Files\The Imaging Source Europe GmbH\IC4 GenTL Driver\bin\` - - Check vendor documentation for CTI file location +## Documentation -#### Linux (Aravis for GenICam Cameras - Recommended) -NOT tested -```bash -# Ubuntu/Debian -sudo apt-get install gir1.2-aravis-0.8 python3-gi +> Full documentation is a work in progress and will be posted to DeepLabCut's official docs site when ready. -# Fedora -sudo dnf install aravis python3-gobject -``` +--- -#### macOS (Aravis) -NOT tested -```bash -brew install aravis -pip install pygobject -``` +## System requirements (quick summary) -#### Basler Cameras (All Platforms) -NOT tested -```bash -# Install Pylon SDK from Basler website -# Then install pypylon -pip install pypylon -``` +- **Python 3.10, 3.11 or 3.12** +- One inference backend (choose at least one): + - **PyTorch** *(recommended for best performance & compatibility)* + - **TensorFlow** *(for backwards compatibility with existing models; Windows installs are no longer available for Python > 3.10)* +- A supported camera backend (OpenCV webcams by default; additional backends supported) + +--- + +## Installation + +While `deeplabcut-live-gui` is available on **PyPI**, the newest PySide6-based GUI and features are currently best obtained by installing from source. -### Hardware Acceleration (Optional) +To get the **latest version**, please follow the **instructions below**. + +### 1) Clone the repository -For NVIDIA GPU encoding (highly recommended for high-resolution/high-FPS recording): ```bash -# Ensure NVIDIA drivers are installed -# FFmpeg with NVENC support will be used automatically +git clone https://github.com/DeepLabCut/DeepLabCut-live-GUI.git +cd DeepLabCut-live-GUI ``` -## Quick Start - -1. **Launch the GUI**: - ```bash - dlclivegui - ``` - -2. **Select Camera Backend**: Choose from the dropdown (opencv, gentl, aravis, basler) - -3. **Configure Camera**: Set FPS, exposure, gain, and other parameters - -4. **Start Preview**: Click "Start Preview" to begin camera streaming - -5. **Optional - Load DLC Model**: Browse to your exported DLCLive model directory - -6. **Optional - Start Inference**: Click "Start pose inference" for real-time tracking - -7. **Optional - Record Video**: Configure output path and click "Start recording" - -## Configuration - -The GUI uses a single JSON configuration file containing all experiment settings: - -```json -{ - "camera": { - "name": "Camera 0", - "index": 0, - "fps": 60.0, - "backend": "gentl", - "exposure": 10000, - "gain": 5.0, - "crop_x0": 0, - "crop_y0": 0, - "crop_x1": 0, - "crop_y1": 0, - "max_devices": 3, - "properties": {} - }, - "dlc": { - "model_path": "/path/to/exported-model", - "model_type": "pytorch", - }, - "recording": { - "enabled": true, - "directory": "~/Videos/deeplabcut-live", - "filename": "session.mp4", - "container": "mp4", - "codec": "h264_nvenc", - "crf": 23 - }, - "bbox": { - "enabled": false, - "x0": 0, - "y0": 0, - "x1": 200, - "y1": 100 - } -} -``` +### Option A — Install with `uv` -### Configuration Management +#### 2) Create & activate a new virtual environment -- **Load**: File → Load configuration… (or Ctrl+O) -- **Save**: File → Save configuration (or Ctrl+S) -- **Save As**: File → Save configuration as… (or Ctrl+Shift+S) +```bash +uv venv dlclivegui -All GUI fields are automatically synchronized with the configuration file. +# Linux/macOS: +source dlclivegui/bin/activate -## Camera Backends +# Windows (Command Prompt): +.\dlclivegui\Scripts\activate.bat -### Backend Selection Guide +# Windows (PowerShell): +.\dlclivegui\Scripts\Activate.ps1 +``` -| Backend | Platform | Use Case | Auto-Detection | -|---------|----------|----------|----------------| -| **opencv** | All | Webcams, simple USB cameras | Basic | -| **gentl** | Windows, Linux | Industrial cameras via CTI files | Yes | -| **aravis** | Linux, macOS | GenICam/GigE cameras | Yes | -| **basler** | All | Basler cameras specifically | Yes | +#### 2) Choose inference backend and install -### Backend-Specific Configuration +You may install **PyTorch** or **TensorFlow** extras (or both), but you must choose at least one to run inference. -#### OpenCV -```json -{ - "camera": { - "backend": "opencv", - "index": 0, - "fps": 30.0 - } -} -``` -**Note**: Exposure and gain controls are disabled for OpenCV backend due to limited driver support. - -#### GenTL (Harvesters) -```json -{ - "camera": { - "backend": "gentl", - "index": 0, - "fps": 60.0, - "exposure": 15000, - "gain": 8.0, - } -} +- **PyTorch (recommended):** + +```bash +uv pip install -e .[pytorch] ``` +- **TensorFlow (backwards compatibility):** + +```bash +uv pip install -e .[tf] +``` -See [Camera Backend Documentation](docs/camera_support.md) for detailed setup instructions. +--- -## DLCLive Integration +### Option B — Install with `mamba` / `conda` -### Model Types +#### 1) Create & activate a new conda environment -The GUI supports PyTorch DLCLive models: +```bash +conda create -n dlclivegui python=3.12 +conda activate dlclivegui +``` -1. **PyTorch**: PyTorch-based models (requires PyTorch installation) +#### 2) Install with your inference backend -Select the model type from the dropdown before starting inference. +- **PyTorch (recommended):** -### Processor System +```bash +pip install -e .[pytorch] +``` -The GUI includes a plugin system for custom pose processing: +- **TensorFlow:** -```python -# Example processor -class MyProcessor: - def process(self, pose, timestamp): - # Custom processing logic - x, y = pose[0, :2] # First keypoint - print(f"Position: ({x}, {y})") - def save(self): - pass +```bash +pip install -e .[tf] ``` -Place processors in `dlclivegui/processors/` and refresh to load them. +--- -See [Processor Plugin Documentation](docs/PLUGIN_SYSTEM.md) for details. +## Run the application -### Auto-Recording Feature +> [!TIP] +> For GPU/CUDA support specifics and framework compatibility, please follow the official PyTorch/TensorFlow install guidance for your OS and drivers. -Enable "Auto-record video on processor command" to automatically start/stop recording based on processor signals. Useful for event-triggered recording in behavioral experiments. +After installation, start the application with: -## Performance Optimization +```bash +dlclivegui # in conda/mamba -### High-Speed Camera Tips +# OR: +uv run dlclivegui +``` -1. **Use Hardware Encoding**: Select `h264_nvenc` codec for NVIDIA GPUs -2. **Adjust Buffer Count**: Increase buffers for GenTL/Aravis backends - ```json - "properties": {"n_buffers": 20} - ``` -3. **Optimize CRF**: Lower CRF = higher quality but larger files (default: 23) -4. **Disable Visualization**: Uncheck "Display pose predictions" during recording -5. **Crop Region**: Use cropping to reduce frame size before inference +> [!IMPORTANT] +> Activate your venv/conda environment before launching so the GUI can access installed dependencies. -### Project Structure +--- -``` -dlclivegui/ -├── __init__.py -├── gui.py # Main PySide6 application -├── config.py # Configuration dataclasses -├── camera_controller.py # Camera capture thread -├── dlc_processor.py # DLCLive inference thread -├── video_recorder.py # Video encoding thread -├── cameras/ # Camera backend modules -│ ├── base.py # Abstract base class -│ ├── factory.py # Backend registry and detection -│ ├── opencv_backend.py -│ ├── gentl_backend.py -│ ├── aravis_backend.py -│ └── basler_backend.py -└── processors/ # Pose processor plugins - ├── processor_utils.py - └── dlc_processor_socket.py -``` +## Typical workflow +The new GUI supports **one or more cameras**. -## Documentation +Typical workflow: -- [Camera Support](docs/camera_support.md) - All camera backends and setup -- [Aravis Backend](docs/aravis_backend.md) - GenICam camera setup (Linux/macOS) -- [Processor Plugins](docs/PLUGIN_SYSTEM.md) - Custom pose processing -- [Installation Guide](docs/install.md) - Detailed setup instructions -- [Timestamp Format](docs/timestamp_format.md) - Timestamp synchronization +1. **Configure Cameras** (choose backend and devices) + - Adjust camera settings (serial, exposure, ROI/cropping, etc.) +2. **Start Preview** + - Adjust visualization settings (keypoint color map, bounding boxes, etc.) +3. **Start inference** + - Choose a DeepLabCut Live model + - Choose which camera to run inference on (currently one at a time) +4. **Start recording** + - Adjust recording settings (codec, output format, etc.) + - Record video and timestamps to organized session folders -## System Requirements +> [!NOTE] +> OpenCV-compatible cameras (USB webcams, OBS virtual camera) work out of the box. +> For additional camera ecosystems (Basler, GenTL, Aravis, etc.), see the relevant documentation. + +--- -### Recommended -- Python 3.10+ -- 8 GB RAM -- NVIDIA GPU with CUDA support (for DLCLive inference and video encoding) -- USB 3.0 or GigE network (for industrial cameras) -- SSD storage (for high-speed recording) +## Current limitations -### Tested Platforms -- Windows 11 +- Pose inference runs on **one selected camera at a time** (even in multi-camera mode) +- Camera features support and availability depends on backend capabilities and hardware + - OpenCV controls for resolution/FPS are best-effort and device-driver dependent +- DeepLabCut-Live models must be exported and compatible with the chosen backend +- Performance depends on resolution, frame rate, GPU availability, and codec choice -## License +--- -This project is licensed under the GNU Lesser General Public License v3.0. See the [LICENSE](LICENSE) file for more information. +## References -## Citation +If you use this code we kindly ask you to please cite: -Cite the original DeepLabCut-live paper: -```bibtex -@article{Kane2020, - title={Real-time, low-latency closed-loop feedback using markerless posture tracking}, - author={Kane, Gary A and Lopes, Gonçalo and Saunders, Jonny L and Mathis, Alexander and Mathis, Mackenzie W}, - journal={eLife}, - year={2020}, - doi={10.7554/eLife.61909} -} -``` +- **[Kane et al., eLife 2020](https://elifesciences.org/articles/61909)** + - If preferred, see the **[Preprint](https://www.biorxiv.org/content/10.1101/2020.08.04.236422v2)** + +--- + +## Contributing / Feedback + +This project is under active development — feedback from real experimental use is highly valued. + +Please report issues, suggest features, or contribute on [GitHub](https://github.com/DeepLabCut/DeepLabCut-live-GUI).