feat: update repository guidelines for improved Python code quality and Rust-Python integration

Author: kosiew

AGENTS.md

@@ -1,102 +1,81 @@
-# Repository instructions for datafusion-python
-
-## Style and Linting
-- Python formatting and linting is handled by **ruff**.
-- Follow [PEP 8](https://www.python.org/dev/peps/pep-0008/) and these key
-  ruff rules:
-  - Use double quotes for string literals
-  - Use explicit relative imports such as `from .module import Class`
-  - Limit lines to 88 characters
-  - Provide type hints for parameters and return values
-  - Avoid unused imports
-  - Prefer f-strings over other string formatting
-  - Avoid `else` blocks after `return`
-  - Use `isinstance()` instead of direct type comparison
-  - Keep docstrings concise in Google format
-  - Assign exception messages to variables before raising
-- All modules, classes, and functions should include docstrings.
-- Tests must use descriptive function names, pytest style assertions, and be
-  grouped in classes when appropriate.
-- Rust code must pass `cargo fmt`, `cargo clippy`, and `cargo tomlfmt`.
-- Install the pre-commit hooks with `pre-commit install` and run them with
-  `pre-commit run --files <file1> <file2>` or `pre-commit run --all-files`.
-
-Install the required Rust components before running pre-commit:
+# Repository Guidelines for datafusion-python
+
+Our goal is to deliver robust, maintainable, and user-friendly Python bindings over the DataFusion Rust core. Focus on high-quality solutions first—linting and formatting come later as a final check.
+
+---
+
+## 1. Solution-First Approach
+
+* **Clarify Requirements:** Understand the use case and expected behavior before coding. Ask questions if something is unclear.
+* **Design Thoughtfully:** Sketch function signatures, data flow, and API ergonomics. Strive for simplicity and consistency with Python conventions.
+* **Demonstrate Usage:** Include concise code examples in docstrings or README updates to illustrate functionality.
+
+## 2. Python Code Quality & Best Practices
+
+### Idiomatic Python
+
+* Follow [PEP 8](https://www.python.org/dev/peps/pep-0008/) for naming, spacing, and structure.
+* Use **double quotes** for string literals and **f-strings** for interpolation.
+* Favor explicit relative imports (`from .module import Class`).
+* Limit lines to **88 characters**. Break long expressions with parentheses.
+
+### Type Hints & Docstrings
+
+* Provide type hints for all public functions and methods.
+* Use Google-style docstrings. Include brief descriptions, parameter types, return types, and examples.
+* Keep docstrings concise and focused on “why” and “how” to use the API.
+
+### Readability & Maintainability
+
+* Write small, single-purpose functions (≈50 lines or fewer).
+* Break complex logic into well-named helper functions.
+* Avoid unnecessary abstractions; follow patterns established in the codebase.
+* Use meaningful variable and function names—no `temp`, `foo`, or one-letter names except in short comprehensions.
+
+## 3. Rust-Python Integration
+
+Classes and functions with Rust-backed implementations must stay in sync:
+
+1. **Update Rust First:** Implement and test changes in the Rust core before touching Python bindings.
+2. **Test Rust Behavior:** Verify new Rust functionality with unit and integration tests.
+3. **Sync Python Bindings:** Adjust `pyo3` bindings or wrapper functions to reflect the Rust API.
+4. **Python Tests:** Add or update pytest tests to cover the new or changed behavior.
+
+## 4. Testing & Validation
+
+* **Unit Tests:** Use pytest with descriptive test names and style assertions. Group related tests in `Test*` classes when it improves organization.
+* **Integration Tests:** Validate end-to-end workflows, especially for CLI commands or data-processing pipelines.
+* **Continuous Testing:** Ensure tests pass reliably in CI. Leverage `pytest -v` for verbosity.
+
+## 5. Documentation & Examples
+
+* **README:** Update usage examples and installation instructions for new features.
+* **Docstring Examples:** Embed minimal examples illustrating typical use cases.
+* **Docs Directory:** For extended tutorials or guides, add Markdown files under `docs/` and link them in the TOC.
+
+## 6. Collaboration & Review
+
+* **Pull Requests:** Summarize changes, rationale, and instructions for testing.
+* **Code Reviews:** Focus feedback on clarity, correctness, and design consistency.
+* **Discussions:** Open issues for major API decisions or design proposals.
+
+## 7. Optional Final Checks
+
+Run these after your solution is complete and tests are passing:
 
 ```bash
+# Python formatting & linting (ruff)
+pre-commit install
+pre-commit run --all-files
+# or manually:
+./ci/scripts/python_lint.sh
+
+# Rust formatting & linting (optional for bindings changes)
 rustup component add rustfmt clippy
+taplo format --check
+./ci/scripts/rust_fmt.sh
+./ci/scripts/rust_clippy.sh
+./ci/scripts/rust_toml_fmt.sh
 ```
 
-If installation is blocked by a proxy, see the [offline installation guide](https://rust-lang.github.io/rustup/installation/other.html).
-
-- The same checks can be run manually using the scripts in `ci/scripts`:
-  - `./ci/scripts/python_lint.sh`
-  - `./ci/scripts/rust_fmt.sh`
-  - `./ci/scripts/rust_clippy.sh`
-  - `./ci/scripts/rust_toml_fmt.sh`
-
-## Rust Code Style
-
-When generating Rust code for DataFusion, follow these guidelines:
-
-1. Do not add unnecessary parentheses around `if` conditions:
-   - Correct: `if some_condition`
-   - Incorrect: `if (some_condition)`
-
-2. Do not use `expect()` or `panic!()` except in tests - use proper error handling with `Result` types
-
-3. Follow the standard Rust style conventions from rustfmt
-
-## Rust-Python Integration
-
-**Important:** Some classes in this repository have their core implementations in Rust with Python bindings. For any changes related to these classes in Python:
-
-1. **Always update the Rust implementation first** before making corresponding changes to the Python bindings
-2. Ensure the Rust changes are properly tested in Rust before updating Python code
-3. Update Python bindings to reflect the new Rust API
-4. Add or update Python tests to cover the new functionality
-
-This ensures consistency between the Rust core and Python interface, and prevents binding mismatches.
-
-## Code Organization
-- Keep functions focused and under about 50 lines.
-- Break complex tasks into well-named helper functions and reuse existing
-  helpers when possible.
-- Prefer adding parameters to existing functions rather than creating new
-  versions with similar behavior.
-- Avoid unnecessary abstractions and follow established patterns in the
-  codebase.
-
-## Comments
-- Add meaningful comments for complex logic and avoid obvious comments.
-- Start inline comments with a capital letter.
-
-## Running Tests
-1. Ensure submodules are initialized:
-   ```bash
-   git submodule update --init
-   ```
-2. Create a development environment and install dependencies:
-   ```bash
-   uv sync --dev --no-install-package datafusion
-   ```
-3. Build the Python extension:
-   ```bash
-   uv run --no-project maturin develop --uv
-   ```
-4. Execute the test suite:
-   ```bash
-   uv run --no-project pytest -v .
-   ```
-
-## Building Documentation
-- Documentation dependencies can be installed with the `docs` group:
-  ```bash
-  uv sync --dev --group docs --no-install-package datafusion
-  ```
-- Build the docs with:
-  ```bash
-  uv run --no-project maturin develop --uv
-  uv run --no-project docs/build.sh
-  ```
-- The generated HTML appears under `docs/build/html`.
+Ensure these checks pass before merging, but never sacrifice clarity or correctness for formatting. 🚀