|
1 | | -# Developer Guide |
| 1 | +# Contributing Guide |
2 | 2 |
|
3 | | -## Table of Contents |
4 | | - |
5 | | -- [Contribute to DataJoint Python Documentation](#contribute-to-datajoint-python-documentation) |
6 | | -- [Setup Development Environment](#setup-development-environment) |
7 | | - - [Prerequisites](#prerequisites) |
8 | | - - [With Virtual Environment](#with-virtual-environment) |
9 | | - - [With DevContainer](#with-devcontainer) |
10 | | - - [Extra Efficiency, Optional But Recommended](#extra-efficiency-optional-but-recommended) |
11 | | - - [Pre-commit Hooks](#pre-commit-hooks) |
12 | | - - [Integration Tests](#integration-tests) |
13 | | - - [VSCode](#vscode) |
14 | | - - [Jupyter Extension](#jupyter-extension) |
15 | | - - [Debugger](#debugger) |
16 | | - - [MySQL CLI](#mysql-cli) |
17 | | - |
18 | | -## Contribute to DataJoint Python Documentation |
19 | | - |
20 | | -> Contributions to documentations are equivalently important to any code for the community, please help us to resolve any confusions in documentations. |
21 | | -
|
22 | | -[Here](https://github.com/datajoint/datajoint-python/blob/master/docs/README.md) is the instructions for contributing documentations, or you can find the same instructions at `$PROJECT_DIR/docs/README.md` in the repository. |
23 | | - |
24 | | -[Back to top](#table-of-contents) |
25 | | - |
26 | | -## Setup Development Environment |
27 | | - |
28 | | -> We have [DevContainer](https://containers.dev/) ready for contributors to develop without setting up their environment. If you are familiar with DevContainer, Docker or Github Codespace, this is the recommended development environment for you. |
29 | | -> If you have never used Docker, it might be easier for you to use a virtual environment through `conda/mamba/venv`, it is also very straightforward to set up. |
30 | | -
|
31 | | -### Prerequisites |
32 | | - |
33 | | -- Clone datajoint-python repository |
| 3 | +## Quick Start |
34 | 4 |
|
35 | 5 | ```bash |
36 | | -# If you have your SSH key set up with GitHub, you can clone using SSH |
37 | | -git clone git@github.com:datajoint/datajoint-python.git |
38 | | -# Otherwise, you can clone using HTTPS |
| 6 | +# Clone the repository |
39 | 7 | git clone https://github.com/datajoint/datajoint-python.git |
40 | | -``` |
41 | | -- If you don't use DevContainer, then either install Anaconda/[Miniconda](https://www.anaconda.com/docs/getting-started/miniconda/install)/Mamba, or just use Python's built-in `venv` module without install anything else. |
42 | | - |
43 | | -### With Virtual Environment |
| 8 | +cd datajoint-python |
44 | 9 |
|
45 | | -```bash |
46 | | -# Check if you have Python 3.10 or higher, if not please upgrade |
47 | | -python --version |
48 | | -# Create a virtual environment with venv |
| 10 | +# Create virtual environment (Python 3.10+) |
49 | 11 | python -m venv .venv |
50 | | -source .venv/bin/activate |
51 | | -pip install -e .[dev] |
| 12 | +source .venv/bin/activate # On Windows: .venv\Scripts\activate |
52 | 13 |
|
53 | | -# Or create a virtual environment with conda |
54 | | -conda create -n dj python=3.13 # any 3.10+ is fine |
55 | | -conda activate dj |
56 | | -pip install -e .[dev] |
57 | | -``` |
| 14 | +# Install with development dependencies |
| 15 | +pip install -e ".[dev]" |
58 | 16 |
|
59 | | -[Back to top](#table-of-contents) |
| 17 | +# Install pre-commit hooks |
| 18 | +pre-commit install |
60 | 19 |
|
61 | | -### With DevContainer |
| 20 | +# Run tests |
| 21 | +pytest tests |
| 22 | +``` |
62 | 23 |
|
63 | | -#### Launch Environment |
| 24 | +## Development Environment |
64 | 25 |
|
65 | | -Here are some options that provide a great developer experience: |
| 26 | +### Local Setup |
66 | 27 |
|
67 | | -- **Cloud-based IDE**: (*recommended*) |
68 | | - - Launch using [GitHub Codespaces](https://github.com/features/codespaces) using the option `Create codespace on master` in the codebase repository on your fork. |
69 | | - - Build time for a 2-Core codespace is **~6m**. This is done infrequently and cached for convenience. |
70 | | - - Start time for a 2-Core codespace is **~2m**. This will pull the built codespace from cache when you need it. |
71 | | - - *Tip*: GitHub auto names the codespace but you can rename the codespace so that it is easier to identify later. |
72 | | -- **Local IDE (VSCode - Dev Containers)**: |
73 | | - - Ensure you have [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) |
74 | | - - Ensure you have [Docker](https://docs.docker.com/get-docker/) |
75 | | - - Ensure you have [VSCode](https://code.visualstudio.com/) |
76 | | - - Install the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) |
77 | | - - `git clone` the codebase repository and open it in VSCode |
78 | | - - Use the `Dev Containers extension` to `Reopen in Container` (More info in the `Getting started` included with the extension) |
79 | | - - You will know your environment has finished loading once you see a terminal open related to `Running postStartCommand` with a final message: `Done. Press any key to close the terminal.`. |
80 | | -- **Local IDE (Docker Compose)**: |
81 | | - - Ensure you have [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) |
82 | | - - Ensure you have [Docker](https://docs.docker.com/get-docker/) |
83 | | - - `git clone` the codebase repository and open it in VSCode |
84 | | - - Issue the following command in the terminal to build and run the Docker container: `HOST_UID=$(id -u) PY_VER=3.11 DJ_VERSION=$(grep -oP '\d+\.\d+\.\d+' src/datajoint/version.py) docker compose --profile test run --rm -it djtest -- sh -c 'pip install -qe ".[dev]" && bash'` |
85 | | - - Issue the following command in the terminal to stop the Docker compose stack: `docker compose --profile test down` |
| 28 | +Requirements: |
86 | 29 |
|
87 | | -[Back to top](#table-of-contents) |
| 30 | +- Python 3.10 or higher |
| 31 | +- MySQL 8.0+ or Docker (for running tests) |
88 | 32 |
|
89 | | -## Extra Efficiency, Optional But Recommended |
| 33 | +The `[dev]` extras install all development tools: pytest, pre-commit, black, ruff, and documentation builders. |
90 | 34 |
|
91 | | -### Pre-commit Hooks |
| 35 | +### Using Docker for Database |
92 | 36 |
|
93 | | -We recommend using [pre-commit](https://pre-commit.com/) to automatically run linters and formatters on your code before committing. |
94 | | -To set up pre-commit, run the following command in your terminal: |
| 37 | +Tests require a MySQL database. Start one with Docker: |
95 | 38 |
|
96 | 39 | ```bash |
97 | | -pip install pre-commit |
98 | | -pre-commit install |
| 40 | +docker compose up -d db |
99 | 41 | ``` |
100 | 42 |
|
101 | | -You can manually run pre-commit on all files with the following command: |
| 43 | +Configure connection (or set environment variables): |
102 | 44 |
|
103 | 45 | ```bash |
104 | | -pre-commit run --all-files |
| 46 | +export DJ_HOST=localhost |
| 47 | +export DJ_USER=root |
| 48 | +export DJ_PASS=password |
105 | 49 | ``` |
106 | | -This will run all the linters and formatters specified in the `.pre-commit-config.yaml` file. If all check passed, you can commit your code. Otherwise, you need to fix the failed checks and run the command again. |
107 | 50 |
|
108 | | -> Pre-commit will automatically run the linters and formatters on all staged files before committing. However, if your code doesn't follow the linters and formatters, the commit will fail. |
109 | | -> Some hooks will automatically fix your problem, and add the fixed files as git's `unstaged` files, you just need to add them(`git add .`) to git's `staged` files and commit again. |
110 | | -> Some hooks will not automatically fix your problem, so you need to check the pre-commit failed log to fix them manually and include the update to your `staged` files and commit again. |
| 51 | +### Alternative: GitHub Codespaces |
111 | 52 |
|
112 | | -If you really don't want to use pre-commit, or if you don't like it, you can uninstall it with the following command: |
| 53 | +For a pre-configured environment, use [GitHub Codespaces](https://github.com/features/codespaces): |
113 | 54 |
|
114 | | -```bash |
115 | | -pre-commit uninstall |
116 | | -``` |
| 55 | +1. Fork the repository |
| 56 | +2. Click "Create codespace on master" |
| 57 | +3. Wait for environment to build (~6 minutes first time, ~2 minutes from cache) |
117 | 58 |
|
118 | | -But when you issue a pull request, the same linter and formatter check will run against your contribution, you are going to have the same failure as well. So without pre-commit, you need to **manually run these linters and formatters before committing your code**: |
| 59 | +## Code Quality |
119 | 60 |
|
120 | | -- Syntax tests |
| 61 | +### Pre-commit Hooks |
121 | 62 |
|
122 | | -The following will verify that there are no syntax errors. |
| 63 | +Pre-commit runs automatically on `git commit`. To run manually: |
123 | 64 |
|
124 | | -``` |
125 | | -flake8 datajoint --count --select=E9,F63,F7,F82 --show-source --statistics |
| 65 | +```bash |
| 66 | +pre-commit run --all-files |
126 | 67 | ``` |
127 | 68 |
|
128 | | -- Style tests |
| 69 | +Hooks include: |
129 | 70 |
|
130 | | -The following will verify that there are no code styling errors. |
| 71 | +- **ruff** — Linting and import sorting |
| 72 | +- **black** — Code formatting |
| 73 | +- **mypy** — Type checking (optional) |
131 | 74 |
|
132 | | -``` |
133 | | -flake8 --ignore=E203,E722,W503 datajoint --count --max-complexity=62 --max-line-length=127 --statistics |
134 | | -``` |
135 | | - |
136 | | -The following will ensure the codebase has been formatted with [black](https://black.readthedocs.io/en/stable/). |
| 75 | +### Running Tests |
137 | 76 |
|
138 | | -``` |
139 | | -black datajoint --check -v --diff |
140 | | -``` |
| 77 | +```bash |
| 78 | +# Full test suite with coverage |
| 79 | +pytest -sv --cov=datajoint tests |
141 | 80 |
|
142 | | -The following will ensure the test suite has been formatted with [black](https://black.readthedocs.io/en/stable/). |
| 81 | +# Single test file |
| 82 | +pytest tests/test_connection.py |
143 | 83 |
|
| 84 | +# Single test function |
| 85 | +pytest tests/test_connection.py::test_dj_conn -v |
144 | 86 | ``` |
145 | | -black tests --check -v --diff |
146 | | -``` |
147 | | - |
148 | | -[Back to top](#table-of-contents) |
149 | | - |
150 | | -### Integration Tests |
151 | | - |
152 | | -The following will verify there are no regression errors by running our test suite of unit and integration tests. |
153 | | - |
154 | | -- Entire test suite: |
155 | | - ``` |
156 | | - pytest -sv --cov-report term-missing --cov=datajoint tests |
157 | | - ``` |
158 | | - |
159 | | -- A single functional test: |
160 | | - ``` |
161 | | - pytest -sv tests/test_connection.py::test_dj_conn |
162 | | - ``` |
163 | | -- A single class test: |
164 | | - ``` |
165 | | - pytest -sv tests/test_aggr_regressions.py::TestIssue558 |
166 | | - ``` |
167 | 87 |
|
168 | | -[Back to top](#table-of-contents) |
| 88 | +## Submitting Changes |
169 | 89 |
|
170 | | -### VSCode |
| 90 | +1. Create a feature branch from `master` |
| 91 | +2. Make your changes |
| 92 | +3. Ensure tests pass and pre-commit is clean |
| 93 | +4. Submit a pull request |
171 | 94 |
|
172 | | -#### Jupyter Extension |
| 95 | +PRs trigger CI checks automatically. All checks must pass before merge. |
173 | 96 |
|
174 | | -Be sure to go through this documentation if you are new to [Running Jupyter Notebooks with VSCode](https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_create-or-open-a-jupyter-notebook). |
| 97 | +## Documentation |
175 | 98 |
|
176 | | -#### Debugger |
177 | | - |
178 | | -[VSCode Debugger](https://code.visualstudio.com/docs/editor/debugging) is a powerful tool that can really accelerate fixes. |
179 | | - |
180 | | -Try it as follows: |
181 | | - |
182 | | -- Create a python script of your choice |
183 | | -- `import datajoint` (This will use the current state of the source) |
184 | | -- Add breakpoints by adding red dots next to line numbers |
185 | | -- Select the `Run and Debug` tab |
186 | | -- Start by clicking the button `Run and Debug` |
187 | | - |
188 | | -[Back to top](#table-of-contents) |
189 | | - |
190 | | -### MySQL CLI |
191 | | - |
192 | | -> Installation instruction is in [here](https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-install.html) |
193 | | -
|
194 | | -It is often useful in development to connect to DataJoint's relational database backend directly using the MySQL CLI. |
195 | | - |
196 | | -Connect as follows to the database running within your developer environment: |
197 | | - |
198 | | -``` |
199 | | -mysql -hdb -uroot -ppassword |
200 | | -``` |
| 99 | +Docstrings use NumPy style. See [DOCSTRING_STYLE.md](https://github.com/datajoint/datajoint-python/blob/master/DOCSTRING_STYLE.md) for guidelines. |
201 | 100 |
|
202 | | -[Back to top](#table-of-contents) |
| 101 | +User documentation is maintained at [docs.datajoint.com](https://docs.datajoint.com). |
0 commit comments