diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 673b32c8..2378992a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -164,44 +164,14 @@ Links: - - -With `tox`, developers can run the full test suite for Python 2.7 and 3.x. -The same test suite is used in GitHub CI: +With `tox`, developers can run the full test suite with several Python versions. +The same test suite is used in GitHub CI. +To install the latest version of `tox`, in an activated Python `venv`, run: ```bash -pip3 install --user --upgrade 'py>=1.11.0' 'virtualenv<20.22' 'tox==4.5.1'; hash -r; tox +pip3 install --user --upgrade tox; hash -r; tox ``` -Explanation: - -- `tox>=4` is needed in order to fix reading the python2.7 dependencies from `pyproject.toml` -- `tox==4.5.1` is the last version not depending on `virtualenv>=20.23` (breaks Python 2.7) -- The latest versions of `tox` need `'py>=1.11.0'`. This ensures that it is at least `1.11`. -- `virtualenv-20.22` breaks using python2.7 for the `py27`, so has to be `virtualenv<20.22`. - -## Installation of all development dependencies - -Using pip-tools, you can extract the requirements and extras from `pyptoject.toml`: - -```bash -PYTHON=python3.10 -EXTRAS=.,test,mypy,pyright,pytype,tox -PFLAGS="--no-warn-conflicts" -$PYTHON -m pip install pip-tools==7.3.0 -$PYTHON -m piptools compile --extra=$EXTRAS -o - pyproject.toml | - $PYTHON -m pip install -r /dev/stdin $PFLAGS -``` - -With this, you can run most of the CI tests run by `tox` and GitHub CI also from the shell. - -You can use e.g.: `tox -e py27-test -e py3.10-covcombine-check` -The syntax is `-e py-[-factor2]`. -A few of the factors are: - -- `test`: runs `pytest` -- `cov`: runs `pytest --cov` and generates `XML` and `HTML` reports in `.tox/py-cov/logs/` -- `check`: runs `mypy` -- `fox`: runs like `cov` but then opens the `HTML` reports in Firefox - ## Recommended `tox` and `pytest` plugins for development When updating existing tests or developing new code with new test coverage, we might want to @@ -270,7 +240,7 @@ To run `podman` as your user, run these as your user: ```bash systemctl enable --now --user podman.socket # Only configures the podman socket systemctl start --user podman.socket # Start the docker-compatible unix socket -# Ubuntu only, Fedora 37 configures it already with more unqualifies search registries: +# Ubuntu only, Fedora configures unqualified search registries out-of-the-box: echo 'unqualified-search-registries = ["docker.io"]' | sudo tee -a /etc/containers/registries.conf sudo touch /etc/containers/nodocker # Quiet the docker emulation notification echo "--container-daemon-socket unix://$XDG_RUNTIME_DIR/podman/podman.sock" >>~/.actrc diff --git a/README.md b/README.md index ee12582e..2b8557c2 100644 --- a/README.md +++ b/README.md @@ -13,9 +13,6 @@ The package name is `python-libs` which is also the `rpm` package name in XenSer XCP-ng packages it as [xcp-python-libs](https://github.com/xcp-ng-rpms/xcp-python-libs) ([koji](https://koji.xcp-ng.org/packageinfo?packageID=400)). -It supports Python 2.7 and is currently in progress to get further fixes for >= 3.6. -It depends on `six`, and on Python 2.7, also `configparser` and `pyliblzma`. - ## Test-driven Development (TDD) Model Please see [CONTRIBUTING.md] for installing a local development environment. @@ -25,11 +22,8 @@ Test-driven development. The Continuous Integration Tests feature: -- Combined coverage testing of Python 2.7 and Python 3.8 code branches -- Automatic Upload of the combined coverage to CodeCov (from the GitHub Workflow) -- Checking of the combined coverage against the diff to master: Fails if changes are not covered! -- Pylint report in the GitHub Action Summary page, with Warning and Error annotatios, even in the code review. -- Check that changes don't generate pylint warnings (if warning classes which are enabled in .pylintrc) +- `pylint` report in the GitHub Action Summary page with warning and error annotations in the code review. +- Checks that changes don't generate `pylint` warnings - Static analysis using `mypy`, `pylint`, `pyright` and `pytype` This enforces that any change (besides whitespace): diff --git a/pyproject.toml b/pyproject.toml index 6a303a41..a16babb3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -55,10 +55,12 @@ test = [ "typing_extensions" ] coverage = [ + "python-libs[test]", "coverage[toml]", "diff_cover" ] mypy = [ + "python-libs[test]", "lxml", "mypy", "mypy-extensions", @@ -68,6 +70,7 @@ mypy = [ "types-toml", ] pytype = [ + "python-libs[test]", "pandas", "pytype", ] diff --git a/tox.ini b/tox.ini index 1db47099..179d613f 100644 --- a/tox.ini +++ b/tox.ini @@ -1,4 +1,5 @@ [tox] +min_version = 4.6 # Set the envlist: # Defines the environments that tox runs by default (when -e is not used) # @@ -35,7 +36,6 @@ commands = [testenv] description = Run in a {basepython} virtualenv: cov: {[cov]description} - covcombine: {[covcombine]description} covcp: Copy the generated .converage and coverage.xml to the UPLOAD_DIR dir fox: {[fox]description} lint: {[lint]description} @@ -47,27 +47,24 @@ description = Run in a {basepython} virtualenv: pytype: Run pytype for static analyis, intro: https://youtu.be/abvW0mOrDiY # checkers(mypy) need the pytest dependices as well: extras = - {check,pytype}: {[check]extras} - {cov,covcp,covcombine,fox,check,lint,test,pytype,pyright,mdreport}: {[test]extras} - {cov,covcp,covcombine,fox}: {[cov]extras} + check: {[check]extras} + pytype: {[pytype]extras} + {lint,test,pyright,mdreport}: {[test]extras} + {cov,covcp,fox}: {[cov]extras} deps = mdreport: pytest-md-report - {py27-test,py27-cov}: pyftpdlib - {py27-test,py27-cov}: pylint - {cov,covcp,covcombine,fox}: coverage[toml] - {cov,covcp,covcombine,fox}: diff-cover + {cov,covcp,fox}: coverage[toml] + {cov,covcp,fox}: diff-cover {lint,fox}: {[lint]deps} pyright: pyright - pytype: {[pytype]deps} allowlist_externals = - {cov,covcp,covcombine,fox,check,lint,test,mdreport}: echo - {cov,covcp,covcombine,fox,check,lint,test,mdreport}: sh - {cov,covcp,covcombine,fox}: cp - {covcombine,fox}: tox + {cov,covcp,fox,check,lint,test,mdreport}: echo + {cov,covcp,fox,check,lint,test,mdreport}: sh + {cov,covcp,fox}: cp check: cat fox: firefox passenv = - {pytype,covcombine,lint,test}: GITHUB_STEP_SUMMARY + {pytype,lint,test}: GITHUB_STEP_SUMMARY pytype: GITHUB_SERVER_URL pytype: GITHUB_REPOSITORY pytype: GITHUB_HEAD_REF @@ -106,20 +103,16 @@ commands = pyright: {[pyright]commands} check: {[check]commands} pytype: {[pytype]commands} - {cov,covcp,covcombine,check,fox,test,mdreport}: {[test]commands} - # covcombine shall not call [cov]commands: diff-cover shall check the combined cov: + {cov,covcp,check,fox,test,mdreport}: {[test]commands} {cov,covcp}: {[cov]commands} - {py27-test}: pylint --py3k --disable=no-absolute-import xcp/ covcp: cp -av {envlogdir}/coverage.xml {envlogdir}/coverage.lcov {envlogdir}/.coverage {envlogdir}/junit.xml {env:UPLOAD_DIR:.} - covcombine: {[covcombine]commands} - fox: {[covcombine]commands} + fox: {[cov]commands} fox: {[lint]commands} fox: {[fox]commands} [cov] description = Generate coverage html reports (incl. diff-cover) for this environment setenv = PY3_DIFFCOVER_OPTIONS=--ignore-whitespace --show-uncovered - py27: PY3_DIFFCOVER_OPTIONS= extras = coverage test commands = @@ -133,27 +126,6 @@ commands = --html-report {envlogdir}/coverage-diff.html \ {envlogdir}/coverage.xml -[covcombine] -description = Generate combined coverage reports with py27-test coverage merged -commands = - tox -e py27-test - sh -c 'export COVERAGE_FILE=$COVERAGE_FILE-combined; \ - coverage combine --keep {envlogdir}/../../py27-test/log/.coverage {envlogdir}/.coverage;\ - coverage xml -o {envlogdir}/coverage.xml;\ - coverage html -d {envlogdir}/htmlcov;\ - coverage html -d {envlogdir}/htmlcov-tests --include="tests/*"' - sh -c '\ - diff-cover --compare-branch=origin/master --ignore-staged --ignore-unstaged \ - --ignore-whitespace --show-uncovered --fail-under {env:DIFF_COVCOMBINE_MIN:100} \ - --html-report {envlogdir}/coverage-diff.html \ - --markdown-report {envlogdir}/coverage-diff.md \ - {envlogdir}/coverage.xml; EXIT_CODE=$?;echo $EXIT_CODE; \ - GITHUB_STEP_SUMMARY={env:GITHUB_STEP_SUMMARY:.git/GITHUB_STEP_SUMMARY.md}; \ - if [ -n "$GITHUB_STEP_SUMMARY" ]; then \ - mkdir -p ${GITHUB_STEP_SUMMARY%/*};sed "/title/,/\/style/d" \ - {envlogdir}/coverage-diff.html >>"$GITHUB_STEP_SUMMARY"; fi; \ - exit $EXIT_CODE' - [lint] description = Run pylint and fail on warnings remaining on lines in the diff to master deps = pylint @@ -168,7 +140,7 @@ commands = --html-report {envlogdir}/pylint-diff.html {envlogdir}/pylint.txt [fox] -description = Generate combined coverage html reports and open them in firefox +description = Generate pylint and coverage html reports and open them in firefox commands = firefox {envlogdir}/coverage-diff.html \ {envlogdir}/htmlcov/index.html \ {envlogdir}/htmlcov-tests/index.html \ @@ -183,7 +155,6 @@ commands = firefox {envlogdir}/coverage-diff.html # the local venvs will be the same as the venvs created by tox on the GitHub runners: [gh-actions] python = - 2.7: py27 3.6: py36 3.7: py37 3.8: py38 @@ -210,8 +181,9 @@ commands = pyright [pytype] -deps = pytype - pandas +extras = pytype +description = Run pytype (note: It fails to find pytest_httpserver on 3.12!) +basepython = py311 commands = # Needs python >= 3.10: Needed to parse the newer syntax for "Type2 | Type2" pytype --version