chore: Bump Ruff to 0.15.0

[Dev-iL]

Tooling: update Ruff + apply ruff format + fix most existing issues

Changes

• Update Ruff version in dependencies and hooks to 0.15.0
• Fix issues reported by ruff check .

How I tested this

Untested beyond running hooks successfully.

Notes

• No LLM was used for this PR. All non-autofix changes were done manually.
• Most changes are to notebooks, which were previously not linted.
• TODO: add more ruff inspections, in particular ones with autofixes.

Remaining Ruff violation statistics

$ ruff check . --select=ALL --statistics --preview
warning: `incorrect-blank-line-before-class` (D203) and `no-blank-line-before-class` (D211) are incompatible. Ignoring `incorrect-blank-line-before-class`.
warning: `multi-line-summary-first-line` (D212) and `multi-line-summary-second-line` (D213) are incompatible. Ignoring `multi-line-summary-second-line`.
2545    UP006           [*] non-pep585-annotation
2006    DOC201          [ ] docstring-missing-returns
1945    D103            [ ] undocumented-public-function
1869    COM812          [*] missing-trailing-comma
1715    ANN201          [ ] missing-return-type-undocumented-public-function
1616    E501            [ ] line-too-long
1466    D401            [ ] non-imperative-mood
1153    D400            [ ] missing-trailing-period
1138    D415            [ ] missing-terminal-punctuation
 993    CPY001          [ ] missing-copyright-notice
 974    ANN001          [ ] missing-type-function-argument
 834    D205            [-] missing-blank-line-after-summary
 789    ERA001          [ ] commented-out-code
 788    D102            [ ] undocumented-public-method
 718    UP045           [-] non-pep604-annotation-optional
 643    D100            [ ] undocumented-public-module
 583    UP035           [-] deprecated-import
 537    T201            [ ] print
 531    UP007           [-] non-pep604-annotation-union
 489    S101            [ ] assert
 422    RUF067          [ ] non-empty-init-module
 360    D101            [ ] undocumented-public-class
 354    C408            [ ] unnecessary-collection-call
 346    TRY003          [ ] raise-vanilla-args
 338    PLR2004         [ ] magic-value-comparison
 333    ANN401          [ ] any-type
 302    INP001          [ ] implicit-namespace-package
 277    ANN202          [ ] missing-return-type-private-function
 272    ARG002          [ ] unused-method-argument
 270    G004            [-] logging-f-string
 265    EM102           [ ] f-string-in-exception
 260    TD002           [ ] missing-todo-author
 259    TD003           [ ] missing-todo-link
 255    FIX002          [ ] line-contains-todo
 236    ANN204          [ ] missing-return-type-special-method
 225    RUF013          [ ] implicit-optional
 223    RET504          [ ] unnecessary-assign
 208    ARG001          [ ] unused-function-argument
 201    ANN003          [ ] missing-type-kwargs
 200    TD004           [ ] missing-todo-colon
 195    SLF001          [ ] private-member-access
 190    DOC501          [ ] docstring-missing-exception
 183    FBT001          [ ] boolean-type-hint-positional-argument
 183    N806            [ ] non-lowercase-variable-in-function
 157    D404            [ ] docstring-starts-with-this
 150    D212            [*] multi-line-summary-first-line
 144    PLR0913         [ ] too-many-arguments
 143    EM101           [ ] raw-string-in-exception
 136    D104            [ ] undocumented-public-package
 136    FBT002          [ ] boolean-default-value-positional-argument
 134    PLC0415         [ ] import-outside-top-level
 133    RUF052          [ ] used-dummy-variable
 131    E402            [ ] module-import-not-at-top-of-file
 129    PLR6301         [ ] no-self-use
 127    D202            [*] blank-line-after-function
 127    PIE790          [*] unnecessary-placeholder
 116    RUF059          [ ] unused-unpacked-variable
 115    F821            [ ] undefined-name
 110    N803            [ ] invalid-argument-name
 109    N802            [ ] invalid-function-name
 109    PTH123          [*] builtin-open
 105    PLR0917         [ ] too-many-positional-arguments
  93    PTH118          [ ] os-path-join
  80    D107            [ ] undocumented-public-init
  80    PT006           [ ] pytest-parametrize-names-wrong-type
  79    D209            [*] new-line-after-last-paragraph
  78    PLW1514         [ ] unspecified-encoding
  74    N801            [ ] invalid-class-name
  68    FURB118         [ ] reimplemented-operator
  65    C901            [ ] complex-structure
  62    D403            [*] first-word-uncapitalized
  60    RET505          [*] superfluous-else-return
  57    RUF012          [ ] mutable-class-default
  56    ANN002          [ ] missing-type-args
  56    PLR6201         [ ] literal-membership
  53    PT011           [ ] pytest-raises-too-broad
  51    UP008           [*] super-call-with-parameters
  49    D413            [*] missing-blank-line-after-last-section
  46    E721            [ ] type-comparison
  46    I001            [*] unsorted-imports
  45    D105            [ ] undocumented-magic-method
  45    D200            [ ] unnecessary-multiline-docstring
  43    BLE001          [ ] blind-except
  43    D106            [ ] undocumented-public-nested-class
  40    A002            [ ] builtin-argument-shadowing
  40    T203            [ ] p-print
  38    PGH004          [ ] blanket-noqa
  38    PTH110          [*] os-path-exists
  37    DOC402          [ ] docstring-missing-yields
  37    E266            [*] multiple-leading-hashes-for-block-comment
  36    PLR0912         [ ] too-many-branches
  35    DTZ005          [ ] call-datetime-now-without-tzinfo
  35    S113            [ ] request-without-timeout
  32    UP015           [*] redundant-open-modes
  31    FBT003          [ ] boolean-positional-value-in-call
  31    PERF401         [ ] manual-list-comprehension
  30    PLW2901         [ ] redefined-loop-name
  30    TRY201          [ ] verbose-raise
  29    RUF005          [ ] collection-literal-concatenation
  25    FURB110         [-] if-exp-instead-of-or-operator
  24    TRY400          [ ] error-instead-of-exception
  23    RUF029          [ ] unused-async
  23    SIM102          [ ] collapsible-if
  22    ANN205          [ ] missing-return-type-static-method
  22    PERF102         [ ] incorrect-dict-iterator
  22    RUF100          [*] unused-noqa
  21    C416            [ ] unnecessary-comprehension
  21    FURB103         [-] write-whole-file
  21    UP028           [ ] yield-in-for-loop
  20    A001            [ ] builtin-variable-shadowing
  20    FURB101         [-] read-whole-file
  20    SIM108          [ ] if-else-block-instead-of-if-exp
  18    PGH003          [ ] blanket-type-ignore
  18    TRY300          [ ] try-consider-else
  17    D411            [*] no-blank-line-before-section
  16    C419            [ ] unnecessary-comprehension-in-call
  16    PT001           [*] pytest-fixture-incorrect-parentheses-style
  16    S311            [ ] suspicious-non-cryptographic-random-usage
  16    SIM118          [-] in-dict-keys
  15    ANN206          [ ] missing-return-type-class-method
  15    C420            [*] unnecessary-dict-comprehension-for-iterable
  15    D301            [ ] escape-sequence-in-docstring
  15    PLC2701         [ ] import-private-name
  15    PT012           [ ] pytest-raises-with-multiple-statements
  15    PTH208          [ ] os-listdir
  15    S403            [ ] suspicious-pickle-import
  15    SIM114          [*] if-with-same-arms
  14    ICN001          [ ] unconventional-import-alias
  14    PT022           [*] pytest-useless-yield-fixture
  14    PTH103          [*] os-makedirs
  14    RUF036          [ ] none-not-at-end-of-union
  14    S301            [ ] suspicious-pickle-usage
  13    DTZ007          [ ] call-datetime-strptime-without-zone
  13    PIE804          [*] unnecessary-dict-kwargs
  13    S607            [ ] start-process-with-partial-path
  13    UP031           [ ] printf-string-formatting
  12    PIE810          [ ] multiple-starts-ends-with
  12    PLC1901         [ ] compare-to-empty-string
  12    PLW0603         [ ] global-statement
  12    RET503          [ ] implicit-return
  12    RET506          [*] superfluous-else-raise
  12    SIM115          [ ] open-file-with-context-handler
  12    SIM117          [-] multiple-with-statements
  11    A004            [ ] builtin-import-shadowing
  11    DTZ001          [ ] call-datetime-without-tzinfo
  11    PD015           [ ] pandas-use-of-pd-merge
  11    PLR0915         [ ] too-many-statements
  11    PLR6104         [ ] non-augmented-assignment
  11    PYI041          [*] redundant-numeric-union
  11    RET507          [*] superfluous-else-continue
  11    RUF010          [*] explicit-f-string-type-conversion
  11    SIM300          [*] yoda-conditions
  11    TRY002          [ ] raise-vanilla-class
  10    FAST002         [ ] fast-api-non-annotated-dependency
  10    N818            [ ] error-suffix-on-exception-name
  10    NPY002          [ ] numpy-legacy-random
  10    PLR1714         [ ] repeated-equality-comparison
  10    S603            [ ] subprocess-without-shell-equals-true
   9    D410            [*] no-blank-line-after-section
   9    PD002           [ ] pandas-use-of-inplace-argument
   9    PYI063          [ ] pep484-style-positional-only-parameter
   9    RUF002          [ ] ambiguous-unicode-character-docstring
   9    RUF015          [ ] unnecessary-iterable-allocation-for-first-element
   9    S104            [ ] hardcoded-bind-all-interfaces
   9    UP032           [*] f-string
   8    AIR311          [ ] airflow3-suggested-update
   8    FURB171         [ ] single-item-membership-test
   8    PD011           [ ] pandas-use-of-dot-values
   8    PERF203         [ ] try-except-in-loop
   8    S404            [ ] suspicious-subprocess-import
   8    TRY004          [ ] type-check-without-type-error
   8    UP037           [*] quoted-annotation
   7    C403            [ ] unnecessary-list-comprehension-set
   7    DOC102          [ ] docstring-extraneous-parameter
   7    PTH113          [*] os-path-isfile
   7    PTH119          [ ] os-path-basename
   7    PTH120          [ ] os-path-dirname
   7    PYI016          [*] duplicate-union-member
   7    RUF031          [*] incorrectly-parenthesized-tuple-in-subscript
   7    SIM103          [ ] needless-bool
   7    SIM113          [ ] enumerate-for-loop
   6    B008            [ ] function-call-in-default-argument
   6    EXE002          [ ] shebang-missing-executable-file
   6    FIX004          [ ] line-contains-hack
   6    N812            [ ] lowercase-imported-as-non-lowercase
   6    N999            [ ] invalid-module-name
   6    PIE808          [*] unnecessary-range-start
   6    PLC2801         [ ] unnecessary-dunder-call
   6    PT014           [ ] pytest-duplicate-parametrize-test-cases
   6    PT018           [ ] pytest-composite-assertion
   6    PTH109          [-] os-getcwd
   6    RET502          [*] implicit-return-value
   6    S307            [ ] suspicious-eval-usage
   6    SIM110          [ ] reimplemented-builtin
   6    SIM401          [ ] if-else-block-instead-of-dict-get
   6    UP034           [*] extraneous-parentheses
   6    W605            [*] invalid-escape-sequence
   5    B007            [ ] unused-loop-control-variable
   5    C401            [ ] unnecessary-generator-set
   5    C414            [ ] unnecessary-double-cast-or-process
   5    D300            [*] triple-single-quotes
   5    D417            [ ] undocumented-param
   5    D419            [ ] empty-docstring
   5    FIX001          [ ] line-contains-fixme
   5    PIE800          [*] unnecessary-spread
   5    PLR0402         [*] manual-from-import
   5    PLR0911         [ ] too-many-return-statements
   5    PLR0914         [ ] too-many-locals
   5    PLR1702         [ ] too-many-nested-blocks
   5    PLR2044         [*] empty-comment
   5    PLW0108         [ ] unnecessary-lambda
   5    PLW1510         [ ] subprocess-run-without-check
   5    PLW1641         [ ] eq-without-hash
   5    PYI051          [ ] redundant-literal-union
   5    S102            [ ] exec-builtin
   5    T100            [ ] debugger
   5    TD001           [ ] invalid-todo-tag
   4    C405            [ ] unnecessary-literal-set
   4    C417            [ ] unnecessary-map
   4    D406            [*] missing-new-line-after-section-name
   4    D407            [*] missing-dashed-underline-after-section
   4    DJ001           [ ] django-nullable-model-string-field
   4    EXE001          [ ] shebang-not-executable
   4    G001            [ ] logging-string-format
   4    N811            [ ] constant-imported-as-non-constant
   4    PLC0207         [*] missing-maxsplit-arg
   4    PLR1704         [ ] redefined-argument-from-local
   4    PTH102          [*] os-mkdir
   4    RSE102          [-] unnecessary-paren-on-raise-exception
   4    RUF045          [ ] implicit-class-var-in-dataclass
   4    RUF047          [*] needless-else
   4    S110            [ ] try-except-pass
   4    S324            [ ] hashlib-insecure-hash-function
   4    SIM105          [ ] suppressible-exception
   4    SIM201          [ ] negate-equal-op
   4    SIM210          [ ] if-expr-with-true-false
   4    W391            [*] too-many-newlines-at-end-of-file
   3    AIR301          [ ] airflow3-removal
   3    ARG005          [ ] unused-lambda-argument
   3    B905            [ ] zip-without-explicit-strict
   3    C402            [ ] unnecessary-generator-dict
   3    D405            [*] non-capitalized-section-name
   3    D412            [*] blank-lines-between-header-and-content
   3    F401            [*] unused-import
   3    FURB142         [*] for-loop-set-mutations
   3    FURB152         [*] math-constant
   3    LOG004          [ ] log-exception-outside-except-handler
   3    PERF402         [ ] manual-list-copy
   3    PLC0105         [ ] type-name-incorrect-variance
   3    PLC0414         [ ] useless-import-alias
   3    PLR5501         [*] collapsible-else-if
   3    PT019           [ ] pytest-fixture-param-without-value
   3    PTH111          [ ] os-path-expanduser
   3    PYI030          [*] unnecessary-literal-union
   3    RUF022          [*] unsorted-dunder-all
   3    RUF027          [ ] missing-f-string-syntax
   3    S310            [ ] suspicious-url-open-usage
   3    SIM101          [ ] duplicate-isinstance-call
   3    TRY301          [ ] raise-within-try
   2    AIR002          [ ] airflow-dag-no-schedule-argument
   2    D210            [ ] surrounding-whitespace
   2    FURB154         [*] repeated-global
   2    N813            [ ] camelcase-imported-as-lowercase
   2    NPY201          [*] numpy2-deprecation
   2    PLR0904         [ ] too-many-public-methods
   2    PLR1711         [*] useless-return
   2    PLW0177         [ ] nan-comparison
   2    PLW0602         [ ] global-variable-not-assigned
   2    PT003           [ ] pytest-extraneous-scope-function
   2    PTH107          [*] os-remove
   2    PTH206          [ ] os-sep-split
   2    PTH207          [ ] glob
   2    RET501          [*] unnecessary-return-none
   2    RUF006          [ ] asyncio-dangling-task
   2    RUF017          [ ] quadratic-list-summation
   2    RUF021          [*] parenthesize-chained-operators
   2    S105            [ ] hardcoded-password-string
   2    S106            [ ] hardcoded-password-func-arg
   2    S608            [ ] hardcoded-sql-expression
   2    S701            [ ] jinja2-autoescape-false
   2    TC001           [ ] typing-only-first-party-import
   2    TRY203          [ ] useless-try-except
   2    TRY401          [ ] verbose-log-message
   2    UP004           [*] useless-object-inheritance
   2    UP022           [ ] replace-stdout-stderr
   2    UP036           [ ] outdated-version-block
   1    ARG003          [ ] unused-class-method-argument
   1    ASYNC240        [ ] blocking-path-method-in-async-function
   1    B020            [ ] loop-variable-overrides-iterator
   1    B021            [ ] f-string-docstring
   1    C409            [ ] unnecessary-literal-within-tuple-call
   1    C411            [ ] unnecessary-list-call
   1    C413            [*] unnecessary-call-around-sorted
   1    D402            [ ] signature-in-docstring
   1    DOC202          [ ] docstring-extraneous-returns
   1    FAST001         [ ] fast-api-redundant-response-model
   1    FAST003         [ ] fast-api-unused-path-parameter
   1    FURB113         [ ] repeated-append
   1    FURB145         [*] slice-copy
   1    FURB189         [ ] subclass-builtin
   1    ISC004          [ ] implicit-string-concatenation-in-collection-literal
   1    N805            [ ] invalid-first-argument-name-for-method
   1    N816            [ ] mixed-case-variable-in-global-scope
   1    PD003           [ ] pandas-use-of-dot-is-null
   1    PD010           [ ] pandas-use-of-dot-pivot-or-unstack
   1    PERF403         [ ] manual-dict-comprehension
   1    PLC0132         [ ] type-param-name-mismatch
   1    PLC0206         [ ] dict-index-missing-items
   1    PLR0916         [ ] too-many-boolean-expressions
   1    PLR1730         [*] if-stmt-min-max
   1    PLR1733         [*] unnecessary-dict-index-lookup
   1    PLW1508         [ ] invalid-envvar-default
   1    PLW3201         [ ] bad-dunder-method-name
   1    PT013           [ ] pytest-incorrect-pytest-import
   1    PTH101          [*] os-chmod
   1    PTH122          [ ] os-path-splitext
   1    PTH202          [*] os-path-getsize
   1    PTH204          [*] os-path-getmtime
   1    PYI006          [ ] bad-version-info-comparison
   1    PYI024          [ ] collections-named-tuple
   1    PYI025          [ ] unaliased-collections-abc-set-import
   1    PYI032          [*] any-eq-ne-annotation
   1    RUF001          [ ] ambiguous-unicode-character-string
   1    RUF019          [*] unnecessary-key-check
   1    RUF038          [ ] redundant-bool-literal
   1    S103            [ ] bad-file-permissions
   1    SIM109          [ ] compare-with-tuple
   1    SIM910          [*] dict-get-with-none-default
   1    TD005           [ ] missing-todo-description
   1    TD006           [*] invalid-todo-capitalization
   1    UP030           [ ] format-literals
Found 35358 errors.
[*] 7204 fixable with the `--fix` option (6763 hidden fixes can be enabled with the `--unsafe-fixes` option).

Checklist

• PR has an informative and human-readable title (this will be pulled into the release notes)
• Changes are limited to a single goal (no scope creep)
• Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future TODOs are captured in comments
• Project documentation has been updated if adding/changing functionality.

[Dev-iL]

@skrawcz Multiple tests are failing due to incompatibilities with new dependency versions (e.g. protobuf, pandas v3).

1. Are there scheduled CI runs that surface such issues in isolation?
2. What is the preferred strategy for resolving those? Setting an upper limit on dependency versions (probably undesired)? Conditional code (try except )? Anything else?

[skrawcz]

@skrawcz Multiple tests are failing due to incompatibilities with new dependency versions (e.g. protobuf, pandas v3).

1. Are there scheduled CI runs that surface such issues in isolation?

we could run scheduled CIs -- that would catch these sooner.

2. What is the preferred strategy for resolving those? Setting an upper limit on dependency versions (probably undesired)? Conditional code (try except )? Anything else?

Hard question to answer. My main concern is backwards compatibility. I think libraries should not try to pin. But we need to have an opinion of what we're supporting. E.g. do we support all pandas versions? what's our minimum? But we don't want to support too much due to carrying cost. So I think we just need feedback on what target versions we want to support.