Skip to content

Update dependencies #499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
victorlin wants to merge 3 commits into
base: master
Choose a base branch
from
+18 −42
Open

Update dependencies #499

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
16d4fb7
Use fsspec[s3] extra, declare dependency on boto3
victorlin Dec 30, 2025
d0aa9f7
Pin wrapt to version 1
victorlin Dec 30, 2025
a4fea46
Bump wrapt to version 2
victorlin Dec 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,12 @@ development source code and as such may not be routinely kept up to date.

## Bug fixes

* Updated the s3fs dependency to avoid versions starting with 2025.12.0. This
maintains a dependency on `boto3` which is used for various commands.
([#495](https://github.com/nextstrain/cli/pull/495))
* Updated the dependency list to explicitly include `boto3`, used for various
commands.
([#499](https://github.com/nextstrain/cli/pull/499))
* Updated code to support the latest version of `wrapt`, used to show
`--dry-run` outputs.
([#499](https://github.com/nextstrain/cli/pull/499))

# 10.4.1 (14 October 2025)

Expand Down
9 changes: 6 additions & 3 deletions doc/changes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,12 @@ development source code and as such may not be routinely kept up to date.
(v-next-bug-fixes)=
### Bug fixes

* Updated the s3fs dependency to avoid versions starting with 2025.12.0. This
maintains a dependency on `boto3` which is used for various commands.
([#495](https://github.com/nextstrain/cli/pull/495))
* Updated the dependency list to explicitly include `boto3`, used for various
commands.
([#499](https://github.com/nextstrain/cli/pull/499))
* Updated code to support the latest version of `wrapt`, used to show
`--dry-run` outputs.
([#499](https://github.com/nextstrain/cli/pull/499))

(v10-4-1)=
## 10.4.1 (14 October 2025)
Expand Down
4 changes: 2 additions & 2 deletions nextstrain/cli/console.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from contextlib import contextmanager, ExitStack, redirect_stdout, redirect_stderr
from functools import wraps
from typing import Callable, TextIO
from wrapt import ObjectProxy
from wrapt import BaseObjectProxy # pyright: ignore[reportAttributeAccessIssue]


def auto_dry_run_indicator(getter: Callable[..., bool] = lambda opts, *args, **kwargs: opts.dry_run):
Expand Down Expand Up @@ -83,7 +83,7 @@ def dry_run_indicator(dry_run: bool = False):
yield dry_run


class LinePrefixer(ObjectProxy): # pyright: ignore[reportUntypedBaseClass]
class LinePrefixer(BaseObjectProxy): # pyright: ignore[reportUntypedBaseClass]
"""
Add *prefix* to every line written to *file*.

Expand Down
38 changes: 4 additions & 34 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ def find_namespaced_packages(namespace):
python_requires = '>=3.8',

install_requires = [
"boto3 ==1.*",
"certifi",
"docutils",
"fasteners",
Expand All @@ -102,43 +103,12 @@ def find_namespaced_packages(namespace):
"requests",
"typing_extensions >=3.7.4",
"wcmatch >=6.0",
"wrapt",

# We use fsspec's S3 support, which has a runtime dep on s3fs. s3fs
# itself requires aiobotocore, which in turn requires very specific
# versions of botocore (because aiobotocore is a giant monkey-patch).
#
# We also use boto3, which also requires botocore, usually with minimum
# versions closely matching the boto3 version (they're released in near
# lock step).
#
# If we declare a dep on boto3 directly, this leads to conflicts during
# dependency resolution when a newer boto3 (from our declaration here)
# requires a newer botocore than is supported by s3fs → aiobotocore's
# declarations.
#
# Resolve the issue by using a specially-provided package extra from
# s3fs (first introduced with 2021.4.0, removed in 2025.12.0) which
# causes them to declare an explicit dependency on aiobotocore's
# specially-provided package extra on boto3 so that dependency resolver
# can figure it out properly.
#
# Note that the upper limit is not future-proof and will likely cause
# issues down the road. There may be a better combination to use here,
# but that needs extra digging.
#
# More background:
# <https://github.com/dask/s3fs/issues/357>
# <https://github.com/nextstrain/cli/issues/133>
# <https://github.com/fsspec/s3fs/issues/994>
#
# What a mess.
#
"wrapt >=2.0.0",

# Avoiding 2023.9.1 due to change in `auto_mkdir` parameter in
# https://github.com/fsspec/filesystem_spec/pull/1358 that causes the
# error described in https://github.com/fsspec/s3fs/issues/790
"fsspec !=2023.9.1",
"s3fs[boto3] >=2021.04.0, !=2023.9.1, <2025.12.0",
"fsspec[s3] !=2023.9.1",
Copy link
Contributor

@joverlee521 joverlee521 Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested some aws-batch commands with the CI build and seeing a warning that s3fs is very old

$ curl -fsSL --proto '=https' https://nextstrain.org/cli/installer/mac | bash -s ci-build/20610155097
$ nextstrain build --aws-batch --attach c79d54ab-ae3c-4ca7-9e68-d2d90b3e1c35 --download logs/** --no-logs .
Attaching to Nextstrain AWS Batch Job ID: c79d54ab-ae3c-4ca7-9e68-d2d90b3e1c35
Job is SUCCEEDED
Job SUCCEEDED after 395.4 minutes (exited 0)
Downloading select files modified by job to /Users/joverlee/Repos/nextstrain/tb
  - logs/**
fsspec.registry:301: UserWarning: Your installed version of s3fs is very old and known to cause
severe performance issues, see also https://github.com/dask/dask/issues/10276

To fix, you should specify a lower version bound on s3fs, or
update the current installation.

Looking at the CI logs, this is installing s3fs-0.4.2. I wonder if declaring the s3fs depedency separately would help?

Copy link
Member Author

@victorlin victorlin Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dang, good catch. The fsspec[s3] extra doesn't impose any version constraints on s3fs. I wonder why it's resolving to an older version instead of latest. I'll restore the s3fs minimum version constraint to see if that fixes things.

Copy link
Contributor

@joverlee521 joverlee521 Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wondered if Python 3.8 was causing the downgrade, but even the tests with Python 3.13 are installing older versions of s3fs...

Looks like aiobotocore relaxed botocore + boto3 dependencies starting with 3.0.0, but latest release of s3fs (2025.12.0) still pins aiobotocore>=2.5.4,<3.0.0 (source). Maybe we just need to wait on the new release of s3fs?

Copy link
Member Author

@victorlin victorlin Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I hope a new version of s3fs that supports aiobotocore 3.x will help. The current pin is incompatible with versions of boto3 and wrapt resolved initially based on our project's dependency list. This first round of version resolutions is prioritized by pip, so it backtracks on s3fs until it finds version 0.4.2 which pip thinks can be installed with aiobotocore 3.x since it predates any aiobotocore version restriction.

Reverting a4fea46 and tightly pinning boto3 ~=1.41.0 fixes things, but this is the exact problem that's being described in the lengthy comment that we're trying to remove.

Copy link
Member Author

@victorlin victorlin Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Continued in #496 (comment)


# From 2.0.0 onwards, urllib3 is better typed, but not usable (given
# our dep tree) on 3.8 and 3.9 so we use types-urllib3 there (see
Expand Down
Loading