Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
b4ee412
feat(remotes) Add remote repository import package
tony Feb 1, 2026
8262e9e
feat(cli/import) Add vcspull import command
tony Feb 1, 2026
9c809ed
feat(cli[__init__]) Register import command
tony Feb 1, 2026
d77ed64
test(remotes) Add tests for remote importers
tony Feb 1, 2026
cf2a460
test(cli[import]) Add tests for import command
tony Feb 1, 2026
407e8b6
fix(test[test_log]) Add import_repos logger to expected list
tony Feb 1, 2026
dcd9b76
docs(CHANGES) Note vcspull import command (#510)
tony Feb 1, 2026
2a1dcad
fix(remotes[gitea]) Use proper URL parsing for host detection
tony Feb 1, 2026
4689286
fix(test[test_gitea]) Use exact URL assertion
tony Feb 1, 2026
dd00cb5
refactor(cli/import) Remove unused SERVICES_REQUIRING_URL constant
tony Feb 1, 2026
376db3c
fix(cli/import[topic_list]) Filter empty strings from topic list
tony Feb 1, 2026
fa06089
fix(cli/import[output]) Gate log messages behind human output mode
tony Feb 1, 2026
1b88935
fix(remotes/github[pagination]) Fix early termination in pagination
tony Feb 1, 2026
9e2f82d
fix(remotes/gitlab[pagination]) Fix early termination in pagination
tony Feb 1, 2026
4549d81
fix(remotes/gitea[pagination]) Fix early termination in pagination
tony Feb 1, 2026
9af17ec
test(remotes[pagination]) Add xfail regression test for pagination du…
tony Feb 1, 2026
aa28913
fix(remotes/github[pagination]) Use consistent per_page for pagination
tony Feb 1, 2026
d2d2209
fix(remotes/gitea[pagination]) Use consistent limit for pagination
tony Feb 1, 2026
78d7443
test(remotes[pagination]) Remove xfail markers from pagination tests
tony Feb 1, 2026
613da15
fix(remotes/gitlab[pagination]) Use consistent per_page for pagination
tony Feb 1, 2026
0633eda
docs(cli/import) Document GitLab subgroup support in help text
tony Feb 1, 2026
c78bcd9
test(remotes/gitlab) Add tests for GitLab subgroup support
tony Feb 1, 2026
898e987
fix(test/remotes) Clear env tokens in authentication tests
tony Feb 1, 2026
e592d30
feat(cli/import) Show help when import called without required args
tony Feb 1, 2026
1325962
fix(remotes/codecommit) Apply filter_repo before yielding repos
tony Feb 1, 2026
034dcaf
feat(remotes[ssh_url]) Add ssh_url field to RemoteRepo
tony Feb 8, 2026
12e5597
feat(cli/import[--https]) Default to SSH URLs, add --https flag
tony Feb 8, 2026
5f0a061
docs(CHANGES) Document SSH-default clone URLs for vcspull import
tony Feb 8, 2026
7af8a48
fix(remotes/base[query_string]) Use urllib.parse.urlencode for query …
tony Feb 8, 2026
43ab6da
fix(remotes[url_encoding]) URL-encode options.target in endpoint paths
tony Feb 8, 2026
6343506
style(remotes/base[imports]) Use namespace import for dataclasses
tony Feb 8, 2026
8cbfb63
fix(cli/import[config_validation]) Validate config structure and narr…
tony Feb 8, 2026
e3f50dc
refactor(shadowing) Rename t loop variable to avoid shadowing typing
tony Feb 8, 2026
6171aec
docs(remotes/base[RemoteRepo]) Fix docstring type for topics field
tony Feb 8, 2026
83dafe9
docs(remotes[terminology]) Fix Scraping → Import terminology in docst…
tony Feb 8, 2026
fa78f5c
refactor(test[MockHTTPResponse]) Deduplicate MockHTTPResponse across …
tony Feb 8, 2026
e8d68f3
fix(remotes/codecommit[region]) Extract region from clone URL when no…
tony Feb 8, 2026
0ef722c
docs(cli/import[--token]) Add security note to --token help text
tony Feb 8, 2026
1598bdd
test(remotes/codecommit) Add CodeCommitImporter unit tests
tony Feb 8, 2026
569fe79
fix(remotes/codecommit[output]) Force --output json in AWS CLI commands
tony Feb 8, 2026
309aa1c
fix(remotes/gitlab[archived]) Fix archived parameter semantics
tony Feb 8, 2026
d77ffb3
fix(remotes[topics]) Guard against null topics in API responses
tony Feb 8, 2026
92ab260
fix(test[codecommit]) Mock subprocess in _get_importer CodeCommit tests
tony Feb 8, 2026
a3cf645
fix(cli/import[gitlab_groups]) Preserve namespace by default
tony Feb 14, 2026
fc6b152
test(cli/import[gitlab]) Cover nested and flatten group variants
tony Feb 14, 2026
3ef1fe2
docs(CHANGES): Clarify GitLab subgroup workspace mapping
tony Feb 14, 2026
1a40dc6
test(remotes/gitlab[mock_data]) Use realistic namespace fields in sub…
tony Feb 14, 2026
dbb603f
test(remotes/gitlab[conftest]) Add full_path to fixture namespace mock
tony Feb 14, 2026
22e8305
fix(remotes/github[_log_rate_limit]) Guard int() cast on rate-limit h…
tony Feb 14, 2026
875c67f
fix(remotes/base[_handle_http_error]) Coerce message to str before .l…
tony Feb 14, 2026
de9236e
fix(cli/import[language-warning]) Warn when --language used with GitL…
tony Feb 14, 2026
5bf0e7e
fix(cli/import[input-eof]) Handle EOFError and non-TTY stdin in confi…
tony Feb 14, 2026
9f82b26
fix(remotes/github[_parse_repo]) Use .get() for required API fields
tony Feb 14, 2026
cb2870f
fix(remotes/gitlab[_fetch_search]) Add archived=false param to search
tony Feb 14, 2026
c297d05
style(cli[__init__]) Use t.overload instead of from typing import ove…
tony Feb 14, 2026
21cccac
fix(cli/import[codeberg-url]) Pass --url to Codeberg importer
tony Feb 14, 2026
4ce711f
fix(remotes/base[ImportOptions]) Validate limit >= 1 in __post_init__
tony Feb 14, 2026
640757f
fix(config[atomic-write]) Use temp-file-then-rename for config saves
tony Feb 14, 2026
183ef6b
chore(lint) Fix ruff and mypy issues from review fixes
tony Feb 14, 2026
37af8d5
fix(cli/_output[finalize]) Emit empty JSON array when buffer is empty
tony Feb 14, 2026
5a001e5
fix(config[find_home_config_files]) Respect filetype filter parameter
tony Feb 14, 2026
ee0c301
fix(cli/import[config-resolve]) Catch MultipleConfigWarning
tony Feb 14, 2026
0036f8e
fix(cli/import[ImportOptions]) Catch ValueError for invalid limit
tony Feb 14, 2026
9a48f56
style(cli/import[colors]) Route color output through Colors helper
tony Feb 14, 2026
6febb65
fix(cli[import-exit-code]) Return non-zero exit code on import errors
tony Feb 14, 2026
4c23506
fix(remotes[codecommit-pagination]) Handle nextToken in list-reposito…
tony Feb 14, 2026
a009a04
test(cli/import[resolve-config]) Add .yml extension test fixture
tony Feb 14, 2026
3d205b9
fix(cli/import[unsupported-filters]) Warn about --topics/--min-stars …
tony Feb 14, 2026
fc85ee8
fix(remotes[github-search-cap]) Warn when search exceeds 1000-result …
tony Feb 14, 2026
cdf3b05
docs(doctests[remotes,output]): Add doctests to feasible public methods
tony Feb 14, 2026
ab77872
test(cli/import[config-write]) Replace yaml.dump with save_config_yaml
tony Feb 14, 2026
221e4b1
test(mocks[documentation]): Add comments to all monkeypatch.setattr c…
tony Feb 14, 2026
ceb5628
docs(CHANGES[import]): Restructure changelog for v1.55.x release
tony Feb 14, 2026
d43aac9
docs(cli/import[pages]): Add CLI and API documentation for vcspull im…
tony Feb 15, 2026
1f42879
docs(import[docstrings]): Add auth details to importer docstrings
tony Feb 15, 2026
5d5e141
docs(cli/import[auth]): Expand authentication guide with per-service …
tony Feb 15, 2026
c0a054d
docs(cli/import[help]): Protect --flatten-groups with backticks in de…
tony Feb 15, 2026
443f647
refactor(cli/import[common]): Extract shared parsers and _run_import …
tony Feb 15, 2026
1df8882
refactor(cli/import[services]): Add per-service subparser modules
tony Feb 15, 2026
4ab197d
refactor(cli/import[dispatch]): Wire import_cmd subparsers into CLI d…
tony Feb 15, 2026
f09ea85
refactor(cli/import[cleanup]): Remove old import_repos.py
tony Feb 15, 2026
e148ad2
style(cli/_formatter): Add import-command flags to colorization sets
tony Feb 15, 2026
f020615
test(cli/import[subparsers]): Update tests for per-service subparser …
tony Feb 15, 2026
89dcd2f
docs(cli/import): Update API docs for import_cmd package
tony Feb 15, 2026
c686b1a
docs(cli/import[pages]): Split import docs into per-service nested pages
tony Feb 15, 2026
bddfd99
fix(cli/import[traversal]) Reject subgroup paths that escape workspace
tony Feb 15, 2026
df942bf
fix(config[atomic_write]) Preserve file permissions on atomic write
tony Feb 15, 2026
dcfed92
fix(cli/import[json]) Accept JSON config files in import command
tony Feb 15, 2026
160f8b8
fix(remotes/github[ghe]) Normalize GitHub Enterprise URL to /api/v3
tony Feb 15, 2026
c20a281
fix(cli/import[exit_code]) Return non-zero exit for non-mapping works…
tony Feb 15, 2026
97100b0
fix(cli/import[exit_code]) Return non-zero exit for non-interactive a…
tony Feb 15, 2026
dbe3627
fix(remotes[null_owner]) Guard against null owner/namespace in API re…
tony Feb 15, 2026
b1db6c0
fix(cli/import[codecommit]) Correct help text from prefix to substrin…
tony Feb 15, 2026
4494572
fix(remotes[http_client]) Use urlsplit/urlunsplit for query param mer…
tony Feb 15, 2026
4c81a32
fix(remotes[http_client]) Warn when auth token sent over non-HTTPS
tony Feb 15, 2026
6091c51
test(config[json]) Add tests for save_config_json and JSON import path
tony Feb 15, 2026
d6208b0
fix(remotes/github[search]) Cap search pagination at 1000 results
tony Feb 15, 2026
b10c8fd
fix(remotes/codecommit[timeout]) Add subprocess timeout to AWS CLI calls
tony Feb 15, 2026
d395ddf
refactor(cli/import[logging]) Remove unused logger from 5 CLI handler…
tony Feb 15, 2026
5086392
fix(cli/import[config_load]) Use json.loads for JSON config files
tony Feb 15, 2026
b0e2e3a
fix(test[mypy]) Use MockHTTPResponse in URL merge test
tony Feb 15, 2026
b49033e
refactor(cli/import[config_load]) Use ConfigReader._from_file() for c…
tony Feb 15, 2026
d6f87ed
docs(cli/import[auth]) Split multi-command code blocks in service pages
tony Feb 15, 2026
741cac6
docs(CHANGES[import]) Add labels between SSH and HTTPS examples
tony Feb 15, 2026
a786511
docs(README[import]) Add vcspull import section and mention
tony Feb 15, 2026
76944a1
docs(CLAUDE[code-blocks]) Prefer longform flags and split multi-flag …
tony Feb 15, 2026
596be9f
docs(cli[flags]) Use longform flags and split multi-flag commands
tony Feb 15, 2026
92c468c
chore(scripts[gitlab]) Remove legacy generate_gitlab scripts
tony Feb 15, 2026
90bbd7d
docs(configuration[generation]) Replace script listings with import r…
tony Feb 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,24 @@ $ vcspull search django
$ vcspull search "name:flask"
```

**Prefer longform flags** — use `--workspace` not `-w`, `--file` not `-f`.

**Split multi-flag commands** — when a command has 2+ flags/options, place each on its own `\`-continuation line, indented by 4 spaces.

Good:

```console
$ vcspull import gh my-org \
--mode org \
--workspace ~/code/
```

Bad:

```console
$ vcspull import gh my-org --mode org -w ~/code/
```

## Debugging Tips

When stuck in debugging loops:
Expand Down
119 changes: 119 additions & 0 deletions CHANGES
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,125 @@ $ uvx --from 'vcspull' --prerelease allow vcspull
_Notes on upcoming releases will be added here_
<!-- END PLACEHOLDER - ADD NEW CHANGELOG ENTRIES BELOW THIS LINE -->

### New features

#### New command: `vcspull import` (#510)

Import repositories from GitHub, GitLab, Codeberg/Gitea/Forgejo, and AWS
CodeCommit directly into your vcspull configuration.

Import a user's repositories:

```console
$ vcspull import github torvalds \
--workspace ~/repos/linux \
--mode user
```

Import an organization's repositories:

```console
$ vcspull import github django \
--workspace ~/study/python \
--mode org
```

Search and import repositories:

```console
$ vcspull import github "machine learning" \
--workspace ~/ml-repos \
--mode search \
--min-stars 1000
```

Use with self-hosted GitLab:

```console
$ vcspull import gitlab myuser \
--workspace ~/work \
--url https://gitlab.company.com
```

Import from AWS CodeCommit:

```console
$ vcspull import codecommit \
--workspace ~/work/aws \
--region us-east-1
```

Preview without writing (dry run):

```console
$ vcspull import codeberg user \
--workspace ~/oss \
--dry-run
```

**Key features:**

- Service aliases: `gh`, `gl`, `cb`, `cc`, `aws`
- Filtering: `--language`, `--topics`, `--min-stars`, `--archived`, `--forks`
- Output modes: human-readable (default), `--json`, `--ndjson`
- Interactive confirmation before writing; use `--yes`/`-y` to skip
- Repositories already in the config are detected and skipped
- Non-zero exit code on errors (for CI/automation)
- No new dependencies (uses stdlib `urllib` for HTTP)

#### `vcspull import`: SSH clone URLs by default (#510)

Clone URLs default to SSH. Use `--https` to get HTTPS URLs instead:

SSH (default):

```console
$ vcspull import github torvalds \
--workspace ~/repos/linux \
--mode user
```

Use `--https` for HTTPS clone URLs:

```console
$ vcspull import github torvalds \
--workspace ~/repos/linux \
--mode user \
--https
```

#### `vcspull import`: GitLab subgroups map to workspace roots (#510)

For GitLab organization/group imports, subgroup namespaces are preserved
under the workspace root by default:

```console
$ vcspull import gitlab vcs-python-group-test \
--workspace ~/projects/python \
--mode org
```

This writes repositories into workspace sections like:

- `~/projects/python/`
- `~/projects/python/<subgroup>/`
- `~/projects/python/<subgroup>/<subsubgroup>/`

Use `--flatten-groups` to collapse subgroup repositories into a single
workspace root:

```console
$ vcspull import gitlab vcs-python-group-test \
--workspace ~/projects/python \
--mode org \
--flatten-groups
```

### Bug fixes

- Config writes now use atomic temp-file-then-rename to prevent data loss
during interrupted writes (#510)

### Tests

- Fix `pytest-asyncio` deprecation warning in isolated `pytester` runs by
Expand Down
32 changes: 28 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ You can test the unpublished version of vcspull before its released.
## Configuration

Add your repos to `~/.vcspull.yaml`. You can edit the file by hand or let
`vcspull add` or `vcspull discover` create entries for you.
`vcspull add`, `vcspull discover`, or `vcspull import` create entries for you.

```yaml
~/code/:
Expand Down Expand Up @@ -119,10 +119,32 @@ $ vcspull discover ~/code --recursive
```

The scan shows each repository before import unless you opt into `--yes`. Add
`-w ~/code/` to pin the resulting workspace root or `-f` to write somewhere other
`--workspace ~/code/` to pin the resulting workspace root or `-f/--file` to write somewhere other
than the default `~/.vcspull.yaml`. Duplicate workspace roots are merged by
default; include `--no-merge` to keep them separate while you review the log.

### Import from remote services

Pull repository lists from GitHub, GitLab, Codeberg, Gitea, Forgejo, or AWS
CodeCommit directly into your configuration:

```console
$ vcspull import github myuser \
--workspace ~/code/ \
--mode user
```

```console
$ vcspull import gitlab my-group \
--workspace ~/work/ \
--mode org
```

Use `--dry-run` to preview changes, `--https` for HTTPS clone URLs, and
`--language`/`--topics`/`--min-stars` to filter results. See the
[import documentation](https://vcspull.git-pull.com/cli/import/) for all
supported services and options.

### Inspect configured repositories

List what vcspull already knows about without mutating anything:
Expand Down Expand Up @@ -164,7 +186,9 @@ After importing or editing by hand, run the formatter to tidy up keys, merge
duplicate workspace sections, and keep entries sorted:

```console
$ vcspull fmt -f ~/.vcspull.yaml --write
$ vcspull fmt \
--file ~/.vcspull.yaml \
--write
```

Use `vcspull fmt --all --write` to format every YAML file that vcspull can
Expand Down Expand Up @@ -205,7 +229,7 @@ or svn project with a git dependency:
Clone / update repos via config file:

```console
$ vcspull sync -f external_deps.yaml '*'
$ vcspull sync --file external_deps.yaml '*'
```

See the [Quickstart](https://vcspull.git-pull.com/quickstart.html) for
Expand Down
15 changes: 15 additions & 0 deletions docs/api/cli/import.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# vcspull import - `vcspull.cli.import_cmd`

```{eval-rst}
.. automodule:: vcspull.cli.import_cmd
:members:
:show-inheritance:
:undoc-members:
```

```{eval-rst}
.. automodule:: vcspull.cli.import_cmd._common
:members:
:show-inheritance:
:undoc-members:
```
1 change: 1 addition & 0 deletions docs/api/cli/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

sync
add
import
discover
list
search
Expand Down
25 changes: 16 additions & 9 deletions docs/cli/add.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@ merges duplicate workspace roots by default, and prompts before writing unless
you pass `--yes`.

```{note}
This command replaces the manual import functionality from `vcspull import`.
For bulk scanning of existing repositories, see {ref}`cli-discover`.
This command replaces the old `vcspull import <name> <url>` from v1.36--v1.39.
For bulk scanning of local repositories, see {ref}`cli-discover`.
For bulk import from remote services (GitHub, GitLab, etc.), see {ref}`cli-import`.
```

## Command
Expand Down Expand Up @@ -97,7 +98,8 @@ vcspull searches for configuration files in this order:
Specify a file explicitly with `-f/--file`:

```console
$ vcspull add ~/study/python/pytest-docker -f ~/configs/python.yaml
$ vcspull add ~/study/python/pytest-docker \
--file ~/configs/python.yaml
```

## Handling duplicates
Expand All @@ -114,22 +116,27 @@ a summary of the merge. Prefer to inspect duplicates yourself? Add
2. Run `vcspull list` to verify the new entry (see {ref}`cli-list`).
3. Run `vcspull sync` to clone or update the working tree (see {ref}`cli-sync`).

## Migration from vcspull import
## Migration from the old vcspull import

If you previously used `vcspull import <name> <url>`, switch to the path-first
workflow:
The `vcspull import <name> <url>` command from v1.36--v1.39 has been replaced
by `vcspull add`:

```diff
- $ vcspull import flask https://github.com/pallets/flask.git -c ~/.vcspull.yaml
+ $ vcspull add ~/code/flask --url https://github.com/pallets/flask.git -f ~/.vcspull.yaml
+ $ vcspull add ~/code/flask --url https://github.com/pallets/flask.git --file ~/.vcspull.yaml
```

Key differences:

- `vcspull add` now derives the name from the filesystem unless you pass
`--name`.
- `vcspull add` derives the name from the filesystem unless you pass `--name`.
- The parent directory becomes the workspace automatically; use `--workspace`
to override.
- Use `--url` to record a remote when the checkout does not have one.

```{note}
Starting with v1.55, `vcspull import` is a *different* command that bulk-imports
repositories from remote services (GitHub, GitLab, etc.). See {ref}`cli-import`
for details.
```

[pip vcs url]: https://pip.pypa.io/en/stable/topics/vcs-support/
10 changes: 6 additions & 4 deletions docs/cli/discover.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,9 @@ $ vcspull discover ~ --recursive --workspace-root ~/code/ --yes
Specify a custom config file with `-f/--file`:

```console
$ vcspull discover ~/company --recursive -f ~/company/.vcspull.yaml
$ vcspull discover ~/company \
--recursive \
--file ~/company/.vcspull.yaml
```

If the config file doesn't exist, it will be created.
Expand Down Expand Up @@ -195,7 +197,7 @@ Scan to specific config:
$ vcspull discover ~/company/repos \
--recursive \
--yes \
-f ~/company/.vcspull.yaml
--file ~/company/.vcspull.yaml
```

## After discovering repositories
Expand Down Expand Up @@ -229,7 +231,7 @@ If you previously used `vcspull import --scan`:

```diff
- $ vcspull import --scan ~/code --recursive -c ~/.vcspull.yaml --yes
+ $ vcspull discover ~/code --recursive -f ~/.vcspull.yaml --yes
+ $ vcspull discover ~/code --recursive --file ~/.vcspull.yaml --yes
```

Changes:
Expand Down Expand Up @@ -273,7 +275,7 @@ $ vcspull discover ~/projects --recursive --yes
```console
$ vcspull discover ~/company \
--recursive \
-f ~/company/.vcspull.yaml \
--file ~/company/.vcspull.yaml \
--workspace-root ~/work/ \
--yes
```
Expand Down
18 changes: 4 additions & 14 deletions docs/cli/fmt.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,22 +59,12 @@ Run the formatter in dry-run mode first to preview the adjustments:
$ vcspull fmt --file ~/.vcspull.yaml
```

Then add `--write` (or `-w`) to persist them back to disk:
Then add `--write` to persist them back to disk:

```console
$ vcspull fmt --file ~/.vcspull.yaml --write
```

Short form for preview:

```console
$ vcspull fmt -f ~/.vcspull.yaml
```

Short form to apply:

```console
$ vcspull fmt -f ~/.vcspull.yaml -w
$ vcspull fmt \
--file ~/.vcspull.yaml \
--write
```

Use `--all` to iterate over the default search locations: the current working
Expand Down
34 changes: 34 additions & 0 deletions docs/cli/import/codeberg.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
(cli-import-codeberg)=

# vcspull import codeberg

Import repositories from Codeberg.

## Command

```{eval-rst}
.. argparse::
:module: vcspull.cli
:func: create_parser
:prog: vcspull
:path: import codeberg
```

## Authentication

- **Env vars**: `CODEBERG_TOKEN` (primary), `GITEA_TOKEN` (fallback)
- **Token type**: API token
- **Scope**: no scopes needed for public repos; token required for private repos
- **Create at**: <https://codeberg.org/user/settings/applications>

Set the token:

```console
$ export CODEBERG_TOKEN=...
```

Then import:

```console
$ vcspull import codeberg myuser --workspace ~/code/
```
Loading