[BUG] NPM v7 uses SSH instead of an explicit HTTPS for GitHub repos

[uhop]

Current Behavior:

When I use a git repository via an HTTP link NPM "takes liberties" with it, which breaks my build:

$ npm init -y
Wrote to /Users/eugene.lazutkin/Work/temp/package.json:

{
  "name": "temp",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


$ npm i --save https://github.com/uhop/stream-chain.git

added 1 package, and audited 2 packages in 3s

found 0 vulnerabilities

It produces package-lock.json:

{
  "name": "temp",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "version": "1.0.0",
      "license": "ISC",
      "dependencies": {
        "stream-chain": "github:uhop/stream-chain"
      }
    },
    "node_modules/stream-chain": {
      "version": "2.2.4",
      "resolved": "git+ssh://git@github.com/uhop/stream-chain.git#459f5a1708c138b6e0abaae4cf103c3488e1e78e",
      "license": "BSD-3-Clause"
    }
  },
  "dependencies": {
    "stream-chain": {
      "version": "git+ssh://git@github.com/uhop/stream-chain.git#459f5a1708c138b6e0abaae4cf103c3488e1e78e",
      "from": "stream-chain@github:uhop/stream-chain"
    }
  }
}

Note that https://github.com/uhop/stream-chain.git was replaced with github:uhop/stream-chain, which is probably OK in this case. But other two links (?) are rewritten from https://github.com/uhop/stream-chain.git to git+ssh://git@github.com/uhop/stream-chain.git, which is clearly bad.

The problem is that a build bot we use in similar situations can access private git repositories using HTTP, but not SSH for security reasons. It fails on an authentication. Rewriting https://github.com/uhop/stream-chain.git to git+ssh://git@github.com/uhop/stream-chain.git is not acceptable for that reasons.

The fix is relatively minor yet unpleasant: we have to replace npm ci with npm i, which takes more time and introduced instabilities with other dependencies.

Expected Behavior:

When running npm ci it should use the original URL with the HTTP authentication instead of SSH.

Steps To Reproduce:

See the description and do the same steps using git repositories (github only?) as dependencies.

Environment:

OS: Mac
Node: 15.7.0
NPM: 7.4.3

[ljharb]

Can you try with npm v7.5.2? I think this has already been fixed.

[uhop]

Just tried with exactly same results.

$ npm --version
7.5.2

package-lock.json is absolutely identical.

[uhop]

I think there is no error in the code.

Just to point it out explicitly:

• The dependency is set as https://github.com/uhop/stream-chain.git — note https:// part of it.
• In package-lock.json it is mentioned three times.
  • Two if them as git+ssh://git@github.com/uhop/stream-chain.git#459f5a1708c138b6e0abaae4cf103c3488e1e78e.
  • Note git+ssh://git@github.com part.
  • It definitely tries to access the repository using ssh rather than https protocol and fails.
    • Why is it important?
      • Those protocols have a different way to authenticate a user.
      • Our build environment is not set up for ssh, yet we can use https.

I don't know if the bug is specific to GitHub: adding https://github.com/uhop/stream-chain.git was transformed into github:uhop/stream-chain (please peruse my bug report above), and who knows what protocol it assumes. But not all repositories support ssh. We clearly have a bug on our hands.

I hope I explained the problem better now. Please do not hesitate to ask, if something is left unclear.

[uhop]

It came to my attention that this is a documented behavior that breaks private repositories and environments without ssh access (from https://blog.npmjs.org/post/626173315965468672/npm-v7-series-beta-release-and-semver-major):

Git dependencies on known git hosts (GitHub, BitBucket, etc.) will always attempt to fetch package contents from the relevant tarball CDNs if possible, falling back to git+ssh for private packages. resolved value in package-lock.json will always reflect the git+ssh url value. Saved value in package.json dependencies will always reflect the canonical shorthand value.

Clearly this is not correct and should be fixed.

[plaa]

This is a breaking change in npm v7 and prevents anyone using even public packages without ssh keys.

We have made a fork of an npm package and host it as a public repo. We refer to it using a github: link. In v6 this github link was populated to package-lock.json as well, which worked fine. In v7 it is converted into git+ssh: which fails in our CI/CD pipeline as it doesn't have any ssh keys configured.

We don't want to add ssh keys to our CI pipeline just to fetch public packages, so I don't see any way we can upgrade. I'd be glad to hear of any workaround.

Specific example:

In package.json referring to https://github.com/Vincit/winston-azure-transport:

    "winston-azure-transport": "github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215"

Diff of package-lock.json between npm v6 and v7:

     "winston-azure-transport": {
-      "version": "github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215",
-      "from": "github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215",```
+      "version": "git+ssh://git@github.com/Vincit/winston-azure-transport.git#ed07d0685ca601638ab8ebcf660128848dd30215",
+      "from": "winston-azure-transport@github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215",

Our CI now fails with:

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@github.com/Vincit/winston-azure-transport.git
npm ERR! 
npm ERR! git@github.com: Permission denied (publickey).
npm ERR! fatal: Could not read from remote repository.
npm ERR! 
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! 
npm ERR! exited with error code: 128