Publishing to IPFS

[cstamas]

The rough idea:

• human friendly starting point: dedicate domain (for humans) and have it DNSLink-ed to a IPFS key
• IPFS key is used for publishing IPNS
• optionally: share the key in secure manner with publishers (or just have one publisher or automate it)
• Hint: in case of key being compromised, just generate a new key, and update domain DNSLink to point to the new key, republish repository with new key
• trust is established very similarly as in Maven Central: domain tells you (via DNSLink) which IPNS you need to resolve to get to the latest CID (repository root).
• repo is usable with Maven via plain HTTP transport as well fx https://ipfs.io/ipns/maven.ipfs.io[/subpath] but we should discourage misuse of public gateways and herd users toward running local IPFS nodes instead.

The setup would look like DNS domain (via DNSlink) -> IPNS -> CID and is a double indirection.

Repository would be usable from Maven repository URL as fx ipfs:/maven.ipfs.io (maybe some subpath like ipfs:/maven.ipfs.io/releases. Given Maven release artifacts are immutable, repo leaves are usually unchanged, it is merely the merkle tree root that changes, as repository "grow". Publishers, consumers and interested parties would keep repository pinned in their nodes (see below, would happen automatically).

Consumers just need to add extension, declare IPFS repository in env/POM and consume artifacts "as usual".

Publishers just need to add extension, have the publishing private key added in their IPFS node, and would deploy as usual (just by invoking mvn deploy while running an IPFS node).

Requirements for consumers and publishers:

• added extension
• both consumers and publishers should run IPFS node locally (even a IPFS Desktop).

(Behind the scenes) Workflow for consumers (assume repository URL is ipfs:/maven.ipfs.io):

• name resolve maven.ipfs.io (to make sure you are up to date)
• pin add $CID pin resolved CID to local node
• files cp /ipfs/$CID /somewhere (exposes $CID via UnixFS)
• consume GAVs via UnixFS

For publishers:

• same as for consumers
• deploy files to repository structure files write... and update/maintain metadata via UnixFS
• publish new CID with $KEY on Maven session end

Ideally, publisher has to have private $KEY in IPFS Node named as $NAME, for example:

$ ipfs key list
self
ipfs.maveniverse.eu   <- key used to publish ipfs.maveniverse.eu domain linked with this key
$ 

---

This story above is implemented in https://github.com/maveniverse/ipfs

The extension adds:

• Resolver transport for IPFS. With this extension loaded, one can consume repositories hosted on IPFS via UnixFS (repo URL becomes ipfs:/name[/subpath] where name is CID, or any "resolvable" string like IPNS or DNSlinked domain, example https://ipfs.maveniverse.eu/)
• Aside of transport, Maven extension helps in deployment publishing automation (on Maven session end performs IPNS publishing).
• the majority of things is done here https://github.com/maveniverse/ipfs/blob/950dc9193b4e0b3f2b0605c9747500d50d46108a/core/src/main/java/eu/maveniverse/maven/ipfs/core/internal/IpfsNamespacePublisherImpl.java while the rest just "ties in" things and calls publisher when appropriate.

This story above is exactly how https://ipfs.maveniverse.eu/ domain is implemented. Content is backed by two 24/7 running IPFS geographically distant nodes (but is still slow 😞 ).

[cstamas]

This idea above would work on smaller scale (ie. pinning something huge would make local node explode), so "forges" like java-ipfs or maveniverse.eu is okay to work with (dozen MB). But repositories like whole Maven Central could not work like this.

Step 2 would be to provide some "translation service" ideally where trust can be directly established, for example:

• Take a Maven GAV
• (for start) we could rely on Central to retrieve GAV SHA1 (Maven Central publishes mandatory SHA1 checksums for everything) -- this would require that artifacts are published there as well (and is like "source of truth")
• Create some SHA1 -> CID lookup (or pubsub asking "who knows CID for this SHA1?")
• If CID known, make consumer go for CID directly (without pinning anything)
• If CID unknown, make consumer GET content from Central (push to IPFS, spread the mapping via pubsub?)
• Bytes gotten should correspond to SHA1 client started with (and then one knows "is same as it would be from Central" too).
• this way we are totally decoupled from GAVs (names), and we would care only about content (sha1/cid).

This is to be done in distributed cache (not there yet; just experimenting yet):
https://github.com/maveniverse/mimir/tree/main/node/ipfs

[vorburger]

@cstamas I'm interested in helping out with this - but I won't have the req. time needed... 😭

As mentioned on #256 (comment), I'm up for 🤑 sponsoring 📌 pinning📍 if that's a welcome contribution to moving this forward? Do we want to settle that in a sub-issue dedicated to that?

The full bigger picture isn't really entirely clear to me just yet, after reading above. My first reaction is that what you're trying to solve and come up with sounds complicated. Shouldn't all this ideally be completely independent of Maven Central?

At least for starters, if you were open to postpone the "Maven GAV to IPFS CID mapping" problem space to another decade, isn't there a much simpler way to go about this that's worth first exploring and attempting to popularize?

So say I'm project Z and I want to depend on project X. So I'll just SOMEHOW get a CID for X! Probably from X's GitHub README. Or because I've received it over IPoAC? It really doesn't matter.

Now couldn't https://github.com/maveniverse/ipfs, or something like it, just directly resolve that CID? I understand that a Maven POM can't directly have a dependency to an ipfs://QmXV... URI (like I had explored e.g. on [my] https://docs.enola.dev/use/fetch/#ipfs or as supported by https://curl.se/docs/ipfs.html) - or perhaps some future sort of IPFS PURL?

But I guess Maven absolutely does require a GAV, doesn't it? Then... let's just make up a convention! maveniverse.eu:ipfs:QmXV... - I don't suppose it could be as simple as (something like) that?

I'm probably forgetting that a Maven artifact isn't just a JAR 🫙 but necessarily also at least a POM, and hopefully also a JavaDoc, and a Source JAR... and possibly other classifiers. I guess we would also need a solution for that, but this seems like a solvable problem, with a bit more 🤔 thought... (Big ZIP with everything? A Maven IPLD?!)

Just some thoughts that crosses my mind which I thought I would dump here.

PS: What does any of this have to do directly with https://github.com/ipfs-shipyard/java-ipfs-http-client? 😂 I guess it doesn't really matter what project/ repo / place we're having this conversation at.

[vorburger]

But I guess Maven absolutely does require a GAV, doesn't it? Then... let's just make up a convention! maveniverse.eu:ipfs:QmXV... - I don't suppose it could be as simple as (something like) that?

Or if you think that's ugly, and you still want to see the Group and Name, although it's actually useless, but perhaps just familiar for humans, what if there was a convention that a V of a GAV that looks like a CID simply causes fetching from IPFS? So e.g. com.github.ipfs:java-ipfs-http-client:QmXV...? Just a thought.

PS: What does any of this have to do directly with https://github.com/ipfs-shipyard/java-ipfs-http-client? 😂 I guess it doesn't really matter what project/ repo / place we're having this conversation at.

Do at least some parts of this conversation perhaps belong on https://github.com/maveniverse/ipfs/issues?

[cstamas]

By removing namespaces (ie. by using fixed G:A as you propose and V as CID for example) the trust is gone. Anyone could publish slf4j-api with any malicious content they want.

By insisting on domain (needs domain ownership) -> IPNS (needs private key) -> CID the trust is established in very similar way like on Central: "I trust domain maveniverse.eu", and IPNS comes from that domain, and CID cannot be "faked" (nobody else can publish under that IPNS), or in other words, if anyone builds/deploys same CID as I did, they de-facto built bit-by-bit same thing, same JAR as I did. No harm done. As merkle tree is built from leaves toward top, the "root" is changing (not the leaves, they are immutable), and I (domain owner + key owner) am controlling pointers to root.

Hence the Maven@IPFS actually recommends (and later will even enforce) that if domain is maveniverse.eu, then groupID allowed to get from there is eu.maveniverse ONLY. I could still publish malicious artifacts, but only under my own namespace.

And yes, Maven needs metadata (for plugin and version discovery; let's remain at immutable release artifacts for now) and POMs and checksums and signatures...

---

This above is for publishing. The "Maven GAV to IPFS CID mapping" thing is unrelated to publishing (above). My bad: I should not have mix in caching in this issue, that is another interesting case for IPFS: you and I consume same slf4j-api, so we both probably will cache same content (same JAR for slf4j-api), hence same CID (with some fineprint, but more about it later) -- so why not share it. But again, we deal with two distinct user stories:

• publishing, like maveniverse.eu forge would do
• caching, (only releases that are immutable) artifacts from various places (Central, Maveniverse IPFS repo, etc)

---

See for example https://github.com/ipfs-shipyard/npm-on-ipfs