From 3c4ec9b3c473d684460e1df8b010c6ee4910848d Mon Sep 17 00:00:00 2001 From: Andrew Nesbitt Date: Thu, 18 Mar 2021 11:55:23 +0000 Subject: [PATCH 1/9] Create content-addressed-pkg-mgmt.md --- proposals/content-addressed-pkg-mgmt.md | 111 ++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 proposals/content-addressed-pkg-mgmt.md diff --git a/proposals/content-addressed-pkg-mgmt.md b/proposals/content-addressed-pkg-mgmt.md new file mode 100644 index 00000000..85d9a539 --- /dev/null +++ b/proposals/content-addressed-pkg-mgmt.md @@ -0,0 +1,111 @@ +# Content Addressed Package Management + +Authors: @andrew + +Initial PR: TBD + + + + + +## Purpose & impact +#### Background & intent +_Describe the desired state of the world after this project? Why does that matter?_ + + +Package management is an excellent use case for IPFS and Filecoin, as was [explored a couple of years ago](https://github.com/ipfs-inactive/package-managers), +but has yet to see major adoption within any large package managers. + +Making package managers content addressable would unlock the ability for the data within package management registries to become more portable, +opening it up to much need innovation in areas such as transport protocols, storage backends and network topologies. + +To enable this a specification for a protocol for content addressed package management would be written, along with a [working implementation](https://github.com/forestpm/forest), +that outlines a standardized way for consumers, publishers and maintainers to share package management data that allows people to opt-in to various levels of decentralization without +sacrificing existing usability, performance and security. + + + +#### Assumptions & hypotheses +_What must be true for this project to matter?_ + + +#### User workflow example +_How would a developer or user use this new capability?_ + + +#### Impact +_How would this directly contribute to web3 dev stack product-market fit?_ + + + +#### Leverage +_How much would nailing this project improve our knowledge and ability to execute future projects?_ + + + +#### Confidence +_How sure are we that this impact would be realized? Label from [this scale](https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15)_. + + + + +## Project definition +#### Brief plan of attack + + + +#### What does done look like? +_What specific deliverables should completed to consider this project done?_ + +#### What does success look like? +_Success means impact. How will we know we did the right thing?_ + + + +#### Counterpoints & pre-mortem +_Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ + +#### Alternatives +_How might this project’s intent be realized in other ways (other than this project proposal)? What other potential solutions can address the same need?_ + +#### Dependencies/prerequisites + + +#### Future opportunities + + +## Required resources + +#### Effort estimate + + +#### Roles / skills needed + From eae013342c9bf38acab7ea96065eb7edc5660c6e Mon Sep 17 00:00:00 2001 From: Andrew Nesbitt Date: Thu, 18 Mar 2021 12:27:31 +0000 Subject: [PATCH 2/9] Update content-addressed-pkg-mgmt.md --- proposals/content-addressed-pkg-mgmt.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/proposals/content-addressed-pkg-mgmt.md b/proposals/content-addressed-pkg-mgmt.md index 85d9a539..d09342a5 100644 --- a/proposals/content-addressed-pkg-mgmt.md +++ b/proposals/content-addressed-pkg-mgmt.md @@ -1,6 +1,6 @@ # Content Addressed Package Management -Authors: @andrew +Authors: [@andrew](https://github.com/andrew) Initial PR: TBD @@ -33,19 +33,20 @@ but has yet to see major adoption within any large package managers. Making package managers content addressable would unlock the ability for the data within package management registries to become more portable, opening it up to much need innovation in areas such as transport protocols, storage backends and network topologies. -To enable this a specification for a protocol for content addressed package management would be written, along with a [working implementation](https://github.com/forestpm/forest), -that outlines a standardized way for consumers, publishers and maintainers to share package management data that allows people to opt-in to various levels of decentralization without -sacrificing existing usability, performance and security. - - +To enable this a specification for a protocol for content addressed package management would be written, along with a [working implementation](https://github.com/forestpm/forest), that outlines a standardized way for consumers, publishers and maintainers to share package management data that allows people to opt-in to various levels of decentralization without sacrificing existing usability, performance and security. #### Assumptions & hypotheses _What must be true for this project to matter?_ +* package management data is content addressable #### User workflow example _How would a developer or user use this new capability?_ +* Developers can quickly and safetly fetch packages directly from other developers, falling back to upstream registry +* Develoeprs can easily onboard onto content addressed package management without high commitment costs +* IT teams can easily mirror packages within an internal network with low infrastructure costs +* Third-parties can co-host packages from a registry in a trustless, permissionless way #### Impact _How would this directly contribute to web3 dev stack product-market fit?_ @@ -55,12 +56,24 @@ Explain how this addresses known challenges or opportunities. What awesome potential impact/outcomes/results will we see if we nail this project? --> +Past attempts at decentralizing packagement have attempted to import whole registries into IPFS but come with a very high commitment cost and painful onboarding UX, as such have had low adoption so far. + +In taking a slightly different approach, focusing on content addressing data from registries, we allow each user to co-host the packages that they use, unlocking resiliance and performance gains without requiring significant infrastructure or upfront buy-in from upstream registry maintainers. + +Almost every software developer uses package management in some form, reducing the commitment costs of switching to content addressed packagement could unlock the data from large centralized registries and enable significant innovation in the package management space, enabling innovations like blockchain-based solutions that work seemlessly with existing centralized infrastructure. + + #### Leverage _How much would nailing this project improve our knowledge and ability to execute future projects?_ +Formalizing the existing patterns in packagement in content addressed terminology will enable other projects and contributors to build on top of those formats in an open form. + +Brand new package managers will be able to directly implement the best practices as well as reducing their infrastructure overheads by adhering to a content addressed package management protocol. + +Documenting the architecture pattern of how we content addressed and decentralized package management can then be used as a blueprint for other similar internet infrastructure projects. #### Confidence _How sure are we that this impact would be realized? Label from [this scale](https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15)_. From 1a335a5ff6cf7d0909fcc718ecdb08237c74ce85 Mon Sep 17 00:00:00 2001 From: Andrew Nesbitt Date: Thu, 18 Mar 2021 14:11:40 +0000 Subject: [PATCH 3/9] Update content-addressed-pkg-mgmt.md --- proposals/content-addressed-pkg-mgmt.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/proposals/content-addressed-pkg-mgmt.md b/proposals/content-addressed-pkg-mgmt.md index d09342a5..ea9ddb06 100644 --- a/proposals/content-addressed-pkg-mgmt.md +++ b/proposals/content-addressed-pkg-mgmt.md @@ -86,15 +86,22 @@ _How sure are we that this impact would be realized? Label from [this scale](htt +- TODO + #### What does done look like? _What specific deliverables should completed to consider this project done?_ +- TODO + #### What does success look like? _Success means impact. How will we know we did the right thing?_ +- increasing number of users sharing packages on the dht +- projects and package managers adopting and building on the protocol +- increased discussion around content addressing in software development and tooling #### Counterpoints & pre-mortem _Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ From 3667b4635c7c2bb978e5abb963708ea136e04686 Mon Sep 17 00:00:00 2001 From: Andrew Nesbitt Date: Thu, 18 Mar 2021 14:12:25 +0000 Subject: [PATCH 4/9] Update content-addressed-pkg-mgmt.md --- proposals/content-addressed-pkg-mgmt.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/proposals/content-addressed-pkg-mgmt.md b/proposals/content-addressed-pkg-mgmt.md index ea9ddb06..b36e5349 100644 --- a/proposals/content-addressed-pkg-mgmt.md +++ b/proposals/content-addressed-pkg-mgmt.md @@ -79,7 +79,7 @@ Documenting the architecture pattern of how we content addressed and decentraliz _How sure are we that this impact would be realized? Label from [this scale](https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15)_. - +- TODO ## Project definition #### Brief plan of attack @@ -106,15 +106,23 @@ Provide success criteria. These might include particular metrics, desired change #### Counterpoints & pre-mortem _Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ +- TODO + #### Alternatives _How might this project’s intent be realized in other ways (other than this project proposal)? What other potential solutions can address the same need?_ +- TODO + #### Dependencies/prerequisites +- TODO + #### Future opportunities +- TODO + ## Required resources #### Effort estimate @@ -127,5 +135,9 @@ For a team of 3-5 people with the appropriate skills: Describe any choices and uncertainty in this scope estimate. (E.g. Uncertainty in the scope until design work is complete, low uncertainty in execution thereafter.) --> +- TODO + #### Roles / skills needed + +- TODO From 1405d4fea2f39e0c1ec6a1b8d78442c8714849cd Mon Sep 17 00:00:00 2001 From: Andrew Nesbitt Date: Thu, 18 Mar 2021 15:45:21 +0000 Subject: [PATCH 5/9] Update content-addressed-pkg-mgmt.md --- proposals/content-addressed-pkg-mgmt.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/proposals/content-addressed-pkg-mgmt.md b/proposals/content-addressed-pkg-mgmt.md index b36e5349..39ae3b90 100644 --- a/proposals/content-addressed-pkg-mgmt.md +++ b/proposals/content-addressed-pkg-mgmt.md @@ -116,12 +116,13 @@ _How might this project’s intent be realized in other ways (other than this pr #### Dependencies/prerequisites -- TODO +- not directly dependent on, but can helped with https://github.com/protocol/beyond-bitswap/pull/29 #### Future opportunities - -- TODO +A standardized package metadata format can enable the development of: +- a content addressable, public transparency log for all package managers, similar to the ]certificate transparency project](https://certificate.transparency.dev/) +- package maintainers could publish signed nfts of each release of their packages, enabling alternative sources of funding for open source ## Required resources @@ -135,9 +136,12 @@ For a team of 3-5 people with the appropriate skills: Describe any choices and uncertainty in this scope estimate. (E.g. Uncertainty in the scope until design work is complete, low uncertainty in execution thereafter.) --> -- TODO +- Large, 6-10 weeks #### Roles / skills needed -- TODO +- package management knowledge +- javascript development +- ipfs experience +- protocol design/specification experience From 50867350539205f7bdfad23d6dce33e748c78940 Mon Sep 17 00:00:00 2001 From: Andrew Nesbitt Date: Thu, 18 Mar 2021 16:17:05 +0000 Subject: [PATCH 6/9] Update content-addressed-pkg-mgmt.md --- proposals/content-addressed-pkg-mgmt.md | 44 ++++++++++++++++--------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/proposals/content-addressed-pkg-mgmt.md b/proposals/content-addressed-pkg-mgmt.md index 39ae3b90..c436733a 100644 --- a/proposals/content-addressed-pkg-mgmt.md +++ b/proposals/content-addressed-pkg-mgmt.md @@ -1,4 +1,4 @@ -# Content Addressed Package Management +# Content Addressed Package Management Authors: [@andrew](https://github.com/andrew) @@ -13,24 +13,24 @@ A proposal should contain enough detail for others to understand how this projec for our unified stack of protocols, what is included in scope of the project, where to get started if a project team were to take this on, and any other information relevant for prioritizing this project against others. It does not need to describe the work in much detail. Most technical design and planning would take place after a proposal is adopted. -Good project scope aims for ~3-5 engineers for 1-3 months (though feel free to suggest larger-scoped projects anyway). +Good project scope aims for ~3-5 engineers for 1-3 months (though feel free to suggest larger-scoped projects anyway). Projects do not include regular day-to-day maintenance and improvement work, e.g. on testing, tooling, validation, code clarity, refactors for future capability, etc. --> -## Purpose & impact +## Purpose & impact #### Background & intent _Describe the desired state of the world after this project? Why does that matter?_ -Package management is an excellent use case for IPFS and Filecoin, as was [explored a couple of years ago](https://github.com/ipfs-inactive/package-managers), +Package management is an excellent use case for IPFS and Filecoin, as was [explored a couple of years ago](https://github.com/ipfs-inactive/package-managers), but has yet to see major adoption within any large package managers. -Making package managers content addressable would unlock the ability for the data within package management registries to become more portable, +Making package managers content addressable would unlock the ability for the data within package management registries to become more portable, opening it up to much need innovation in areas such as transport protocols, storage backends and network topologies. To enable this a specification for a protocol for content addressed package management would be written, along with a [working implementation](https://github.com/forestpm/forest), that outlines a standardized way for consumers, publishers and maintainers to share package management data that allows people to opt-in to various levels of decentralization without sacrificing existing usability, performance and security. @@ -86,12 +86,22 @@ _How sure are we that this impact would be realized? Label from [this scale](htt -- TODO +- Research and document various package manager metadata formats, api endpoints and processes to extract requirements for spec +- Initial draft of data model and formats for spec +- Refactor core data model of https://github.com/forestpm/forest to implement draft spec +- Deploy analytics service for monitoring DHT for package usage +- Build and deploy marketing website for https://github.com/forestpm/forest and onboard early adopters +- Reach out to key stakeholders in package management for early thoughts, feedback and requirements +- Refine and document spec + implementation +- Publish v1 of https://github.com/forestpm/forest +- Publish specification #### What does done look like? _What specific deliverables should completed to consider this project done?_ -- TODO +- A published draft of a v0.1 of the content addressed package management protocol specification +- A functional implementation of the specification with support for multiple package managers +- Documented discussion of protocol and process with a selection of package manager maintainers, publishers and consumers #### What does success look like? _Success means impact. How will we know we did the right thing?_ @@ -99,24 +109,26 @@ _Success means impact. How will we know we did the right thing?_ -- increasing number of users sharing packages on the dht -- projects and package managers adopting and building on the protocol -- increased discussion around content addressing in software development and tooling +- Increasing number of users sharing packages on the dht +- Projects and package managers adopting and building on the protocol +- Increased discussion around content addressing in software development and tooling #### Counterpoints & pre-mortem _Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ -- TODO +- Content discovery and publishing on the IPFS DHT of millions of packages may affect growth once a certain level of adoption is reached +- If the user experience is poor or switching costs are too high, reaching critical levels of adoption will be difficult +- There may be incentives for some package management infrastructure organizations to retain centralized control and choose to avoid adoption #### Alternatives _How might this project’s intent be realized in other ways (other than this project proposal)? What other potential solutions can address the same need?_ -- TODO +- [Valist](https://github.com/valist-io/valist) is an alternative project, although it appears to be closely tied to Ethereum. #### Dependencies/prerequisites -- not directly dependent on, but can helped with https://github.com/protocol/beyond-bitswap/pull/29 +- Whilst not directly dependent on, https://github.com/protocol/beyond-bitswap/pull/29 can help ease onboarding #### Future opportunities @@ -127,7 +139,7 @@ A standardized package metadata format can enable the development of: ## Required resources #### Effort estimate - * package management data is content addressable +* package manager users care about the verifiablity of their software supply chain #### User workflow example _How would a developer or user use this new capability?_ From 7b28e55a8e248c420c423f337259d2c0838f906f Mon Sep 17 00:00:00 2001 From: Andrew Nesbitt Date: Thu, 18 Mar 2021 16:53:11 +0000 Subject: [PATCH 9/9] Update content-addressed-pkg-mgmt.md --- proposals/content-addressed-pkg-mgmt.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/content-addressed-pkg-mgmt.md b/proposals/content-addressed-pkg-mgmt.md index 0b609a43..d503bd1d 100644 --- a/proposals/content-addressed-pkg-mgmt.md +++ b/proposals/content-addressed-pkg-mgmt.md @@ -30,9 +30,13 @@ Outline the status quo, including any relevant context on the problem you’re s Package management is an excellent use case for IPFS and Filecoin, as was [explored a couple of years ago](https://github.com/ipfs-inactive/package-managers), but has yet to see major adoption within any large package managers. +Package management has been plagued with reproducibility problems over the past few years, with the same mistakes often made by different package manager servers and clients because there's no documented standards for designing and building package managers, even less so for content addressed/decentralized package managers. + Making package managers content addressable would unlock the ability for the data within package management registries to become more portable, opening it up to much needed innovation in areas such as transport protocols, storage backends and network topologies. +A documented set of basic standards for how to implement a content address package management system will enable new languages/communities to quickly produce reliable and secure implementations. + To enable this a specification for a protocol for content addressed package management would be written, along with a [working implementation](https://github.com/forestpm/forest), that outlines a standardized way for consumers, publishers and maintainers to share package management data that allows people to opt-in to various levels of decentralization without sacrificing existing usability, performance and security. #### Assumptions & hypotheses