From c5fc70372475c838e973e69eb91cbb2854469cfc Mon Sep 17 00:00:00 2001 From: Carlo Capocasa Date: Wed, 4 Dec 2019 22:38:59 +0100 Subject: [PATCH 1/8] initial --- rfcs/0001-binary-quark.md | 44 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 rfcs/0001-binary-quark.md diff --git a/rfcs/0001-binary-quark.md b/rfcs/0001-binary-quark.md new file mode 100644 index 0000000..2491541 --- /dev/null +++ b/rfcs/0001-binary-quark.md @@ -0,0 +1,44 @@ +- Title: Quarks with binary components +- Date proposed: 2019-12-04 +- RFC PR: https://github.com/supercollider/rfcs/pull/0001 **update this number after RFC PR has been filed** + +# Summary + +Quarks currently do not support C++ components (or other compiled languages), such as UGens, buffer fills, or server side commands. This RFC proposes a standardized method for this to work. + +# Motivation + +Supporting C++ components would enable all 3rd party code distribution to be done with quarks, providing quark features such as automated installation and dependency management to binary extensions. It would potentially enable the retiring of the sc3-plugins repository in the future, simplifying extension installation by providing a single method of installation. Done well, a drastic improvement in easily available sounds and tools can be expected, as well as an increase in productivity. + + +# Specification + +A cross-platform shell script called quark.build is added to quarks with binary extensions, in the canonical case calling cmake and make. ¹ + +Quark installation checks for the presences of this file. If it is present, is is executed after installation, displaying the results in the terminal, and noting success if the return code is 0, failure otherwise. + +Building requires the supercollider headers, so on first build the SC code is checked out locally with the version executing the installation. This location, along with platform information, is provided to the build script as environment variables. + +# Drawbacks + +1. Source distribution is considered to be infeasible by a majority of responders for at least on OSX and Windows, while the RFC author considers it reasonably easy at least on Linux. +2. Using cross platform shell code is very limiting, but that's ok because the sole job of the build script is to call the build system correctly. Seperate build scripts per platform can be used as a plan B. + +# Unresolved Questions + +- Shall binary quarks be distributed in binary form? + + - If yes, does the cross-platform build process happen on a CI system? + - If yes, do we have a volunteer to maintain such as system? + - If yes, Shall other build options than C++ be supported, such as Rust or Nim, perhaps deferred? + - If yes, can we create builds for each common type of executable on Linux? Different binary formats and 32/64 bit add up and require detection + - If no, how is transpiling set up on each possible OS for a quark maintainer's system? Or should each quark be expected to be built by volunteers on all three systems? + - If yes, should source distribution be used as a fallback mechanism? + - If no, how is it ensured that installing build requirements is reasonably easy on OSX and Windows? To answer this question, other systems such as the Ruby Gem 'Nokogiri' should be investigated. + +Preliminary [discussion on the sc-dev mailing list](https://www.listarc.bham.ac.uk/lists/sc-dev/thrd10.html#59665) +Seperate related [discussion on the sc-dev mailing list regarding the sc3-plugins repository](https://www.listarc.bham.ac.uk/lists/sc-dev/msg58832.html) + +--- +¹ The initial draft used a `build.scd` in sclang as a build script, but some users may wish not to depend on sclang, and cross platform shell needs to be used anyway via systemCmd or similar, so there is no gain. + From ac7318ac316d5e38a07a77de662a6ce3145dcbc3 Mon Sep 17 00:00:00 2001 From: Carlo Capocasa Date: Wed, 4 Dec 2019 22:40:16 +0100 Subject: [PATCH 2/8] number --- rfcs/0001-binary-quark.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0001-binary-quark.md b/rfcs/0001-binary-quark.md index 2491541..ed31fd4 100644 --- a/rfcs/0001-binary-quark.md +++ b/rfcs/0001-binary-quark.md @@ -1,6 +1,6 @@ -- Title: Quarks with binary components +- Title: Distributing with binary components - Date proposed: 2019-12-04 -- RFC PR: https://github.com/supercollider/rfcs/pull/0001 **update this number after RFC PR has been filed** +- RFC PR: https://github.com/supercollider/rfcs/pull/7 **update this number after RFC PR has been filed** # Summary From f1a58084c9f3cd51b93cbe428efe75638347d295 Mon Sep 17 00:00:00 2001 From: Carlo Capocasa Date: Wed, 4 Dec 2019 22:46:54 +0100 Subject: [PATCH 3/8] add proof of concept --- rfcs/0001-binary-quark.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/0001-binary-quark.md b/rfcs/0001-binary-quark.md index ed31fd4..5adebab 100644 --- a/rfcs/0001-binary-quark.md +++ b/rfcs/0001-binary-quark.md @@ -19,6 +19,8 @@ Quark installation checks for the presences of this file. If it is present, is i Building requires the supercollider headers, so on first build the SC code is checked out locally with the version executing the installation. This location, along with platform information, is provided to the build script as environment variables. +A working proof of concept for an installable quark with a build script: [ExampleBuildQuark](https://github.com/capocasa/ExampleBuildQuark) + # Drawbacks 1. Source distribution is considered to be infeasible by a majority of responders for at least on OSX and Windows, while the RFC author considers it reasonably easy at least on Linux. From b1f2f474b55dcdcb590a6546b0bdfa64b7e243a1 Mon Sep 17 00:00:00 2001 From: Carlo Capocasa Date: Wed, 4 Dec 2019 22:48:43 +0100 Subject: [PATCH 4/8] clean --- rfcs/0001-binary-quark.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0001-binary-quark.md b/rfcs/0001-binary-quark.md index 5adebab..762e0c1 100644 --- a/rfcs/0001-binary-quark.md +++ b/rfcs/0001-binary-quark.md @@ -1,6 +1,6 @@ - Title: Distributing with binary components - Date proposed: 2019-12-04 -- RFC PR: https://github.com/supercollider/rfcs/pull/7 **update this number after RFC PR has been filed** +- RFC PR: https://github.com/supercollider/rfcs/pull/7 # Summary From 06d600e749e3bff25b7eba64d99f96ff5a0936e2 Mon Sep 17 00:00:00 2001 From: Carlo Capocasa Date: Wed, 4 Dec 2019 22:49:27 +0100 Subject: [PATCH 5/8] tweak --- rfcs/0001-binary-quark.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0001-binary-quark.md b/rfcs/0001-binary-quark.md index 762e0c1..a8662bc 100644 --- a/rfcs/0001-binary-quark.md +++ b/rfcs/0001-binary-quark.md @@ -1,4 +1,4 @@ -- Title: Distributing with binary components +- Title: Distributing quarks with binary components - Date proposed: 2019-12-04 - RFC PR: https://github.com/supercollider/rfcs/pull/7 @@ -32,7 +32,7 @@ A working proof of concept for an installable quark with a build script: [Exampl - If yes, does the cross-platform build process happen on a CI system? - If yes, do we have a volunteer to maintain such as system? - - If yes, Shall other build options than C++ be supported, such as Rust or Nim, perhaps deferred? + - If yes, Shall other build options than C++ be supported, such as Rust or Nim, perhaps later? - If yes, can we create builds for each common type of executable on Linux? Different binary formats and 32/64 bit add up and require detection - If no, how is transpiling set up on each possible OS for a quark maintainer's system? Or should each quark be expected to be built by volunteers on all three systems? - If yes, should source distribution be used as a fallback mechanism? From cbf88f613b48616ee893853bf07ff521f0b0853c Mon Sep 17 00:00:00 2001 From: Carlo Capocasa Date: Fri, 6 Dec 2019 21:17:00 +0100 Subject: [PATCH 6/8] proposition addressing raised issues --- rfcs/0001-binary-quark.md | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/rfcs/0001-binary-quark.md b/rfcs/0001-binary-quark.md index a8662bc..8653513 100644 --- a/rfcs/0001-binary-quark.md +++ b/rfcs/0001-binary-quark.md @@ -4,22 +4,34 @@ # Summary -Quarks currently do not support C++ components (or other compiled languages), such as UGens, buffer fills, or server side commands. This RFC proposes a standardized method for this to work. +Quarks currently do not support C++ components (or other compiled languages), such as UGens, buffer fills, or server side commands. This RFC proposes a means to support these. # Motivation -Supporting C++ components would enable all 3rd party code distribution to be done with quarks, providing quark features such as automated installation and dependency management to binary extensions. It would potentially enable the retiring of the sc3-plugins repository in the future, simplifying extension installation by providing a single method of installation. Done well, a drastic improvement in easily available sounds and tools can be expected, as well as an increase in productivity. +Supporting quarks with binary components enable all 3rd party code distribution to be done with quarks, providing quark features such as automated installation and dependency management to binary extensions. It would potentially enable the retiring of the sc3-plugins repository in the future, simplifying extension installation by providing a single method of installation. Done well, a drastic improvement in easily available sounds and tools can be expected, as well as an increase in productivity. # Specification -A cross-platform shell script called quark.build is added to quarks with binary extensions, in the canonical case calling cmake and make. ¹ +Three phases of implementation or proposed in order to focus development on smaller chunks and prove their utility before moving to the next. -Quark installation checks for the presences of this file. If it is present, is is executed after installation, displaying the results in the terminal, and noting success if the return code is 0, failure otherwise. +## Core phase -Building requires the supercollider headers, so on first build the SC code is checked out locally with the version executing the installation. This location, along with platform information, is provided to the build script as environment variables. +*The goal of the core phase is to get a system ready that builds binary quarks for a group of early testers on different Linux systems* -A working proof of concept for an installable quark with a build script: [ExampleBuildQuark](https://github.com/capocasa/ExampleBuildQuark) +A quark called `binary` is created that can be depended on by quarks that wish to distribute binaries. + +The binary quark checks quarks for the presence of a CMakeLists.txt file, and executes a CMake build after installation if they do. Supercollider headers of an appropriate sclang version are downloaded before build as required. It is assumed that the quark maintainer will set any options to default to the value used for distribution. + +A proof of concept for an installable quark with a build script: [ExampleBuildQuark](https://github.com/capocasa/ExampleBuildQuark) + +## Expansion phase + +The build-from-source approach is evaluated in OSX, with the quark packaging system automatically prompting to install a dependency through the App Store, and automatically downloading another. This approach is evaluated by a group of initial testers on OSX. If it is found to be satisfactory, some newish users can also be exposed to the system. If it works well enough, with some help, this approach is decided to be used. If it does not work well enough, OSX binaries the build-approach for OSX is considered failed and binaries will be distributed during the following phase. + +# Binary distribution phase + +A binary repository is created, and questions on where to host it who should have control, and who performs builds are answered. 32bit and 64bit binaries are transpiled from Linux and possibly OSX using a boilerplate CMakeLists.txt. If the expansion phase showed that source builds on OSX are a significant usability problem during the Expansion builds, and 64bit (or perhaps also 32bit or fat binaries) OSX binaries are added to Windows ones for binary distrubtion. # Drawbacks @@ -28,19 +40,9 @@ A working proof of concept for an installable quark with a build script: [Exampl # Unresolved Questions -- Shall binary quarks be distributed in binary form? +- This plan of action attempts to reconcile all points raised in discussion for the end result. Has this been successful, or should further points be addressed? - - If yes, does the cross-platform build process happen on a CI system? - - If yes, do we have a volunteer to maintain such as system? - - If yes, Shall other build options than C++ be supported, such as Rust or Nim, perhaps later? - - If yes, can we create builds for each common type of executable on Linux? Different binary formats and 32/64 bit add up and require detection - - If no, how is transpiling set up on each possible OS for a quark maintainer's system? Or should each quark be expected to be built by volunteers on all three systems? - - If yes, should source distribution be used as a fallback mechanism? - - If no, how is it ensured that installing build requirements is reasonably easy on OSX and Windows? To answer this question, other systems such as the Ruby Gem 'Nokogiri' should be investigated. Preliminary [discussion on the sc-dev mailing list](https://www.listarc.bham.ac.uk/lists/sc-dev/thrd10.html#59665) Seperate related [discussion on the sc-dev mailing list regarding the sc3-plugins repository](https://www.listarc.bham.ac.uk/lists/sc-dev/msg58832.html) ---- -¹ The initial draft used a `build.scd` in sclang as a build script, but some users may wish not to depend on sclang, and cross platform shell needs to be used anyway via systemCmd or similar, so there is no gain. - From debe07b673769b00962df756c979ca0ee71eb1bd Mon Sep 17 00:00:00 2001 From: Carlo Capocasa Date: Fri, 6 Dec 2019 23:14:37 +0100 Subject: [PATCH 7/8] updates --- rfcs/0001-binary-quark.md | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/rfcs/0001-binary-quark.md b/rfcs/0001-binary-quark.md index 8653513..5e275f1 100644 --- a/rfcs/0001-binary-quark.md +++ b/rfcs/0001-binary-quark.md @@ -15,7 +15,7 @@ Supporting quarks with binary components enable all 3rd party code distribution Three phases of implementation or proposed in order to focus development on smaller chunks and prove their utility before moving to the next. -## Core phase +## 1. Core phase *The goal of the core phase is to get a system ready that builds binary quarks for a group of early testers on different Linux systems* @@ -25,18 +25,32 @@ The binary quark checks quarks for the presence of a CMakeLists.txt file, and ex A proof of concept for an installable quark with a build script: [ExampleBuildQuark](https://github.com/capocasa/ExampleBuildQuark) -## Expansion phase +## 2. Expansion phase The build-from-source approach is evaluated in OSX, with the quark packaging system automatically prompting to install a dependency through the App Store, and automatically downloading another. This approach is evaluated by a group of initial testers on OSX. If it is found to be satisfactory, some newish users can also be exposed to the system. If it works well enough, with some help, this approach is decided to be used. If it does not work well enough, OSX binaries the build-approach for OSX is considered failed and binaries will be distributed during the following phase. -# Binary distribution phase +# 3. Binary distribution phase A binary repository is created, and questions on where to host it who should have control, and who performs builds are answered. 32bit and 64bit binaries are transpiled from Linux and possibly OSX using a boilerplate CMakeLists.txt. If the expansion phase showed that source builds on OSX are a significant usability problem during the Expansion builds, and 64bit (or perhaps also 32bit or fat binaries) OSX binaries are added to Windows ones for binary distrubtion. +## Nitty gritty details + +Some details were omitted from above for easier understanding: + +- The "secret sauce" from pure data's Deken that allows binary extensions to be distributed effectively is an automation of tagging the binary for its architecture, OS version, and exeutable format, in great detail (e.g. 10+ architecures). This proposal suggests to at attempt source distribution so CMake can do that. If source distribution doesn't pan out, then this should be ported to sclang from the pure data's lisp version, which is obviously much harder to get right than letting cmake handle it. + +- Some improvements to quarks were suggested, such as dependencies by version and OS, as well as sclang-version-specific quark installations. These will be done in phase 1 as needed. + +- Preparation for building from source on Linux involves installing a build package and cmake. On OSX, it involves using a terminal command to trigger installation of the xcode-command-line-tools package in the app store, as well as installing binary cmake. The latter can probably be automated by the quark system and placed outside the path so it doesn't conflict with anything on the machine, such as from homebrew, an OSX package system. + +- Since Nokogiri gave up on source builds on Windows, this proposal will too. It is still unclear how exactly a binary repository would work, and who would control it. It is also unclear who will do the builds- the package maintainer, another volunteer, or a CI system. The latter would be preferred by the RFC, if a volunteer can be found to maintain the CI system for windows builds. Github integration would be great. If builds are done by the package maintainer, github release would work. If it is to be done by a third party, then it is unclear where it could be efficiently uploaded. + +- Some kind of "confirmed working on OSX/Windows/Linux" feature would be nice, since package maintainers are unlikely to be able to test on three platforms. That would make it more obvious how well packages are tested. This would have to happen *after* phase 3 though. + # Drawbacks -1. Source distribution is considered to be infeasible by a majority of responders for at least on OSX and Windows, while the RFC author considers it reasonably easy at least on Linux. -2. Using cross platform shell code is very limiting, but that's ok because the sole job of the build script is to call the build system correctly. Seperate build scripts per platform can be used as a plan B. +This approach defers some questions to a later phase, however that seems hard to avoid given the scope is so large. + # Unresolved Questions From fa572835f42e0449308eaa7cd34b004f15fee275 Mon Sep 17 00:00:00 2001 From: Carlo Capocasa Date: Tue, 10 Dec 2019 14:27:38 +0000 Subject: [PATCH 8/8] update to discussion after developer call --- rfcs/0001-binary-quark.md | 61 +++++++++++++++++++++++++-------------- 1 file changed, 39 insertions(+), 22 deletions(-) diff --git a/rfcs/0001-binary-quark.md b/rfcs/0001-binary-quark.md index 5e275f1..35bb727 100644 --- a/rfcs/0001-binary-quark.md +++ b/rfcs/0001-binary-quark.md @@ -4,7 +4,7 @@ # Summary -Quarks currently do not support C++ components (or other compiled languages), such as UGens, buffer fills, or server side commands. This RFC proposes a means to support these. +Quarks currently do not support UGens, buffer fills, server side commands, which requires binary artifacts to be loaded by the scsynth or supernova servers. This RFC proposes to permit. # Motivation @@ -13,49 +13,66 @@ Supporting quarks with binary components enable all 3rd party code distribution # Specification -Three phases of implementation or proposed in order to focus development on smaller chunks and prove their utility before moving to the next. +A boilerplate example quark with cmake and travis scripts is provided and documented on the "Writing Quarks" guide. -## 1. Core phase +The boilerplate allows building quarks that have no special build requirements with minimal editing. All platforms that are supported by SuperCollider are enabled by default. Plugin authors are encouraged to build for all platforms unless the plugin is inherently platform specific. Platforms are specified in a configuration section at the top of the build script. Supernova builds are also enabled by default and can be manually disabled. -*The goal of the core phase is to get a system ready that builds binary quarks for a group of early testers on different Linux systems* +A travis CI environment is activated through github to automatically build on push. If the push was a tag with a semantic version number, a formal github release containing the builds is automatically created using the github API. -A quark called `binary` is created that can be depended on by quarks that wish to distribute binaries. +The Quark system checks for the presence of releases on install, downloads the appropriate release for the system, and places it within a folder in the app support directory alongside downloaded quarks, with the platform and intended scsynth/supernova version number as subdirectories. Then the directory symlinked (if supported) or copied into a "active-quarks" directory inside the extensions directory so they are picked up by servers on boot. When quark directories are modified, symlinks must be deleted and created as appropriate. -The binary quark checks quarks for the presence of a CMakeLists.txt file, and executes a CMake build after installation if they do. Supercollider headers of an appropriate sclang version are downloaded before build as required. It is assumed that the quark maintainer will set any options to default to the value used for distribution. +On executing an quark update, missing binary components are added. This can happen when quarks are transferred to a different platform, or sc is up/downgraded -A proof of concept for an installable quark with a build script: [ExampleBuildQuark](https://github.com/capocasa/ExampleBuildQuark) +## Requirements -## 2. Expansion phase +- Quarks with binary artifacts must be no more difficult to install than those without- ideally, users should not be required to know the difference +- Especially, they must be installable on machines with no root access +- They must be supported on all platforms where binary SuperCollider releases are provided, except where the nature of the quark precludes it +- The build system must have as few dependencies as possible in addition to CMake +- Impact on non-sclang quark systems must be as small as possible -The build-from-source approach is evaluated in OSX, with the quark packaging system automatically prompting to install a dependency through the App Store, and automatically downloading another. This approach is evaluated by a group of initial testers on OSX. If it is found to be satisfactory, some newish users can also be exposed to the system. If it works well enough, with some help, this approach is decided to be used. If it does not work well enough, OSX binaries the build-approach for OSX is considered failed and binaries will be distributed during the following phase. +## Out of scope -# 3. Binary distribution phase +The scope of this RFC is limited to distributing binary artifacts, and the minimal changes to the current quarks system required for that purpose. Changes that go further are out of scope, including: -A binary repository is created, and questions on where to host it who should have control, and who performs builds are answered. 32bit and 64bit binaries are transpiled from Linux and possibly OSX using a boilerplate CMakeLists.txt. If the expansion phase showed that source builds on OSX are a significant usability problem during the Expansion builds, and 64bit (or perhaps also 32bit or fat binaries) OSX binaries are added to Windows ones for binary distrubtion. +- Improving dependency management +- Improving quarks.txt loading behvaior +- Directory-specific quark loading +- Security improvements such as parsing the .quark file instead of executing it +- Reducing dendencies, e.g. dropping dependency on git +- Non-github repositories -## Nitty gritty details +## Deferred -Some details were omitted from above for easier understanding: +These remain within scope for discussion, but will likely not be part of an initial release. -- The "secret sauce" from pure data's Deken that allows binary extensions to be distributed effectively is an automation of tagging the binary for its architecture, OS version, and exeutable format, in great detail (e.g. 10+ architecures). This proposal suggests to at attempt source distribution so CMake can do that. If source distribution doesn't pan out, then this should be ported to sclang from the pure data's lisp version, which is obviously much harder to get right than letting cmake handle it. +- Assisted test tracking- developers are encouraged to test as many built platforms as possible and mark them tested, while others also may mark platforms tested. The Quark installer then displays whether a quark has been found working on the current system. +- Assisted error reporting- errors should be reported via github issues +- Automated source build to assist developers and users of embedded systems or exotic hardware +- Support for languages other than C++ such as Faust, Nim, Rust - at release they will be only supported indirectly via intermediate C++ if applicable -- Some improvements to quarks were suggested, such as dependencies by version and OS, as well as sclang-version-specific quark installations. These will be done in phase 1 as needed. +## Rejected -- Preparation for building from source on Linux involves installing a build package and cmake. On OSX, it involves using a terminal command to trigger installation of the xcode-command-line-tools package in the app store, as well as installing binary cmake. The latter can probably be automated by the quark system and placed outside the path so it doesn't conflict with anything on the machine, such as from homebrew, an OSX package system. +These ideas have been considered and rejected -- Since Nokogiri gave up on source builds on Windows, this proposal will too. It is still unclear how exactly a binary repository would work, and who would control it. It is also unclear who will do the builds- the package maintainer, another volunteer, or a CI system. The latter would be preferred by the RFC, if a volunteer can be found to maintain the CI system for windows builds. Github integration would be great. If builds are done by the package maintainer, github release would work. If it is to be done by a third party, then it is unclear where it could be efficiently uploaded. - -- Some kind of "confirmed working on OSX/Windows/Linux" feature would be nice, since package maintainers are unlikely to be able to test on three platforms. That would make it more obvious how well packages are tested. This would have to happen *after* phase 3 though. +- Automated source build as primary method of distribution, because it is impossible for users without root access +- Manual binary build by quark developers. It is possible with good tagging a la Pure Data's Deken, but too much burden on developers and bad security practice +- Depending on platform specific package managers for library dependencies. Installing these package managers usually requires root access # Drawbacks -This approach defers some questions to a later phase, however that seems hard to avoid given the scope is so large. - +- The automatic build system requires maintenance +- Quark update is required after upgrading or switching platform or supercollider version +- Manual build required for unsupported systems. # Unresolved Questions -- This plan of action attempts to reconcile all points raised in discussion for the end result. Has this been successful, or should further points be addressed? +- Should the binary quark system be implemented as part of supercollider proper, or be installable as a quark of its own that is depended on by quarks that require it? + - If part of supercollider, should servers be modified to load binary artifacts from the quark folders instead of +- Do administrators of systems without root access, such as at universities, prohibit installing binary artifacts as a non-root user? +--- +Initial discussion on the sc-dev mailing list: Preliminary [discussion on the sc-dev mailing list](https://www.listarc.bham.ac.uk/lists/sc-dev/thrd10.html#59665) Seperate related [discussion on the sc-dev mailing list regarding the sc3-plugins repository](https://www.listarc.bham.ac.uk/lists/sc-dev/msg58832.html)