add matrix.org homeserver experimental features policy #3089
Conversation
source: https://gist.github.com/richvdh/d7e83516207efbe9b31c4e2529a4132d Signed-off-by: HarHarLinks <2803622+HarHarLinks@users.noreply.github.com>
Signed-off-by: HarHarLinks <2803622+HarHarLinks@users.noreply.github.com>
Signed-off-by: HarHarLinks <2803622+HarHarLinks@users.noreply.github.com>
HarHarLinks
force-pushed
the
HarHarLinks/homeserver-experimental-features
branch
from
f662782 to
a26c51f
Compare
Deploying matrix-website with
|
| Latest commit: |
2e3cfbf
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://f944320a.matrix-website.pages.dev |
| Branch Preview URL: | https://harharlinks-homeserver-exper.matrix-website.pages.dev |
| The matrix.org homeserver is a public Matrix online service that can be used by anyone to interact with the public Matrix federation. | ||
| It is provided by [Element Creations Ltd](https://element.io) on behalf of The Matrix.org Foundation and consists of multiple software components. | ||
|
|
||
| [Synapse](https://github.com/element-hq/synapse)/[Synapse Pro](https://element.io/server-suite/synapse-pro), [Sydent](https://github.com/element-hq/sydent) and [MAS](https://github.com/element-hq/matrix-authentication-service) implement the APIs described by the [Matrix Specification](https://spec.matrix.org). |
There was a problem hiding this comment.
Is m.org running Synapse pro these days? The API doesn't report it:
curl https://matrix-federation.matrix.org/_matrix/federation/v1/version
{"server":{"name":"Synapse","version":"1.144.0 (b=matrix-org-hotfixes-priv,20614cf2e5)"}}
(How) can the public see said hotfixes?
There was a problem hiding this comment.
Is m.org running Synapse pro these days?
Synapse Pro currently consists mostly of worker processes rewritten in rust. As such, yes, m.org is using some components from Synapse Pro: you just won't see them by hitting /_matrix/federation/v1/version because that's still served by (almost) the FOSS python impl.
(How) can the public see said hotfixes?
Good question. @erikjohnston, @anoadragon453: now that room v12 is out and public, is there any reason for matrix.org to still be tracking the private repository?
There was a problem hiding this comment.
now that room v12 is out and public, is there any reason for matrix.org to still be tracking the private repository?
Not currently, though the team is hesitant to switch back to a public branch for a couple reasons:
- If we need to do another coordinated security release, we'd need to take the branch private again, which takes time (updating internal documentation, updating tooling).
- Not to mention doing so is a fairly strong signal that there is an active security issue :)
- matrix.org is (very slowly) moving towards just running ESS Pro (it's currently Synapse Pro + matrix.org-specific patches + on bare metal, no k8s). In an ideal world, it'd be k8s ESS Pro, with matrix.org-specific patches.
Needless to say, we don't want to "hoard patches". Each one puts a maintenance burden on the team due to git conflicts each week when we update from upstream. It's better for us to port those patches to FOSS or Synapse Pro, than have them sit on a matrix.org-specific branch. In a really ideal world, matrix.org wouldn't even have its own branch, as neither FOSS nor ESS customers really benefit from it.
The matrix-org-hotfixes-priv branch currently contains:
- A bunch of small places where we've updated constants to alleviate some sort of incident.
- Some commented out code which was causing intense resource usage.
- Some (inefficient) code to send server-wide notices to user that we were updating our terms and conditions. It's horrible, you really shouldn't copy it.
All stuff that really should either be removed or reviewed and merged upstream. It's not stuff that'd be useful to directly copy to another homeserver. It may be useful as a point of reference, and I have seen people use it for that in the past...
But with heading towards k8s, and aiming to mostly eliminate this branch, it makes sense to keep it private for now (while we work to trim it way down). Which initially pained me to write - but given the above, it makes sense to me.
Happy to share other aspects of how the homeserver is operated; details on how it's configured, things we look at during incidents, policies for putting unstable content on it, etc. etc. And if matrix.org appears to be doing something odd, happy to answer questions regarding the code it's running in Element Town Square.
| - how we will safely evolve the feature if it turns out more changes are needed to the API. | ||
| - how we will safely remove the `/unstable` prefixes once the feature is stabilised. | ||
|
|
||
| We'll generally want to have a written document addressing these concerns, to make sure that everyone is working on the same understanding. |
There was a problem hiding this comment.
Would it be possible to share these publicly?
There was a problem hiding this comment.
Probably. Generally I'd expect them to end up in a github issue somewhere.
There was a problem hiding this comment.
Is there anything I can do to help this get done?
|
|
||
| If, having considered the risks, we still want to roll out the feature further, the next step might be to roll it out to a larger subset of users on `matrix.org`, e.g. as part of a community testing initiative. | ||
|
|
||
| Enabling the feature for *all* users on `matrix.org` is a last resort. |
There was a problem hiding this comment.
Is there transparency that allows me to learn the current status at any time? If not, how can we achieve it?
Per the spec, one can query https://matrix-client.matrix.org/_matrix/client/versions to obtain a list of unstable_features that are supported and whether they are individually enabled or not. However this list is a bit hard to read? E.g. it is well-known that MSC4186: Simplified Sliding Sync is enabled both on m.org and by default, but it appears there as org.matrix.simplified_msc3575 which is not too obvious. Ignoring this, is it complete? What about experimental federation APIs?
There was a problem hiding this comment.
It's not currently exposed publicly, no. For obvious reasons (the fact it contains secret keys, for a start), the synapse configuration isn't public.
You could consider whether there is a way to expose the experimental_features config somehow, but even then I'd be worried that it would be easy for it to end up exposing information that should remain secret.
Honestly I think you need to consider whether the work involved in exposing this is worthwhile: what do you hope to gain from it, and are you sure that it is the most important thing for the backend team to be working on?
There was a problem hiding this comment.
May be better to create a /_synapse/enabled_experimental_features endpoint in Synapse and expose that, instead of manually updating a doc and trying to keep that in sync. PR's welcome!
There was a problem hiding this comment.
May be better to create a
/_synapse/enabled_experimental_featuresendpoint in Synapse and expose that, instead of manually updating a doc and trying to keep that in sync. PR's welcome!
Why not just include everything in /versions?
| In general therefore, it's much preferred to do the bulk of testing on non–`matrix.org` servers. | ||
| Both `beta.matrix.org` (for public testing) and internal servers are good candidates for doing so. | ||
| This helps avoid many of the above issues. | ||
| (TODO See [more on beta.matrix](https://handbook.element.io/new-vector/internal/-/wikis/Beta.matrix.org).) |
There was a problem hiding this comment.
Is this something also potentially worth publishing?
There was a problem hiding this comment.
it's mostly ops stuff: how to push out changes, etc. It's not really worth publishing, no.
There was a problem hiding this comment.
Ah, I was hoping for general info about what the Foundation offers at beta.matrix.org, perhaps to go into a "for devs" page under homeserver or some such. The goal for beta.matrix.org isn't very clearly defined beyond what this document says, though perhaps intentionally, and it would be communicated through the blog when people are supposed to test something.
There was a problem hiding this comment.
Currently this homeserver instance is used to allow Element app devs to test a feature that we wouldn't otherwise enable on matrix.org due to risk of clients relying on the feature.
We could open that up to be a place where any developer can request an unstable feature be hosted to test against. But that'd be something Element would need to consciously commit to maintaining.
If we decide to stick with the current purpose, it should probably be under the .element.dev domain instead...
| One way to evaluate experimental features is via a feature branch, but long-lived branches are difficult to manage, so a preferable solution is often to merge it to the project's development mainline, but leave it **disabled by default**. | ||
|
|
||
| Element's Backend team is normally happy to accept pull requests for experimental features behind a configuration flag, so long as the changes meet the code quality standards and the feature is in active development. | ||
| (Though note that the team may have to remove experimental features at short notice if they are causing problems.) | ||
|
|
||
| Once the feature is available in mainline, it can be enabled on individual servers for testing and evaluation. |
There was a problem hiding this comment.
General comment on this doc, and this bit in particular: it feels like it's going a long way beyond its remit of documenting matrix.orgs position with respect to experimental features.
I think, if you want something to put under https://matrix.org/homeserver/, what you should be creating here is quite a different document to the one that I shared with you and you used as a basis for this.
That document was written to answer the question "hi I have an exciting feature, where am I allowed to deploy it", which means that it spends a lot of time talking about the early stages of a feature which shouldn't concern matrix.org at all.
Instead, I think you want to come at this from the opposite direction: "here's what is allowed on matrix.org, here's what's definitely not allowed, and here's what we'll want your signature in blood that we can turn it off in 3 months. And here are the reasons we require those things." Maybe you could also throw in some pointers to other ways to try out functionality, but I don't think it should be front and centre.
Description
This adds information about experimental features on matrix.org, based on the respective internal document from Element shared via https://gist.github.com/richvdh/d7e83516207efbe9b31c4e2529a4132d for this purpose. I'm not sure it is clear enough to call it a policy.
This PR should be reviewable by commit: you can follow the evolution from the original upstream document.
Some questions remain (annotated inline).
Related issues
Resolves #2996
Role
🎩 Website & Content WG
Timeline
Signoff
Please sign off your individual commits or whole pull request.