[RFC 0189] Contracts

[ibizaman]

Rendered - Draft PR 1 - Draft PR 2

This PR introduces the contracts RFC. This follows the discussion on the pre-RFC discourse post.

[KiaraGrouwstra]

Thanks for filing. I would like to join here to shepherd this.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-decouple-services-using-structured-typing/58257/42

[KiaraGrouwstra]

maybe for context we could link 'set the provider option' to the lines in question

[7c6f434c]

Standard question for any module-system-based increase in layers: what is the estimated evaluation-time resource impact?

[ibizaman]

That’s a great question. Is there a standard set of benchmarking tooling? After some quick online search, I see some external repos and wiki guides but I was wondering how from scratch I should start from.

[7c6f434c]

Good question; I don't actually know things beyond export NIX_SHOW_STATS=1. There are some scripts in ci/eval but I am not sure whether reusing parts of them is worth it (but maybe looking at what is done there could be useful)

[KiaraGrouwstra]

maybe one way to compare would be getting such stats after/before the PR for say that nextcloud test, like: NIX_SHOW_STATS=1 nix-build --impure nixos/tests/nextcloud -A basic31.

PR

{
"cpuTime": 0.1535560041666031,
"envs": {
  "bytes": 2400992,
  "elements": 167207,
  "number": 132917
},
"gc": {
  "heapSize": 402915328,
  "totalBytes": 68045680
},
"list": {
  "bytes": 288200,
  "concats": 1487,
  "elements": 36025
},
"nrAvoided": 133723,
"nrFunctionCalls": 122934,
"nrLookups": 66643,
"nrOpUpdateValuesCopied": 1889724,
"nrOpUpdates": 6747,
"nrPrimOpCalls": 60323,
"nrThunks": 171874,
"sets": {
  "bytes": 35799712,
  "elements": 2221018,
  "number": 16464
},
"symbols": {
  "bytes": 335942,
  "number": 34889
},
"values": {
  "bytes": 18234936,
  "number": 759789
}
}

parent revision

{
"cpuTime": 0.1342640072107315,
"envs": {
  "bytes": 2304512,
  "elements": 160244,
  "number": 127820
},
"gc": {
  "heapSize": 402915328,
  "totalBytes": 67791424
},
"list": {
  "bytes": 277016,
  "concats": 1378,
  "elements": 34627
},
"nrAvoided": 128713,
"nrFunctionCalls": 118695,
"nrLookups": 63610,
"nrOpUpdateValuesCopied": 1915086,
"nrOpUpdates": 6556,
"nrPrimOpCalls": 58429,
"nrThunks": 161589,
"sets": {
  "bytes": 36067568,
  "elements": 2239118,
  "number": 15105
},
"symbols": {
  "bytes": 299281,
  "number": 30518
},
"values": {
  "bytes": 17928864,
  "number": 747036
}
}

i'm not sure which metrics are interesting (note cpuTime is unstable across invocations) or if we should compare for a different build attribute.
building the nextcloud module (nixos/modules/services/web-apps/nextcloud.nix) instead, for what it's worth, i didn't really get it to show differences beside on a symbols metric.

[axelkar]

This is confusing. Is this what you meant?

Suggested change

	is if we only find one `consumer` and `provider` pair in the whole of nixpkgs.
	is if we only find one `consumer` and one `provider` usage of a contract in Nixpkgs.

[ibizaman]

Yes indeed thats what I meant :)

[axelkar]

The dual-linking approach strikes dangerously close to infinite recursion in my mind. What do you think?

[KiaraGrouwstra]

in my understanding the reason the current implementation works is lazy evaluation

[ibizaman]

Indeed, it's pretty close. It's working because we're linking two different options, depending on the direction.

I'll argue I'm not the first to have though of this pattern. Let's take the secrets contracts as an example. Remember, it looks like so:

services.myservice.secret.provider = config.sops.secrets."myservice/secret".provider;

sops.secrets."myservice/secret".consumer = services.myservice.secret.consumer;

If we compare to the sops-nix module, if we squint we can see the dual linking is nearly there:

services.myservice.secret = config.sops.secrets."myservice/secret".path;

sops.secrets."myservice/secret" = {};

Btw, you would get the infinite recursion if you removed the .consumer or .provider from the snippet above.

[KiaraGrouwstra]

i mean, it works until you try and serialize the whole structure i guess 😅

[axelkar]

Is there an example for this? The contract system will also probably need integration with service managers for scheduling and bringing up dependencies.

[ibizaman]

I can’t believe I forgot to add an example for this IMO important point. I’ll add that.

The intent here is to provide an escape hatch to the outside (of NixOS) world. Which means you’re free to do whatever you want. The use case I had in mind does imply “manual” scheduling. Let’s say you want to integrate a NixOS service with a database that exists already on another server which does not run NixOS. You would create a database provider which hardcodes the url and credentials to the other server. Then the consumer would be able to access it. Of course you must set up the other server beforehand correctly. And of course the other server could be a NixOS machine too.

I can see a separate contract for interacting with service management. This should be merged with the module interfaces initiative.

[ibizaman]

A good example of this would be for a database contract. The options could be:

Filled out by the provider:

• hostname
• database name
• username
• password

I actually can't think of options filled out by the consumer. Maybe extensions needed in the database? Charset?

Usually, you would have a provider like the postgresql module that you would create a database in with the (unfortunately but understandably controversial) ensureUser and ensureDatabase options. Then, you would plug in the new user and database to the consumer that requests a database.

But this would work too outside of the NixOS ecosystem. You could just create a user and database in any system then, inside the NixOS system that has the consumer, just hardcode the 4 options with strings that would point to the other system.

Does that make sense?

[axelkar]

I believe those fields are enough for most configurations, though the password should be specified using passwordFile (or a secret management contract!).

[ibizaman]

That’s spot on @axelkar. I should’ve been more precise. In my project I indeed use a contract for the password itself so it’s not the plain password.

[gytis-ivaskevicius]

What about doing something along the lines of this:

config = {
  
  # each service coming from submodule generator units.xyz is where the magic happens, its fully isolated. I would expect it to run its own modules evaluator in the background, should even help with eval time
  services.nextcloud = units.nextcloud {
     # ... module config ...
     fileBackup.provider = config.services.restic-for-nextcloud.fileBackup;
  };
   
  # Service name does not matter
  services.restic-for-nextcloud = units.restic {
    // Provider-specific options.
    repository = "/var/lib/backups/nextcloud";
    passwordFile = toString (pkgs.writeText "password" "password");
    initialize = true;
  };
};

This way we make modules more portable, contract itself is with root module system and units.xyz which we could implement to support other projects such as home-manager or devshells. and integration between modules is dependent on what other modules explicitly export and ofcourse where user explicitly passes values

Tho there might be cases of infinite recursion sometimes, tho it should be very rare

I think of this as Terraform modules, they have certain inputs and certain outputs - everything else is a black box

[KiaraGrouwstra]

this also reminds me a bit of portable services, and reusing those for the existing modules to prevent duplication.
this example does sound easily extendible in terms of adding new providers - @ibizaman would probably know more about how we were doing there so far.
avoiding dual linkage maybe seems an open question, but yeah, let's try stuff.

[ibizaman]

Same, makes me think of portable services.

I fully agree we should tend to this and it seems indeed compatible with contracts. I would argue though it is orthogonal and a large effort that should be separate from this.

I'm definitely not going to stop anyone from attempting both at the same time but I can't see myself doing it currently.

[ibizaman]

FYI I created a new draft PR with an alternative implementation which is IMO better than the first. It has less weird quirks at least. I added a comparison in the description of the PR. NixOS/nixpkgs#485453

[Pamplemousse]

To me, this proposal is screaming "it is a convention to do dependency injection consistently for modules".
"Dependency injection" is a mean to achieve dependency inversion that could (should?) be very familiar to developpers.

It is mentionned in the Prior Art section, but I feel that it could be mentionned here, and even appear earlier (as early as in Motivation).

[ibizaman]

I didn’t see it at first but it’s evident to my too now

[mrVanDalo]

I hope this is a constructive comment

[mrVanDalo]

In my understanding consumer and provider are switched here.

• nextcloud don't need a fileBackup, it provides a fileBackup (I assume these are files that should be backed up). => nextcloud is a provider
• but restic (and borgBackup) need to know which files to backup (fileBackup) so they would consume these files. => restic is a consumer

I would also argue: a provider provides only one thing but a consumer can consume multiple things.

Maybe this is what is meant here, and I mix up the syntax (or the object naming could be updated), but I couldn't see descriptions on your option definition up there.

[mrVanDalo]

First of all sorry, if this was already discussed. I read the last hour the discusions specifically on this topic, but couldn't find anything. But I might have missed it.

Anyway I'm a huge dependency injection fan. But I don't see to much benefit on your example here.
Usually dependency injection comes with a injector which handles the wiring for you, which is one of the benefits.
But we have to make the wireing here our self.
The nixos module system already comes with options which can be rewired on demand where ever you like
(outside the module definitions),
which is not possible in programming languages like java or go.
For example your example currently in nixos could be done like this

services.restic.bacups.nextcloud.paths = [config.services.nextcloud.datadir]

This approach don't lack any aspect to using the contracts (At least from my perspective).

The one benefit I see from the contracts concept is that we create a transactional type system, which standardizes service plumbing. Which is not nothing.

If you would tell me nixos module is capable of dependency injection, I would imaging something like

options.services.nextcloud.datadir = mkProvider fileBackup {
# same like mkOption but type is dictated by fileBackup (e.g.: 2 fileBackup options can't have different types)
};
...
config = .. # nextcloud configuration using datadir like now

than you could set.

serivces.restic.backup.dependencyInjection.paths = allProvidersOf fileBackup;

In this example allProvidersOf <key/type> will be handled by the dependency injector and
creates a list of config-references.

services.restic.backup.dependencyInjection.paths = [config.services.nextcloud.datadir];

This is because I expect automatic wireing when I hear dependency injection = inversion of control.

I sketched some scenarios (e.g.: reverse proxy -> webapp -> database connetion <- credentials) and I come always to the same conclusion. But I could see a benefit in the credentials part with this approach, but this is not part of you RFC if I understood correctly.

I hope this is helpful.

p.s.: I would be happy for a proper solution here, I faced a similar problem like in your example and solved it like this : https://gist.github.com/mrVanDalo/a1f7ec5cce80d69ac60b6d22feb6d768 (a home manager interface which handles files I need to backup)

[KiaraGrouwstra]

thanks for your response!

i think the dual link here was maybe less intended design, more like a limitation of what we'd managed with the implementation at the time. #506343 addresses that.

mkProvider fileBackup { ... }

the limitation of functions is that they are pure, whereas going thru the module system ensures that we can use the passed information to set config as well.

In this example allProvidersOf <key/type> will be handled by the dependency injector and creates a list of config-references.

this type of aggregation is basically what #506343 does, resulting in the defaultProvider(Name) to let one wire things up.

on naming, i def had trouble picking things that made sense as well. in the case of secrets, for one, i sort of made sense out of this by speaking of an application requesting a secret, that another module might then provide.
in an earlier iteration, i played with terms like input and output, but then ran into similar issues about perspective as you noted here.