Implement the service controller

[sudomateo]

Implement the service controller within the Cloud Controller Manager to manage services of type LoadBalancer.

This one may be tricky since Oxide does not, at the time of this writing, have a load balancer offering. In lieu of that perhaps this service controller can do one of the following.

• Create an instance running HAProxy or Nginx with an external IP whenever a LoadBalancer service is created in Kubernetes.
• Update a floating IP to point to some healthy Kubernetes node. This isn't exactly a load balancer but it could be a proof of concept way to expose node ports externally until a load balancer exists in Oxide.
• Emit some sort of event when a LoadBalancer service is created that a user is meant to receive via automation to update some external load balancer.

Implementing the service controller without an Oxide load balancer present will still provide good learning for future iterations on the Cloud Controller Manager.

[sudomateo]

I spent some time prototyping this and I'm going to move forward with using Oxide floating IPs for this. It requires kube-proxy so that traffic received on the node with the floating IP attached can be routed to the ClusterIP for the service.

Oxide floating IPs are transparent to the instances. Specifically, there are two points to note here.

1. Traffic coming into a floating IP that's attached to the instance will have the destination IP updated to be the instance's internal IP, not the floating IP.
2. There's no network interface on the instance that is configured with the floating IP.

With those notes, here's the flow of traffic:

Client request to Floating IP (e.g., 172.21.252.201:80)
 ↓
Oxide translates destination IP to instance internal IP
 ↓
Traffic arrives at instance internal IP (e.g., 172.30.0.5:80)
 ↓
kube-proxy firewall rules route traffic to ClusterIP
 ↓
Traffic arrives to pods within the service

The key to getting all this to work is ensuring the service controller within the Cloud Controller Manager adds the instance's internal IP to status.loadBalancer.ingress as a VIP IP type. This will allow kube-proxy to create the necessary nftables rules to forward traffic to the ClusterIP. Unfortunately, that means users fetching the service details will see the instance's internal IP as an external IP. However, such is life when we don't currently have a load balancer offering. When that's implemented we'll update this service controller to use it instead.

> kubectl get service nginx
NAME    TYPE           CLUSTER-IP       EXTERNAL-IP                 PORT(S)        AGE
nginx   LoadBalancer   10.106.122.233   172.21.252.201,172.30.0.5   80:30605/TCP   37h