Reflector#start() does not retry infinitely

[ljnelson]

I am not entirely sure but I believe that the relevant Go code ends up retrying forever:

https://github.com/kubernetes/client-go/blob/dcf16a0f3b52098c3d4c1467b6c80c3e88ff65fb/tools/cache/reflector.go#L128-L137

But Reflector#start() will bomb out if it can't list things:

microbean-kubernetes-controller/src/main/java/org/microbean/kubernetes/controller/Reflector.java

Line 625 in ee05d9e

final KubernetesResourceList<? extends T> list = ((Listable<? extends KubernetesResourceList<? extends T>>)this.operation).list();

That line will throw a KubernetesClientException, and that's it. I think this whole method should (internally) be trying this list-and-watch loop forever.

For more details, see also the ListAndWatch function:

https://github.com/kubernetes/client-go/blob/dcf16a0f3b52098c3d4c1467b6c80c3e88ff65fb/tools/cache/reflector.go#L165-L275

Another way to put this is that currently Reflector#start() really just models ListAndWatch, but it should be modeling Run in reflector.go as well.

Complicating matters is the fact that the fabric8 client will automatically try to reconnect watches if they fail.

[ljnelson]

I believe this code:

https://github.com/kubernetes/client-go/blob/dcf16a0f3b52098c3d4c1467b6c80c3e88ff65fb/tools/cache/reflector.go#L226-L274

…is actually implemented by the fabric8 client, more or less:

https://github.com/fabric8io/kubernetes-client/blob/cee8550053e671bb0b3489e37196924f4c949a3f/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/WatchConnectionManager.java#L293-L333

…so the problem may simply reduce to the body of the start() method needing to run infinitely, i.e. to ensure that the listing behavior is also retried ad nauseam.

[ljnelson]

Ah, wait; this is already handled:

microbean-kubernetes-controller/src/main/java/org/microbean/kubernetes/controller/Reflector.java

Lines 1031 to 1057 in ee05d9e

	@Override
	public final void onClose(final KubernetesClientException exception) {
	final String cn = this.getClass().getName();
	final String mn = "onClose";
	if (logger.isLoggable(Level.FINER)) {
	logger.entering(cn, mn, exception);
	}
	synchronized (Reflector.this) {
	// No need to close it; we're being called because it's
	// closing.
	Reflector.this.watch = null;
	}
	if (exception != null) {
	if (logger.isLoggable(Level.WARNING)) {
	logger.logp(Level.WARNING, cn, mn, exception.getMessage(), exception);
	}
	// See
	// https://github.com/kubernetes/client-go/blob/5f85fe426e7aa3c1df401a7ae6c1ba837bd76be9/tools/cache/reflector.go#L204.
	try {
	Reflector.this.start();
	} catch (final IOException ioException) {
	if (logger.isLoggable(Level.SEVERE)) {
	logger.logp(Level.SEVERE, cn, mn, ioException.getMessage(), ioException);
	}
	}

There could be better logging here. I'll add logging and then close this issue.

[ljnelson]

Need to also review the Go code to see if the "full replace" operation occurs in the event of a failure-and-retry occurrence.