kubernetes unit tests fail

[unclejack]

There are some issues with the kubernetes unit tests. These tests fail one time out of about 10-15 CI runs.

I've encountered this issue with #615. I thought those issues were caused by my changes, but the same failures were encountered for many other PRs.

[jojimt]

Logs for the failure?

[unclejack]

I don't have the logs. I've seen them in the CI logs of a run. That run's CI logs are gone now because they get deleted automatically.

I'll keep an eye on the CI to see if it runs into this failure again.

[unclejack]

I couldn't reproduce the failures I've seen on the CI. I'm fixing the issues directly.

[unclejack]

A job on the CI has run into this failure:

time="2017-02-20T08:19:28Z" level=info msg="Output from rename: []" 
time="2017-02-20T08:19:28Z" level=info msg="Output from ip assign: []" 
time="2017-02-20T08:19:29Z" level=info msg="stat : Active Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    \ntcp        0      0 192.168.2.10:8300       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:2380       0.0.0.0:*               LISTEN      4945/etcd           \ntcp        0      0 192.168.2.10:8301       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:8302       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:6640            0.0.0.0:*               LISTEN      4791/ovsdb-server   \ntcp        0      0 127.0.0.1:8400          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      993/sshd            \ntcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1698/master         \ntcp6       0      0 :::2379                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::22                   :::*                    LISTEN      993/sshd            \ntcp6       0      0 ::1:25                  :::*                    LISTEN      1698/master         \ntcp6       0      0 :::4001                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::2375                 :::*                    LISTEN      4588/docker         \n" 
time="2017-02-20T08:19:30Z" level=info msg="stat : Active Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    \ntcp        0      0 192.168.2.10:8300       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:2380       0.0.0.0:*               LISTEN      4945/etcd           \ntcp        0      0 192.168.2.10:8301       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:8302       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:6640            0.0.0.0:*               LISTEN      4791/ovsdb-server   \ntcp        0      0 127.0.0.1:8400          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      993/sshd            \ntcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1698/master         \ntcp6       0      0 :::2379                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::22                   :::*                    LISTEN      993/sshd            \ntcp6       0      0 ::1:25                  :::*                    LISTEN      1698/master         \ntcp6       0      0 :::4001                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::2375                 :::*                    LISTEN      4588/docker         \n" 
time="2017-02-20T08:19:31Z" level=info msg="stat : Active Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    \ntcp        0      0 192.168.2.10:8300       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:2380       0.0.0.0:*               LISTEN      4945/etcd           \ntcp        0      0 192.168.2.10:8301       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:8302       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:6640            0.0.0.0:*               LISTEN      4791/ovsdb-server   \ntcp        0      0 127.0.0.1:8400          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      993/sshd            \ntcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1698/master         \ntcp6       0      0 :::2379                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::22                   :::*                    LISTEN      993/sshd            \ntcp6       0      0 ::1:25                  :::*                    LISTEN      1698/master         \ntcp6       0      0 :::4001                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::2375                 :::*                    LISTEN      4588/docker         \n" 
time="2017-02-20T08:19:32Z" level=info msg="stat : Active Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    \ntcp        0      0 192.168.2.10:8300       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:2380       0.0.0.0:*               LISTEN      4945/etcd           \ntcp        0      0 192.168.2.10:8301       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:8302       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:6640            0.0.0.0:*               LISTEN      4791/ovsdb-server   \ntcp        0      0 127.0.0.1:8400          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      993/sshd            \ntcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1698/master         \ntcp6       0      0 :::2379                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::22                   :::*                    LISTEN      993/sshd            \ntcp6       0      0 ::1:25                  :::*                    LISTEN      1698/master         \ntcp6       0      0 :::4001                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::2375                 :::*                    LISTEN      4588/docker         \n" 
time="2017-02-20T08:19:33Z" level=info msg="stat : Active Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    \ntcp        0      0 192.168.2.10:8300       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:2380       0.0.0.0:*               LISTEN      4945/etcd           \ntcp        0      0 192.168.2.10:8301       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 192.168.2.10:8302       0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:6640            0.0.0.0:*               LISTEN      4791/ovsdb-server   \ntcp        0      0 127.0.0.1:8400          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      993/sshd            \ntcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      4953/consul         \ntcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1698/master         \ntcp6       0      0 :::2379                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::22                   :::*                    LISTEN      993/sshd            \ntcp6       0      0 ::1:25                  :::*                    LISTEN      1698/master         \ntcp6       0      0 :::4001                 :::*                    LISTEN      4945/etcd           \ntcp6       0      0 :::2375                 :::*                    LISTEN      4588/docker         \n" 
time="2017-02-20T08:19:33Z" level=fatal msg="Kube server not ready after 5 sec" 
exit status 1
FAIL	github.com/contiv/netplugin/mgmtfn/k8splugin	5.287s

[unclejack]

Here's another failure. This one is different.

FAIL: kubeClient_test.go:494: TestKube.TestKubeWatch

--ADD--
Drv: &{numAddSvc:0 numDelSvc:0 numProvUpd:0 services:map[] providers:map[]}
kubeClient_test.go:512:
    c.Errorf("Add service was not seen by client")
... Error: Add service was not seen by client

kubeClient_test.go:517:
    verifySvc(c, drv)
kubeClient_test.go:454:
    c.Errorf("Service was not correctly updated on client")
... Error: Service was not correctly updated on client

--DEL--
Drv: &{numAddSvc:0 numDelSvc:0 numProvUpd:0 services:map[] providers:map[]}
kubeClient_test.go:532:
    c.Errorf("Del service was not seen by client")
... Error: Del service was not seen by client

--CLOSE--
Drv: &{numAddSvc:0 numDelSvc:0 numProvUpd:6 services:map[] providers:map[LipService:[123.45.67.89 123.45.67.88]]}
services: <nil>
kubeClient_test.go:551:
    c.Errorf("All updates were not seen by client")
... Error: All updates were not seen by client

kubeClient_test.go:554:
    verifySvc(c, drv)
kubeClient_test.go:454:
    c.Errorf("Service was not correctly updated on client")
... Error: Service was not correctly updated on client

--- FAIL: Test (13.21s)
FAIL
OOPS: 1 passed, 1 FAILED
exit status 1
FAIL	github.com/contiv/netplugin/mgmtfn/k8splugin	13.241s

[unclejack]

This is still a problem.

[unclejack]

I'm reopening this issue because I've seen it again on the CI.