[prometheus monitoring ] expose metrics port for pod workers.

[zak-hassan]

Workers with metrics enabled it only exposes port 8081 and not the prometheus endpoint.

bash$ ./oshinko create test --metrics=prometheus
shared cluster "test" created
bash$ oc get pods
NAME             READY     STATUS    RESTARTS   AGE
test-m-1-42v74   1/1       Running   0          25s
test-w-1-79b7x   1/1       Running   0          25s
bash$ oc describe pod test-w-1-79b7x |grep "Port"
    Port:           8081/TCP

[rebeccaSimmonds19]

At the moment we expose metric ports on the master. At this moment we are unsure if we want to do this and create a service for each worker, especially with respect to larger clusters

[zak-hassan]

Hi,

I wanted to revisit this issue. Is it possible to include agent-bond jars in the s2i and include property files to expose metrics from the driver. @dfeddema had to do a workaround and include the agent-bond.jar binary into https://github.com/dfeddema/play spark demo to get it to expose the metrics. We also had to pass in the path to the jar. @elmiko @tmckayus @rebeccaSimmonds19 What do you think?

[elmiko]

hey Zak, i just want to make sure i've got the request. you want the agent-bond jar to be included in the driver pod from an s2i build? (only asking because the title mentions worker pods)

assuming you mean including in the driver, it seems reasonable and i don't think we include it currently. as for the property files, that would all be handled through ConfigMap's that would be specified at deployment time.

[zak-hassan]

@elmiko Yes that is correct. I have 2 questions:

• We wanted a way to pull metrics from the workers. The current work around was described in the comment above.
• We wanted a way when using s2i to be able to instrument the driver.

I'm available to jump on a call if need be? Contribute even? Thoughts?

[elmiko]

ok, so this is sounding like 2 issues actually.

1. expose metrics from the workers
2. expose metrics from the driver

we know that exposing from the workers is going to be messy until we have a nice workaround for using services on all the worker pods (eg using pod IPs or something).

exposing from the driver would be much easier as we already have services attached to that pod.

i'm just trying to explore the request here to make sure we capture all the requirements. the worker thing is still a pain, not sure the best way to proceed on that. but i'm curious to hear other opinions.

[willb]

If we include agent-bond.jar in the s2i-built images, we'll need some way to override it, because bundling dependencies inevitably means that your bundled dependencies will conflict with a user's desires and requirements. So I'd vote for making extra jars configurable in the s2i (via Maven or Ivy coordinates/etc.) rather than just baking a particular version of a particular metrics agent into every driver (or cluster) we deploy, since we'll ultimately have to do that anyway.

[zak-hassan]

@willb +1. Good point. Yes. This would be a nice feature or even downloading the agen-bond.jar from somewhere maven central would be even okay: https://mvnrepository.com/artifact/io.fabric8/agent-bond-agent/0.1.0

[willb]

So, to be totally clear, you can already load agent-bond.jar from Maven Central and configure Spark to use it with no changes to the existing s2i builders, just by specifying it as an application dependency and supplying extra Spark configuration. However, I think it's worth considering whether or not it makes sense to support decoupling the infrastructure configuration and dependencies from the application configuration and dependencies, viz., by supporting a s2i-specific way to specify OpenShift/k8s-specific configuration and dependencies so that those aren't in the application's pom.xml file.