Added new metric node_agent_collected_metrics along with new internal prom registry

[vishnukumarkvs]

Hi,

To view the amount of metrics a coroot node agent scrapes, I have added a new prometheus metric node_agent_collected_metrics (Type Gauge) to calculate the amount of series it scrapes in a scrape interval.

I have also created a new metrics endpoint on a new port (default: 9093) to differentiate between agent internal metrics and the agent's ebpf scraped metrics. So, prometheus can scrape only agents internal metrics like node_agent_info, collected_metrics etc and skip ebpf metrics(which we send via http push to coroot)

[def]

I don't think we need this as a separate metric, especially on a dedicated endpoint. Also, this metric isn't very actionable, so even if you see a high number of metrics, it's not clear what to do next.

We could instead log some metric statistics in the remote writer, for example the top 10 metric names by number of series.

To understand how many metrics each machine is writing to Prometheus, you can use a query like:

count by (machine_id) ({machine_id!=""})