Monitor custom kubernetes pod metrics using Prometheus

Dinesh Ahuja

I am using Prometheus to monitor my Kubernetes cluster. I have set up Prometheus in a separate namespace. I have multiple namespaces and multiple pods are running. Each pod container exposes a custom metrics at this end point, :80/data/metrics . I am getting the Pods CPU, memory metrics etc, but how to configure Prometheus to pull data from :80/data/metrics in each available pod ? I have used this tutorial to set up Prometheus, Link

Emruz Hossain

You have to add this three annotation to your pods:

prometheus.io/scrape: 'true'
prometheus.io/path: '/data/metrics'
prometheus.io/port: '80'

How it will work?

Look at the kubernetes-pods job of config-map.yaml you are using to configure prometheus,

- job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name

Check this three relabel configuration

- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__

Here, __metrics_path__ and port and whether to scrap metrics from this pod are being read from pod annotations.

For, more details on how to configure Prometheus see here.

この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。

侵害の場合は、連絡してください[email protected]

編集
0

コメントを追加

0

関連記事

分類Dev

prometheus_client custom metrics with Flask

分類Dev

Prometheus with multi-container pod on kubernetes

分類Dev

Monitor Scylla db using Prometheus and Grafana

分類Dev

Custom metrics with keras using oop

分類Dev

GKE - HPA using custom metrics - unable to fetch metrics

分類Dev

How to expose webClient metrics in prometheus?

分類Dev

How to expose webClient metrics in prometheus?

分類Dev

How can I scrape label value using prometheus query from metrics and set alerts

分類Dev

How to map one single file into kubernetes pod using hostPath?

分類Dev

Custom metrics with tf.estimator

分類Dev

I want to ping (icmp) monitor the worker nodes that make up the kubernetes cluster without using the internal IP of the node

分類Dev

how to stop/pause a pod in kubernetes

分類Dev

Kubernetes pod auto-scaling

分類Dev

Kubernetes - Get EBS Volume of a Pod

分類Dev

Can volumeMounts: in kubernetes pod be optional

分類Dev

Kubernetes MLflow Service Pod Connection

分類Dev

Running bash script in a kubernetes pod

分類Dev

after upgrade to Spring Boot 2, how to expose cache metrics to prometheus?

分類Dev

Prometheus gauge with custom collector - go

分類Dev

"Connect: Connection Refused" when Connecting Prometheus to Kubernetes

分類Dev

Setting Up Prometheus with Python in Kubernetes Cluster

分類Dev

Azure kubernetes - Prometheus to handle multiple applications

分類Dev

Getting pod or deployment annotations from Kube-State-Metrics

分類Dev

Inject App Insights for Custom Events and Metrics

分類Dev

Azure Kubernetes-Azure MonitorとSidecarLogshipping?

分類Dev

Azure Kubernetes - not able to connect pod containers

分類Dev

Save file to Kubernetes pod during deployment

分類Dev

Two containers within same pod on Kubernetes

分類Dev

How to Configure Pod initialization in a specific order in Kubernetes?

Related 関連記事

  1. 1

    prometheus_client custom metrics with Flask

  2. 2

    Prometheus with multi-container pod on kubernetes

  3. 3

    Monitor Scylla db using Prometheus and Grafana

  4. 4

    Custom metrics with keras using oop

  5. 5

    GKE - HPA using custom metrics - unable to fetch metrics

  6. 6

    How to expose webClient metrics in prometheus?

  7. 7

    How to expose webClient metrics in prometheus?

  8. 8

    How can I scrape label value using prometheus query from metrics and set alerts

  9. 9

    How to map one single file into kubernetes pod using hostPath?

  10. 10

    Custom metrics with tf.estimator

  11. 11

    I want to ping (icmp) monitor the worker nodes that make up the kubernetes cluster without using the internal IP of the node

  12. 12

    how to stop/pause a pod in kubernetes

  13. 13

    Kubernetes pod auto-scaling

  14. 14

    Kubernetes - Get EBS Volume of a Pod

  15. 15

    Can volumeMounts: in kubernetes pod be optional

  16. 16

    Kubernetes MLflow Service Pod Connection

  17. 17

    Running bash script in a kubernetes pod

  18. 18

    after upgrade to Spring Boot 2, how to expose cache metrics to prometheus?

  19. 19

    Prometheus gauge with custom collector - go

  20. 20

    "Connect: Connection Refused" when Connecting Prometheus to Kubernetes

  21. 21

    Setting Up Prometheus with Python in Kubernetes Cluster

  22. 22

    Azure kubernetes - Prometheus to handle multiple applications

  23. 23

    Getting pod or deployment annotations from Kube-State-Metrics

  24. 24

    Inject App Insights for Custom Events and Metrics

  25. 25

    Azure Kubernetes-Azure MonitorとSidecarLogshipping?

  26. 26

    Azure Kubernetes - not able to connect pod containers

  27. 27

    Save file to Kubernetes pod during deployment

  28. 28

    Two containers within same pod on Kubernetes

  29. 29

    How to Configure Pod initialization in a specific order in Kubernetes?

ホットタグ

アーカイブ