Prometheus+Grafana I heard that the big shots are using this to monitor flink metrics.

Architecture#

Simple drawing and shallow understanding

Flink has a built-in plugin for interacting with Prometheus in the native environment.
Prometheus is used for metric collection.
- Node-exporter is used.
- Pushgateway is used.
Grafana is mainly for personalized dashboards. Although Prometheus pages can be viewed, they are not aesthetically pleasing.

Environment#

Environment/Component	Version
Operating System	CentOS Linux release 7.9.2009 (Core)
Docker	Docker version 1.13.1
Flink	1.13.6 on docker

Deployment#

Prometheus+Grafana are all deployed on Docker.

Preparation

In Linux:

mkdir /opt/grafana-storage  # Grafana data storage directory
chmod -R 777 /opt/grafana-storage
vim /opt/prometheus/prometheus.yml   # Create and edit

prometheus.yml (Don't forget to modify the IP)

global:
  scrape_interval:     60s
  evaluation_interval: 60s
 
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus
 
  - job_name: linux
    static_configs:
      - targets: ['ip:9100']
        labels:
          instance: localhost
  - job_name: 'pushgateway'
    static_configs:
      - targets: ['ip:9091']
        labels:
          instance: 'pushgateway'

Add to the Flink configuration file:

metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
metrics.reporter.promgateway.host: ip
metrics.reporter.promgateway.port: 9091
metrics.reporter.promgateway.jobName: flink
metrics.reporter.promgateway.randomJobNameSuffix: true
metrics.reporter.promgateway.deleteOnShutdown: true

Deployment

mkdir /opt/grafana-storage
chmod -R 777 /opt/grafana-storage

docker run -d -p 3000:3000  --name=grafana   -v /opt/grafana-storage:/var/lib/grafana   grafana/grafana


docker run -d -p 9100:9100  -v "/proc:/host/proc:ro"  -v "/sys:/host/sys:ro"  -v "/:/rootfs:ro"  --net="host"  prom/node-exporter


docker run -d -p 9090:9090  -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml  prom/prometheus

docker run -d -p 9091:9091 prom/pushgateway