Architecture#
Simple drawing and shallow understanding
- Flink has a built-in plugin for interacting with Prometheus in the native environment.
- Prometheus is used for metric collection.
- Node-exporter is used.
- Pushgateway is used.
- Grafana is mainly for personalized dashboards. Although Prometheus pages can be viewed, they are not aesthetically pleasing.
Environment#
| Environment/Component | Version |
|---|---|
| Operating System | CentOS Linux release 7.9.2009 (Core) |
| Docker | Docker version 1.13.1 |
| Flink | 1.13.6 on docker |
Deployment#
Prometheus+Grafana are all deployed on Docker.
- Preparation
In Linux:
prometheus.yml (Don't forget to modify the IP)
Add to the Flink configuration file:
- Deployment
Start a Flink job for testing.
- Observation
Check the startup status of the Prometheus suite through the gateway (up means started)
http://ip:9090/targets#pool-pushgateway
Check the monitored Flink jobs through the gateway
http://ip:9091
Practice#
- Add Prometheus source to Grafana
Settings -> Add Prometheus
Add URL and click save.
- Add Grafana dashboard templates
Search for templates in the Grafana dashboard library and copy the ID.
https://grafana.com/grafana/dashboards/
Import the Grafana dashboard by adding the ID, which will be downloaded automatically.
Then adjust as needed.
Some templates may not have data, not sure why.
Next Steps#
- Configure monitoring indicators in the Flink process, such as benchmark QPS, data volume, and latency.