Introduction#

GitHub this section walk through steps to step up Prometheus

  • Prometheus components and methods to setup
  • Setup the EBS CSI Driver add-on with service account here
  • Setup Prometheus and Grafana using helm chart

Components of Prometheus#

Check docs

  • Prometheus server
  • Alert manager
  • Pushgateway
  • Node exporter
  • PromQL, PrometheusUI, Grafana, API Clients

Setup Prometheus#

Prerequisite: setup the EBS CSI Driver add-on which Kubernetes will call AWS EBS service on behalf of you. First, install the add-on from the AWS EKS console, add-on management.

  • The add-on already created a service account named "xxx"
  • Create an IAM role for the service account
  • Annotate the service account with role arn
  • Follow detail here service account

There are several ways to setup monitoring with Prometheus, please read docs.

The easiest way is to use Prometheus community helm chart. First, add the repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

List charts from the repository

helm search repo prometheus-community

Then install the Prometheus community helm chart with custom configuration

helm install my-prometheus prometheus-community/prometheus -f ./test/prometheus_values.yaml

There are two methods for metric collectioin configuration

  • Via ServiceMonitor and PodMonitor in Prometheus Operator HERE
  • Via scrape_configs in prometheus.yaml HERE

Forward port to see Prometheus server UI

kubectl port-forward deploy/prometheus-server 8080:9090 -n prometheus

First query with Prometheus

sum by (namespace) (kube_pod_info)

Prometheus and Granfana#

To install both Prometheus and Grafana, choose another release

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack -f ./test/prometheus_values.yaml

Then port-forward to login the Grafana UI

kubectl port-forward deploy/prometheus-grafana 8081:3000 -n prometheus

Find the password to login Grafana

kubectl get secret --namespace prometheus prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Login Grafana UI, and go to the menu button, find

  • Dashboard and select Kubernetes/Compute Resources/ Pod and see
  • Explore, select code, and query with PromQL

Reference#