Now that we have deployed Prometheus Operator we can use it to launch an instance of Prometheus.
When we deployed the Operator, one of the tasks it performed when it first launched was to is install a number Custom Resource Definitions (CRDs) into Kubernetes. Out of the box, Kubernetes ships with many powerful Controllers such as the Deployment or Statefulset. CRDs provide a method of building completely bespoke Controllers that provide logic to a specific function. In this case, the CRDs installed by Prometheus Operator provide a means for launching and configuring Prometheus within Kubernetes.
If you run the kubectl get customresourcedefinitions command in your terminal you will see four CRDs provided by the Operator:
$kubectl get customresourcedefinitions
NAME CREATED AT
alertmanagers.monitoring.coreos.com 2019-07-02T13:13:21Z
prometheuses.monitoring.coreos.com 2019-07-02T13:13:21Z
prometheusrules.monitoring.coreos.com 2019-07-02T13:13:21Z
servicemonitors.monitoring.coreos.com 2019-07-02T13:13:21Z
To begin with we will be making use of the prometheuses.monitoring.coreos.com Custom Resource Definition.
Create a file called prometheus.yaml and add the following:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
namespace: prometheus
spec:
baseImage: quay.io/prometheus/prometheus
logLevel: info
podMetadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
labels:
app: prometheus
replicas: 1
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 2Gi
retention: 12h
serviceAccountName: prometheus-service-account
storage:
volumeClaimTemplate:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
version: v2.10.0
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: "prometheus-service-account"
namespace: "prometheus"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: "prometheus-cluster-role"
rules:
- apiGroups:
- ""
resources:
- nodes
- services
- endpoints
- pods
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/metrics
verbs:
- get
- nonResourceURLs:
- "/metrics"
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: "prometheus-cluster-role-binding"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: "prometheus-cluster-role"
subjects:
- kind: ServiceAccount
name: "prometheus-service-account"
namespace: prometheus
Now apply this YAML to your Kubernetes cluster using kubectl apply -f prometheus.yaml. Kubectl will show that it has successfully created the configuration, as shown below:
$kubectl apply -f prometheus.yaml
prometheus.monitoring.coreos.com/prometheus created
serviceaccount/prometheus-service-account created
clusterrole.rbac.authorization.k8s.io/prometheus-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-cluster-role-binding created
Success! If you now list Pods in the prometheus namespace using kubectl get pods --namespace prometheus you will see an instance of Prometheus running alongside the Prometheus Operator:
$kubectl get pods
NAME READY STATUS RESTARTS AGE
prometheus-operator-operator-86bc4d5568-shs69 1/1 Running 0 10m
prometheus-prometheus-0 3/3 Running 0 1m
If you now check for the custom Prometheus resource that you have just installed into the cluster using kubectl get prometheus --namespace prometheus you will see the single result named prometheus:
$kubectl get prometheus --namespace prometheus
NAME AGE
prometheus 6m
The Prometheus Operator acts as a Controller for the Custom Resources. When you deployed the Prometheus resource the Operator created the Prometheus instance, that you just identified when getting a list Pods in the prometheus namespace.
Lets now take a look at the prometheus.yaml file we applied to Kubernetes and see what each section means.
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
namespace: prometheus
Here we define that we wish to create an object called prometheus that is of the type Prometheus as defined by the Kind and that this Kind is part of the API Version monitoring.coreos.com/v1, that was previously installed into Kubernetes by the Prometheus Operator as a Custom Resource Definition. This object will be created in the prometheus namespace.
Everything then under the spec of the YAML file defines what the instance of Prometheus should look like.
replicas: 1
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 2Gi
Here we define the Resource limits (CPU & Memory) that each Prometheus Pod will be granted within Kubernetes. We also specify the number of instances that we require by setting replicas, in this example we have just the 1 instance.
baseImage: quay.io/prometheus/prometheus
version: v2.10.0
Setting the baseImage defines the actual Prometheus Docker image to be used. This will actually be defaulted to the Docker image that is released by the Prometheus project however we included it as an example. The version field sets the version of Prometheus you wish to use. You can see available versions on the GitHub project.
storage:
volumeClaimTemplate:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
This block defines the storage that will be used by Prometheus. By default the Operator will create Prometheus Pods that use local storage only by using an emptyDir. If you wish to retain the state of Prometheus and therefore the metrics that it stores when re-launching Prometheus, such as during a version upgrade, then you need to use persistent storage. The PersistentVolumeClaim (PVC) defines the specification of the storage to be used by Prometheus. In this example we are creating a persistent disk that is 10Gi for each instance of Prometheus that is created.
In your terminal if you execute kubectl get persistentvolumeclaim --namespace prometheus you will see the PVC that has been created and Bound to a Persistent Volume for the single instance of Prometheus that you have created:
$kubectl get persistentvolumeclaim --namespace prometheus
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-pvc-prometheus-prometheus-0 Bound pvc-c85b2a3b-9d7a-11e9-9e3c-42010a840079 10Gi RWO standard 21m
The ServiceAccount, ClusterRoleBinding and ClusterRoleBinding are required for providing the Prometheus Pod with the required permissions to access the Kubernetes API as part of its service discovery process.
Lastly lets look at some of the Prometheus specific configuration of the prometheus.yaml file:
logLevel: info
retention: 12h
Here we define that Prometheus should retain 12 hours of metrics and that it should log using the info log level.
The Prometheus Operator GitHub project provides a full set of API documentation that defines the fields that can be set on the CRDs that it provides. You can see the specification for the Prometheus Kind here.