The kube-state-metrics application extracts metrics from Kubernetes about the state of the different types of objects in Kubernetes such as, Pods, Deployments, StatefulSets, etc.
The project itself describes itself pretty well:
kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. (See examples in the Metrics section below.) It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods. kube-state-metrics is about generating metrics from Kubernetes API objects without modification. This ensures that features provided by kube-state-metrics have the same grade of stability as the Kubernetes API objects themselves. In turn, this means that kube-state-metrics in certain situations may not show the exact same values as kubectl, as kubectl applies certain heuristics to display comprehensible messages. kube-state-metrics exposes raw data unmodified from the Kubernetes API, this way users have all the data they require and perform heuristics as they see fit. The metrics are exported on the HTTP endpoint /metrics on the listening port (default 80). They are served as plaintext. They are designed to be consumed either by Prometheus itself or by a scraper that is compatible with scraping a Prometheus client endpoint. You can also open /metrics in a browser to see the raw metrics.
To use kube-state-metrics we need to deploy a single replica of kube-state-metrics as a Pod in the target Kubernetes cluster.
Create a file called kube-state-metrics.yaml and add the following:
--- apiVersion: v1 kind: ServiceAccount metadata: labels: app: kube-state-metrics name: kube-state-metrics namespace: prometheus --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kube-state-metrics rules: - apiGroups: [""] resources: - configmaps - secrets - nodes - pods - services - resourcequotas - replicationcontrollers - limitranges - persistentvolumeclaims - persistentvolumes - namespaces - endpoints verbs: ["list", "watch"] - apiGroups: ["extensions"] resources: - daemonsets - deployments - replicasets - ingresses verbs: ["list", "watch"] - apiGroups: ["apps"] resources: - daemonsets - deployments - replicasets - statefulsets verbs: ["list", "watch"] - apiGroups: ["batch"] resources: - cronjobs - jobs verbs: ["list", "watch"] - apiGroups: ["autoscaling"] resources: - horizontalpodautoscalers verbs: ["list", "watch"] - apiGroups: ["policy"] resources: - poddisruptionbudgets verbs: ["list", "watch"] - apiGroups: ["certificates.k8s.io"] resources: - certificatesigningrequests verbs: ["list", "watch"] - apiGroups: ["storage.k8s.io"] resources: - storageclasses verbs: ["list", "watch"] - apiGroups: ["autoscaling.k8s.io"] resources: - verticalpodautoscalers verbs: ["list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app: kube-state-metrics name: kube-state-metrics roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-state-metrics subjects: - kind: ServiceAccount name: kube-state-metrics namespace: prometheus --- apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: app: kube-state-metrics name: kube-state-metrics namespace: prometheus spec: replicas: 1 selector: matchLabels: app: kube-state-metrics strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0 type: RollingUpdate template: metadata: labels: app: kube-state-metrics spec: containers: - image: gcr.io/google_containers/kube-state-metrics:v1.6.0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: / port: 8080 scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 30 name: kube-state-metrics ports: - containerPort: 8080 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: / port: 8080 scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 resources: limits: cpu: 500m memory: 768Mi requests: cpu: 250m memory: 768Mi restartPolicy: Always serviceAccount: kube-state-metrics serviceAccountName: kube-state-metrics --- apiVersion: v1 kind: Service metadata: labels: app: kube-state-metrics name: kube-state-metrics namespace: prometheus spec: ports: - name: kube-state-metrics port: 80 protocol: TCP targetPort: 8080 selector: app: kube-state-metrics type: ClusterIP --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app: kube-state-metrics serviceMonitorSelector: prometheus name: kube-state-metrics namespace: prometheus spec: endpoints: - honorLabels: true interval: 30s path: /metrics targetPort: 8080 jobLabel: kube-state-metrics namespaceSelector: matchNames: - prometheus selector: matchLabels: app: kube-state-metrics
There is a lot going on in the YAML code above. But to summarise, we are creating a Cluster Role called kube-state-metrics that includes all the required RBAC permissions for the service to operate successfully. We then bind the Cluster Role to a Service Account that will be used by the Pod that is created. Speaking of that Pod, we create a Deployment object to actually deploy kube-state-metrics to the prometheus namespace, configuring it to use the Service Account we created. Lastly we create a ClusterIP Service and a ServiceMonitor within the prometheus namespace so that Prometheus can scrape the metrics that are exposed by kube-state-metrics.
Go ahead and install kube-state-metrics into your Kubernetes cluster by executing
kubectl apply -f kube-state-metrics.yaml.
You can then use
kubectl get pods --namespace prometheus to see the kube-state-metrics Pod being created by Kubernetes. After a brief moment you can then check the configured Targets in Prometheus and you will see that kube-state-metrics is now being successfully scraped.
The documenation for kube-state-metrics provides a wealth of useful information on the metrics that are exposed by the service. In a later section, we will look at using these metrics to build powerful dashboards for visualising the health of a Kubernetes cluster.
However before we move on lets take a look at how we can make use of kube-state-metrics.
Lets say we wanted to see all the StatefulSets deployed in the Kubernetes cluster we can use the
kube_statefulset_created metric. If we query Prometheus for this metrics, it will return a metric for each StatefulSet in the cluster and include Prometheus Labels that provide us with metadata about the StatefulSet.
Above we can see that a single metric was returned, which is as follows:
The metric shows that the StatefulSet named prometheus-prometheus in the namespace prometheus was created on 3rd July 2019 at 15:26 UTC. The time comes from the value 1562167613 which is a UNIX timestamp.