206 lines
9.2 KiB
ReStructuredText
206 lines
9.2 KiB
ReStructuredText
Monitoring / Metrics with Prometheus
|
|
========================
|
|
|
|
For deployment, we used combination for configuration of prometheus operator and application-monitoring operator.
|
|
|
|
Beware, most of the deployment notes could be mostly obsolete in really short time.
|
|
The POC was done on OpenShift 3.11, which limited us in using older version of prometheus operator,
|
|
as well as the no longer maintained application-monitoring operator.
|
|
|
|
In openshift 4.x that we plan to use in the near future, there is supported way integrated in the openshift deployment:
|
|
|
|
* https://docs.openshift.com/container-platform/4.7/monitoring/understanding-the-monitoring-stack.html
|
|
* https://docs.openshift.com/container-platform/4.7/monitoring/configuring-the-monitoring-stack.html#configuring-the-monitoring-stack
|
|
* https://docs.openshift.com/container-platform/4.7/monitoring/enabling-monitoring-for-user-defined-projects.html
|
|
|
|
The supported stack is more limited, especially w.r.t. adding user defined pod- and service-monitors, but even if we would want to
|
|
run additional prometheus instances, we should be able to skip the instalation of the necessary operators, as all of them should already be present.
|
|
|
|
|
|
Notes on operator deployment
|
|
-------------------
|
|
|
|
Operator pattern is often used with kubernetes and openshift for more complex deployments.
|
|
Instead of applying all of the configuration to deploy your services, you deploy a special,
|
|
smaller service called operator, that has necessary permissions to deploy and configure the complex service.
|
|
Once the operator is running, instead of configuring the service itself with service-specific config-maps,
|
|
you create operator specific kubernetes objects, so-alled CRDs.
|
|
|
|
The deployment of the operator in question was done by configuring the CRDs, roles and rolebinding and operator setup:
|
|
|
|
The definitions are as follows:
|
|
- https://github.com/prometheus-operator/prometheus-operator/tree/v0.38.3/example/prometheus-operator-crd
|
|
- https://github.com/prometheus-operator/prometheus-operator/tree/v0.38.3/example/rbac/prometheus-operator-crd
|
|
- https://github.com/prometheus-operator/prometheus-operator/tree/v0.38.3/example/rbac/prometheus-operator
|
|
|
|
Once the operator is correctly running, you just define a prometheus crd and it will create prometheus deployment for you.
|
|
|
|
The POC lives in https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/application-monitoring.yml
|
|
|
|
|
|
Notes on application monitoring operator deployment
|
|
---------------------------------------------------
|
|
|
|
The application-monitoring operator was created to solve the integration of Prometheus, Alertmanager and Grafana.
|
|
After you configure it, it configures the relevant operators responsible for these services.
|
|
|
|
The most interesting difference between configuring this shared operator,
|
|
compared to configuring these operators individually is that it configures some of the integrations,
|
|
and it integrates well with openshifts auth system through oauth proxy.
|
|
|
|
The biggest drawback is, that the application-monitoring operator is orphanned project,
|
|
but because it mostly configures other operators, it is relatively simple to just recreate
|
|
the configuration for both prometheus and alertmanager to be deployed,
|
|
and deploy the prometheus and alertmanager operators without the help or the application-monitoring operator.
|
|
|
|
Notes on persistence
|
|
--------------------
|
|
|
|
Prometheus by default expects to have a writable /prometheus folder,
|
|
that can serve as persistent storage.
|
|
|
|
For the persistent volume to work for this purpose, it has to
|
|
**needs to have POSIX-compliant filesystem**, and NFS we currently have configured is not.
|
|
This is discussed in the `operational aspects <https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects>`_
|
|
of Prometheus documentation
|
|
|
|
The easiest supported way to have a POSIX-compliant `filesystem is to setup local-storage <https://docs.openshift.com/container-platform/3.11/install_config/configuring_local.html>`_
|
|
in the cluster.
|
|
|
|
In 4.x versions of OpenShift `there is a local-storage-operator <https://docs.openshift.com/container-platform/4.7/storage/persistent_storage/persistent-storage-local.html>`_ for this purpose.
|
|
|
|
This is the simplest way to have working persistence, but it prevents us to have multiple instanes
|
|
across openshift nodes, as the pod is using the underlying filesystem on the node.
|
|
|
|
To ask the operator to create persisted prometheus, you specify in its configuration i.e.:
|
|
|
|
::
|
|
|
|
storage:
|
|
volumeClaimTemplate:
|
|
spec:
|
|
retention: 24h
|
|
storageClassName: local
|
|
resources:
|
|
requests:
|
|
storage: 10Gi
|
|
|
|
By default retention is set to 24 hours and can be over-ridden
|
|
|
|
|
|
Notes on long term storage
|
|
--------------------
|
|
|
|
Usually, prometheus itself is setup to store its metrics for shorter ammount of time,
|
|
and it is expected that for longterm storage and analysis, there is some other storage solution,
|
|
such as influxdb or timescaledb.
|
|
|
|
We are currently running a POC that sychronizes Prometheus with Timescaledb (running on Postgresql)
|
|
through a middleware service called `promscale <https://github.com/timescale/promscale>`_ .
|
|
|
|
Promscale just needs an access to a appropriate postgresql database:
|
|
and can be configured through PROMSCALE_DB_PASSWORD, PROMSCALE_DB_HOST.
|
|
|
|
By default it will ensure the database has timescaledb installed and configures its database
|
|
automatically.
|
|
|
|
We setup prometheus with directive to use promscale service as a backend:
|
|
https://github.com/timescale/promscale
|
|
|
|
::
|
|
|
|
remote_write:
|
|
- url: "http://promscale:9201/write"
|
|
remote_read:
|
|
- url: "http://promscale:9201/read"
|
|
|
|
Notes on auxialiary services
|
|
----------------------------
|
|
|
|
As prometheus is primarily targeted to collect metrics from
|
|
services that have beein instrumented to expose them, if your service is not instrumented,
|
|
or it is not a service, i.e. a batch-job, you need an adapter to help you with the metrics collection.
|
|
|
|
There are two services that help with this.
|
|
|
|
* `blackbox exporter <https://github.com/prometheus/blackbox_exporter>`_ to monitor services that have not been instrumented based on querying public a.p.i.
|
|
* `push gateway <https://prometheus.io/docs/practices/pushing/#should-i-be-using-the-pushgateway>`_ that helps collect information from batch-jobs
|
|
|
|
Maintaining the push-gateway can be relegated to the application developer,
|
|
as it is lightweight, and by colloecting metrics from the namespace it is running in,
|
|
the data will be correctly labeled.
|
|
|
|
With blackbox exporter, it can be beneficial to have it running as prometheus side-car,
|
|
in simmilar fashion, as we configure oauth-proxy, adding this to the containers section
|
|
of the prometheus definition:
|
|
|
|
::
|
|
|
|
- name: blackbox-exporter
|
|
volumeMounts:
|
|
- name: configmap-blackbox
|
|
mountPath: /etc/blackbox-config
|
|
- mountPath: /etc/tls/private
|
|
name: secret-prometheus-k8s-tls
|
|
image: quay.io/prometheus/blackbox-exporter:4.4
|
|
args:
|
|
- '--config.file=/etc/blackbox-config/blackbox.yml'
|
|
ports:
|
|
- containerPort: 9115
|
|
name: blackbox
|
|
|
|
We can then instruct what is to be monitored through the configmap-blackbox, you can find `relevant examples <https://github.com/prometheus/blackbox_exporter/blob/master/example.yml>` in the project repo.
|
|
Beause blackox exporter is in the same pod, we need to use the additional-scrape-config to add it in.
|
|
|
|
Notes on alerting
|
|
-----------------
|
|
|
|
Prometheus as is, can have rules configured that trigger alerts, once
|
|
a specific query evaluates to true. The definition of the rule is explained in the companion docs
|
|
for prometheus for developers and can be created in the namespace of the running application.
|
|
|
|
Here, we need to focus what happens with alert after prometheus realizes it should fire it,
|
|
based on a rule.
|
|
|
|
In prometheus crd definition, there is a section about the alert-manager that is supposed to
|
|
manage the forwarding of these alerts.
|
|
|
|
::
|
|
|
|
alerting:
|
|
alertmanagers:
|
|
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
name: alertmanager-service
|
|
namespace: application-monitoring
|
|
port: web
|
|
scheme: https
|
|
tlsConfig:
|
|
caFile: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
|
|
serverName: alertmanager-service.application-monitoring.svc
|
|
|
|
We already have alertmanager running and configured by the alertmanager-operator.
|
|
Alertmanager itself is really simplistic with a simple ui and api, that allows for silencing an
|
|
alert for a given ammount of time.
|
|
|
|
It is expected that the actual user-interaction is happening elsewhere,
|
|
either through services like OpsGenie, or through i.e. `integration with zabbix <https://devopy.io/setting-up-zabbix-alertmanager-integration/>`_
|
|
|
|
More of a build-it yourself solution is to use i.e. https://karma-dashboard.io/,
|
|
but we haven't tried any of these as the part of our POC.
|
|
|
|
To be able to be notified of the alert, you need to have the `correct reciever configuration <https://prometheus.io/docs/alerting/latest/configuration/#email_config>`_ in the alertmanagers secret:
|
|
|
|
::
|
|
|
|
global:
|
|
resolve_timeout: 5m
|
|
route:
|
|
group_by: ['job']
|
|
group_wait: 10s
|
|
group_interval: 10s
|
|
repeat_interval: 30m
|
|
receiver: 'email'
|
|
receivers:
|
|
- name: 'email'
|
|
email_configs:
|
|
- to: 'asaleh@redhat.com'
|