This project holds the configuration files for our internal Red Hat Observability Service based on Observatorium.

See our website for more information about RHOBS.

• Go 1.17+

This repository contains Jsonnet configuration that allows generating Kubernetes objects that compose RHOBS service and its observability.

The jsonnet files for RHOBS service can be found in services directory. In order to compose RHOBS Service we import many Jsonnet libraries from different open source repositories including kube-thanos for Thanos components, Observatorium for Observatorium, Minio, Memcached, Gubernator, Dex components, thanos-receive-controller for Thanos receive controller component, parca for Parca component, observatorium api for API component, observatorium up for up component, rules-objstore for rules-objstore component.

Currently, RHOBS components are rendered as OpenShift Templates that allows parameters. This is how we deploy to multiple clusters, sharing the same configuration core, but having different details like resources or names.

This is why there might be a gap between vanilla Observatorium and RHOBS. We have plans to resolve this gap in the future.

Running make manifests generates all required files into resources/services directory.

Similarly, in order to have observability (alerts, recording rules, dashboards) for our service we import mixins from various projects and compose all together in observability directory.

Running make prometheusrules grafana generates all required files into resources/observability directory.

Up-to-date list of jsonnet dependencies can be found in jsonnetfile.json. Fetching all deps is done through make vendor utility.

To update a dependency, normally the process would be:

make vendor # This installs dependencies like `jb` thanks to bingo project.
JB=`ls .bin/jb-* -t | head -1`

# Updates `kube-thanos` to master and sets the new hash in `jsonnetfile.lock.json`.
$JB update https://github.com/thanos-io/kube-thanos/jsonnet/kube-thanos@main

# Update all dependancies to master and sets the new hashes in `jsonnetfile.lock.json`.
$JB update

Our deployments our managed by our Red Hat AppSRE team.

Staging: Update the commit hash ref in https://gitlab.cee.redhat.com/service/app-interface/blob/master/data/services/observability/cicd/saas/saas-grafana.yaml

Production: Update the commit hash ref in https://gitlab.cee.redhat.com/service/app-interface/blob/master/data/services/observability/cicd/saas/saas-grafana.yaml

Use synchronize.sh to create a MR against app-interface to update dashboards.

Staging: update the commit hash ref in https://gitlab.cee.redhat.com/service/app-interface/blob/master/data/services/telemeter/cicd/saas.yaml

Production: update the commit hash ref in https://gitlab.cee.redhat.com/service/app-interface/blob/master/data/services/telemeter/cicd/saas.yaml

Jobs runs are posted in:

#sd-app-sre-info for grafana dashboards

and

#team-monitoring-info for everything else.

1. Enable port forwarding for a user - example
2. Add a pod name to the allowed list for port forwarding - example