Trivy-operator: CrashLoopBackOff, how to fix?

Photo by Wexor Tmg on Unsplash

Trivy-operator has sufficient trubleshooting documentation on the official web site. However, not all of the bugs and errors are described there. In this article, I would like to share one of the possible solutions to fix “CrashLoopBackOff” error for trivy-operator pod.

How to understand that this solution is applicable to your case?

• you have a huge cluster with a lot of active pods;
• you activated SBOM reports generation;
• you use webhooks to collect Trivy reports.

Explanation of my particular case

I had a huge cluster with multiple active pods. Since I have activated the SBOM report generation, the trivy-operator pod started to fail after several minutes of activity and got the status of “CrashLoopBackOff”. The only action that helped is to remove SBOM CRD:

kubectl delete crd sbomreports.aquasecurity.github.io
kubectl delete crd clustersbomreports.aquasecurity.github.io

Errors that I saw in logs

I activated the debug mode for trivy-operator to see the errors in details:

trivy-operator:
  operator:
    logDevMode: true

I started to check the errors in logs. Unfortunately, they were not really informative, like this for example:

 Trace[..]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.30.0/tools/cache/reflector.go:232 (..) 

 Trace[..]: ---"Objects listed" error:<nil> .. (..)

 Trace[..]: [..] [..] END

Then finally I got the error message that had more consistent message:

Failed to watch *v1alpha1.SbomReport: failed to list *v1alpha1.SbomReport: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 27; INTERNAL_ERROR; received from peer" logger="UnhandledError"

Step 1

This error means that the Kubernetes API Server that is responsible for communication with trivy-operator is not responding or dropping the connection.

Get janloo-security’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Reason: trivy-operator overloads the Kubernetes API Server.

Solution: reduce the load of Kubernetes API Server.

I adjusted the trivy-operator configuration:

trivy-operator:
  operator:
    scanNodeCollectorLimit: 1   # set to collect information of active pods only from one node, by default 1
    scanJobsConcurrentLimit: 2  # set to run only 2 concurrent scans, by default 10

Step 2

After I adjusted the configuration — Kubernetes API Server stopped dropping the connection. However, the “CrashLoopBackOff” didn’t disappear. This time the pod of trivy-operator was just killed with no error messages.

I started to track the workflow of the pod, and noticed that is was killed automatically after it had reached the resources limits.

Reason: trivy-operator was out of memory because of a lot of activities.

Solution: raise the resources of trivy-operator and reduce the activities that can consume the memory.

I adjusted the trivy-operator configuration again:

trivy-operator:
  operator:
    webhookBroadcastTimeout: 0s  # set to 0s not to wait for the response from webhook and economise memory

  resources:  # up to you to set the resources requests and limits
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      cpu: 2
      memory: 8Gi

Final configuration

After the activation of the following configuration, my trivy-operator got fixed:

trivy-operator:
  operator:
    scanNodeCollectorLimit: 1
    scanJobsConcurrentLimit: 2
    webhookBroadcastTimeout: 0s

  resources:
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      cpu: 2
      memory: 8Gi

Summary

The information, I’ve gathered in this article was really useful for me. I hope, it will be useful for someone else. If you have any recommendations, remarks or thoughts regarding the theme, do not hesitate to leave a comment.