From having joined multiple organizations with extremely out of date documentation, or simply none at all, yes, quite literally no technical documentation as it was all in the heads of a handful of people, it can not only be frustrating for new joiners to the team, but also creates a lot of major functional problems in how the team operates.

How can we expect someone to onboard and learn about the project’s they’re meant to be working on if the only reference for the internal toolings/practices are only verbal? How can we expect the team members with all this knowledge in their heads to take hours out of their day to write documentation when surely they have more important things to do?

These are just some of the questions I hope to help answer and provide tips on generally writing documentation that should be understandable to anyone within the organization, not just people on your team or engineering.

Don’t overcomplicate things unnecessarily.

I always like to think that good documentation should be simple and straight forward, it should explain the context behind the documentation (why is the tool, task, or documentation page being created) in a few sentences, then how to do whatever you are trying to achieve. If it’s a guide on how to set up an EKS cluster, for example, you don’t need to include a bunch of documentation about what K8 is and the history of EKS. These can be created as a separate document under the same section. Instead, the document should outline the scenario in which this document is relevant, and then a clear step by step instruction on how to achieve the EKS set up.

I often like to think, if I can show this document to my tech illiterate family members, be that grandparent or parent, and they can get the gist or a decent understand of the doc, then it’s comprehensive. Of course not all documentation can be simple, but trying to simplify as much as possible within the context of the document topic can help.

Don’t be afraid to link to external sources.

None of us know everything, and we don’t necessarily need to spend time re-writing something generally used in tech within our documentation when there are many, very good, external sources which already explain what we are trying to achieve.

For example, if I’m trying to write a document explaining the architecture of an EKS set up and I want to describe how to resolve an issue during set up that may occur with the kubelet, you can assume 90% of the people reading the doc know what the kubelet is, however, finding a good link that you feel describes very simply what a kubelet is and then referencing that in your doc can allow people to go deeper and gain a better understanding. It can also be helpful when you’re writing the docs as there could be more context around the subject of the document itself.

Define what pre-requisite knowledge is required.

It is very probable that the majority of the people reading your documentation are engineers, however, it’s also possible that non-engineers or people without the same context as you on the project will be reading your document. Adding a section at the start to define what knowledge is required to fully understand the document, or even, complete the required tasks in the document can be critical in helping it be understood by a wider audience.

For the busy engineers, carve out time in your schedule to catch up on documentation.

It’s easier said than done and I’m fully aware of this, we all say “here’s an hour where I will write docs” but then you get a ping or an email or something higher priority comes up and next thing you know there’s a draft or a half written document that get’s forgotten about. I’ve definitely been guilty of this.

Ultimately I think there’s a few things that lead to it, one of them is how highly does the organization value documentation? I’ve been in organizations where the issue of literally no documentation was raised to a manager and the response was “it’s not important, we will get to it later”, of course, later never arrived. Sometimes it’s the mental effort, documentation can be incredibly draining to write, especially at the end of a long project where the initial view of the project has evolved over time and there’s now a lot to document.

Having a regular hour or two during your week, for me a Friday afternoon usually works, in which you mark yourself as do not disturb on slack and just focus on updating any out of date documentation or creating new documentation can help the overall team and organization run a bit smoother.

An architecture diagram says a thousand words.

As the old saying goes “a picture says a thousand words”. Sometimes, if you’re describing the workings of a system and want to effectively show how everything works together, words can often be too confusing or convoluted. Diagrams can often serve to simplify this by creating a visual aid to piece all the technical text together. Many of us are visual learners and having a visual aid means that not only will your reader perhaps understand your overall document better, but it may also help them remember it better.

There are so many more tips, but to avoid writing a massive post I’ll just highlight some of the more common ones I use as well:

• If you’re writing a step by step guide, ask someone who is unfamiliar with the tasks in the guide to test it, if they get confused, lost or have many questions, just means the guide needs to be clearer.
• Keep it short, no one likes to read text that rambles on forever, try to keep the document short and to the point.
• Don’t be afraid to write documentation, you can always go back and edit it. Often if you find there is something that should be documented but isn’t, this is a perfect time for you to not only “add value” to the team, but also practice writing documentation. As with everything, the more you practice the better you become.
• It can sometimes help to define the intended audience at the start of the document. If you’re a DevOps Engineer, you will need separate documentation for developers on how to, for example, interact with the CI/CD tooling vs what your team will need to interact with the tooling.

Thanks for reading :)

Most Engineering interviews in my experience consist of 3–5 rounds, where the first is a recruiter screening, 2–3 rounds of coding/linux systems, system design, or general technical questions, and a final round focused more on company fit, behavioural and situational questions. This can be a lot to prepare for, especially if you haven’t touched things like LeetCode or HackerRank style questions or concepts like Big O, Data Structures or Algorithms in a long time.

• “Explain it to me like I’m 5”: My favourite prompt for having ChatGPT explain relatively complex topics in a very simple way that is easy to understand, mostly because it sometimes starts with “Okay kiddo”.

• If you’re not 100% sure about the topic, just keep asking questions: Unlike when you google a question, ChatGPT will give a direct answer without having to click on multiple links, also, unlike a human, if you spend 40 minutes asking ChatGPT to explain how to calculate Time or Space Complexity, for example, it won’t become frustrated with you if you simply cannot grasp it. So you can ask as many questions needed to understand a topic well.
• “Give me a coding problem and I will propose an optimised solution”: This prompt can lack a bit of creativity from the ChatGPT side, but I’ve found it’s a good starting point if you want to try something else for coding questions than HackerRank and LeetCode. Tip: You can spice up this prompt by getting more specific, for example, “Give me a challenging coding problem where the solution requires dynamic programming”, or “Give me some challenging code and I will calculate the Time and Space Complexity”.
• “Generate 10 interview questions for a [role] about [topic]”: One of the valuable things about ChatGPT is that it can come up with questions for interview topics that may be ones you’ve never heard of before, or about topics you may forget to review.

More general interview advice:

• Don’t lie on your resume and know it inside out: The worst thing you can do is lie on your resume and claim to have experience you don’t have. It’s much more valuable to know the details you put on your resume well and be comfortable to talk about it, than to lie and be caught in a lie as that can immediately kill the interview.
• Prepare examples of past work: It’s always useful to be able to relate a question back to something you’ve worked on in the past, and being prepared with examples can help you answer and get to what you want to convey efficiently.
• Prepare questions for the interviewers: There’s usually a section at the end of each interview when you can ask questions to the interviewer(s). There are many guides and Career blogs/TikToks that give advice on which type of questions you should ask. These can range from “how did I do?” to “What’s the favourite part of your job?”, it can be hard to know what you should actually ask. I’ve found that generally asking about the interviewer’s experience, more about the company and the team or role, and trying to learn more about the tech stack and how the day to day is in the role has been more valuable for knowing what you will get yourself into if you were to receive an offer. Ultimately, the best questions are the ones that make neither you or the interviewer uncomfortable, and can help end the interview on a positive note.
• Try mock interviewing with a Rubber Duck: The Rubber Duck method is known to be a useful tool for developers and engineers to explain code and help work through debugging out loud where you, well, talk to a Rubber Duck. I have a little character from Animal Crossing that I use, but the idea is that when you do coding or system interviews, especially virtually, you need to be able to coherently explain and walk through your thought process out loud, pretending the Rubber Duck is your interviewer can help you get used to this and ultimately make the interview itself more relaxing as you’ve been practicing.

I hope this helps the interviewing process and makes the preparation a bit easier. Interviewing can be very stressful, but at the end of the day, you just need to try your best and what will be will be.

Thanks for reading :)

How to use ChatGPT to help you prepare for interviews was originally published in Dev Genius on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the good things about having worked in a start-up is that there’s little legacy code, but it also means you need to configure and set everything up yourself, which means it’s perfect for learning how to research different toolings, design and implement a system with complete ownership.

For the needs at the time, the metrics monitoring stack that was deployed to EKS via ArgoCD (GitOps) was comprised of:

• OpenTelemetry (OTEL): The collector application allows you to not only collect metrics, but also logs and traces and send these to various locations. In this post, we will be focusing only on metrics. (Part 1)
• Thanos: The question of “Why Thanos and not Prometheus?” was naturally one that had to be asked and ultimately it boiled down to its ability to enable Prometheus operations with higher availability and unlimited storage via S3 (or other Object Store methods). Thanos is also built on top of the Prometheus Operator, meaning it is compatible with PrometheusRule CRDs for setting up alerting rules. (Part 1)
• Grafana: Well known tool for representing Metrics on a Dashboard. (Part 2)
• AlertManager: Receives the alerting from the prometheus rules, in this case, from Thanos Ruler, and performs actions, such as notifying Slack channels, based on these. (Part 2)

The overall idea of the design is to have a bunch of lightweight OTEL deployments which perform the scraping actions, then send the metrics to thanos-receive as a remote endpoint. Thanos will then do all the heavy lifting, it will upload these metrics to S3, allow for direct query, handle alerting rules, and compact the metrics that are older than a certain timeframe. The alerting rules will then forward alerts to AlertManager where it will send the neccesary alerts, and Grafana will take both AlertManager and Thanos as a data source and dashboards and alerting can be viewed elegantly.

These posts will not cover the AWS set up portion which includes VPC Peering or setup, Hosted Zone/DNS setup, exposing Grafana, Thanos, or AlertManager within the organization, ArgoCD or security setup and other related component. This will solely focus on the setup and deployment of each of the components listed above.

Thanos

One of the great things about Kubernetes and EKS, is the support for Helm Charts. With a variety of choices for Thanos, I opted for the Bitnami Helm Chart as, initially, it appeared to provide all the support needed for all the components, as other Helm Charts may have not had support for Thanos Receive or other minor things.

The values.yaml passed to the Helm Chart looked something like this:

objstoreConfig:
  type: S3
  config:
    bucket: thanos-metrics-bucket
    endpoint: s3.region.amazonaws.com
    sts_endpoint: sts.amazonaws.com
    aws_sdk_auth: true
receive:
  enabled: true
  resources:
    limits:
      memory: 25Gi
    requests:
      memory: 25Gi
      cpu: 1000m
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 6
    targetCPU: 75
    targetMemory: 75
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::aws_account_id:role/thanos-receive-sa
query:
  enabled: true
  replicas: 2
storegateway:
  enabled: true
  replicas: 2
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::aws_account_id:role/thanos-storegateway-sa

Some limitations or problems encountered and how they were resolved…

Thanos Ruler does not support Rule Selector in Bitnmai

The biggest limitation encountered when specifically using the Bitnami Helm Chart was the inability to define a ruleSelector for the thanos-ruler component.

This ruleSelector effectively allows you to define some labels or expressions that Thanos Ruler can look for in any PrometheusRule CRDs. The benefit of having this in a large distributed system with multiple teams and users, is that you can allow each team to create their own Alerting Rules by creating a CRD file and including the label required to be picked up by Thanos Ruler.

The Bitnami Chart, however, only supports having this defined as a ConfigMap which then must be uploaded. Having multiple people and teams making relatively frequent updates to a single ConfigMap sounds like a recipe for disaster as potential misconfiguration of one rule could then impact all others not being picked up as well.

To get around this limitation, Thanos Ruler was deployed by using the ThanosRuler CRD from the Prometheus Operator.

All that really needs to be defined for it to work similarly to the Helm deployment is:

apiVersion: monitoring.coreos.com/v1
kind: ThanosRuler
metadata:
  name: thanos-ruler
spec:
  replicas: 3
  ruleSelector:
    matchLabels:
      component: thanos-ruler
  queryEndpoints:
    - dnssrv+_http._tcp.thanos-query.thanos.svc.cluster.local
  alertmanagersConfig:
    name: alertmanager-config-secret
    key: alertmanager-config.yaml

This configuration will then pick up any PrometheusRule CRD’s with component: thanos-ruler in the labels.

Thanos Receive Memory Usage surging

As you probably noticed, thanos-receive is the only component with memory and cpu usage defined, as well as the only one in an AutoScaling group.

This is mainly due to many issues experienced during deployment and continuous running where, as traffic would increase, thanos-receive would begin having memory increase a lot. Often this would result in CrashLoopBackoff errors from OOMKilled, and so more memory was continuously allocated, EKS Nodes upgraded, and the number of thanos-receive replicas increased. This, in part, was caused by misconfiguration of the OpenTelemetry side (covered below), however, the issues persisted even with those issues resolved, though, much less frequent.

The best way I found at the time to resolve these issues was to set up an autoscaling group, where once the existing nodes reached a certain threshold, the number of pods would increase for thanos-receive and this would help manage the traffic and allow the pods and service to go into a crash loop all together while also ensuring that the pods had sufficient memory to avoid memory surging too quickly for the HPA to keep up.

This approach seemed to prove successful, however, it is quite fragile.

OpenTelemetry

As the OTEL deployments are meant to be very lightweight, the majority of the default values found in the OpenTelemetry Collector Helm Chart should suffice, however, we want to enable and disable certain things.

mode: daemonset
config:
  exporters:
    prometheusremotewrite:
      endpoint: http://thanos-receive.thanos.svc.cluster.local:9093
      external_labels:
        cluster: my-eks-cluster-name
  processors:
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 25
    batch:
      send_batch_size: 2048
      timeout: 30s
  receivers:
    prometheus:
      config:
        global:
          scrape_interval: 30s
          scrape_timeout: 15s
        scrape_configs:
          - job_name: 'pods'
            scheme: https
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            kubernetes_sd_configs:
            - role: pod
              selectors:
              - role: pod
                field: "spec.nodeName=$KUBE_NODE_NAME"
  service:
    pipelines:
      logs: null
      traces: null
      metrics:
        exporters:
          - prometheusremotewrite
        processors:
          - batch
          - memory_limiter
        receivers:
          - prometheus
clusterRole:
  create: true
  rules:
    - apiGroups:
        - ''
      resources:
        - 'pods'
        - 'nodes'
        - 'endpoints'
        - 'services'
        - 'namespaces'
      verbs:
        - 'get'
        - 'list'
        - 'watch'
presets:
  clusterMetrics:
    enabled: false
  kubernetesAttributes:
    enabled: true
  kubeletMetrics:
    enabled: true

By default, the OTEL Collector is deployed as a replicaset, this can cause certain duplication issues where each pod in the collector (if HA deployment) is querying each endpoint or node multiple times, instead of just the specific pods or node the collector is deployed on. By switching the deploymentMode to daemonset there will be one OTEL Collector per EKS node. However, by default, you will still find that each collector pod will query all endpoints for the pods in the cluster. This can cause memory usage to surge for the Thanos Receive deployment as it will be receiving a high volume of traffic from the collector.

To resolve this, you can define a selector in your Prometheus Scrape Job to only scrape pods where the nodeName is the same as the OTEL Collector Pod’s nodeName .

kubernetes_sd_configs:
- role: pod
  selectors:
  - role: pod
    field: "spec.nodeName=$K8S_NODE_NAME"

Additionally, if you don’t plan to use OTEL Collector for getting logs or traces, you might want to disable these pipelines, otherwise, these will be created when the pods start up, which could potentially cause misleading errors (making it harder to troubleshoot) or even increase the memory usage of the OTEL pods.

config:
  service:
    pipelines:
      logs: null
      traces: null

It’s also recommended by the Helm Chart, when OTEL Collector is deployed as a daemonset to set the following configuration for the presets:

presets:
  clusterMetrics:
    enabled: false
  kubernetesAttributes:
    enabled: true
  kubeletMetrics:
    enabled: true 

I hope Part 1 was useful and helpful incase you’ve encountered some of the setup issues described or simply want to know how difficult it can be to deploy Thanos or OpenTelemetry for Metrics.

Part 2 coming soon which will cover the Grafana and AlertManager components of the monitoring stack! In the meantime, let me know if you have any experience with these tools, did you encounter any issues, are there things you would have done different, etc :)

Thanks for reading!

Deploying a Modern Monitoring Stack (Part 1) was originally published in Dev Genius on Medium, where people are continuing the conversation by highlighting and responding to this story.

I had been working in the DevOps and Site Reliability space for roughly 5 years by the time that ChatGPT made it’s earth shattering debut to the world, and it truly revolutionized my workflow, but I’ll get into that more later.

Similarly to many of us, I had heard the rumours of ChatGPT being able to write a complete application from scratch, and thinking “it can’t really be that good” decided to test it out and see for myself.

Initially, I started asking it to produce small scripts in Python and Bash, just to dip my toes in the OpenAI/ChatGPT waters.

The results blew me away! 🤯 Not only did it produce code with perfect syntax but it also explained the code to me!

At the time, I had been working on deploying Airflow to an EKS Cluster using Helm via Terraform, this setup took me roughly 3–5 days to get all squared away and tested, so I asked ChatGPT to write configuration for this.

While it did produce good configuration, it wasn’t exactly what I was looking for or hoping for, instead, I asked it’s OpenAI sibling, the Platform Playground the same question.

The OpenAI Platform Playground tool was such a game changer! It produced the code as it appears to have internet access where ChatGPT does not, and therefore can provide more well rounded configuration. The configuration seen here that was output by the Playground tool was almost identical to the one I had configured a few days prior manually, and what took me a few days, it took this AI a few seconds. From here, I began incorporating it into my regular workflow.

I tried asking ChatGPT about specific infrastructure design and configuration options for the metrics set up I was working on at the time. ChatGPT was able to give an answer in a matter of seconds that perfectly answered my questions and advised about best practice where hours of reading Thanos docs and different resources online couldn’t give a definite way forward.

After a few weeks of using both of these tool to simply generate code and answer basic questions, I began running into issues with another project I was working on, so I turned to ChatGPT to see if it could provide me an answer:

While ChatGPT didn’t provide much new information about how to troubleshoot or resolve the error from what I had already tried or found online, it can always help remind you to check something that may have been missed or forgotten. I’m sure with different errors it may give more detailed responses and may even provide the specific solution to a given error. This opens the door for so many, especially more junior engineers, to hone their troubleshooting and triaging skills through ChatGPT. It may also change the entire flow for how we investigate various issues not only in CI/CD and Infrastructure, but even for debugging code.

These two tools combined, ChatGPT and OpenAI Playground, have truly allowed me to improve and optimize my workflow. Helping unblock me at various points over the last two months when I simply cannot find what I need on Google or just need a quick question answered.

Tips and Tricks to get the most out of using ChatGPT for DevOps

• Specify Versions: if you want ChatGPT to generate configuration based on a Helm Chart (for example) that may have multiple versions, I’ve found that it doesn’t always take the latest by default. It can be helpful to include the version of the configuration you want in the request to get more accurate results.
• Trust but verify: as the quote goes “trust but verify”, you can trust that the output that ChatGPT is giving you is good and is error free, but as with any code obtained online, you should always read through it yourself and ensure you understand it before adding it to your project. Never assume that code generated by ChatGPT, no matter how amazing or excellent, is production ready out of the box.
• If ChatGPT isn’t giving the responses you want, you can also try using OpenAI’s Playground. It’s important to add that there is a free trial for this product line, but one the trial ends it becomes paid.
• Give examples: it can be helpful to provide more information when asking ChatGPT something, this can also be done in the form of giving it an example. Instead of just asking regex to find an ip address in a string which may yield, you can instead say regex to find an ip address in a string, for example, in a string 1.1.1.1:0000 the regex should match 1.1.1.1 and it will perhaps generate a more exact response for your query.

Ultimately, at the current version of ChatGPT, I don’t believe DevOps Engineers or SRE’s are at risk for being replaced anytime soon. ChatGPT, however, offers plenty of ways and opportunities for engineers for increase their productivity and help unblock themselves in ways that may have taken days or multiple people’s assistance previously.

Thanks for reading :)

Using ChatGPT for DevOps was originally published in Dev Genius on Medium, where people are continuing the conversation by highlighting and responding to this story.