<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tom Klinovsky on Medium]]></title>
        <description><![CDATA[Stories by Tom Klinovsky on Medium]]></description>
        <link>https://medium.com/@lutraman?source=rss-25ff3ae28cda------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*0PquMRjefC9jt_vg.jpg</url>
            <title>Stories by Tom Klinovsky on Medium</title>
            <link>https://medium.com/@lutraman?source=rss-25ff3ae28cda------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:30:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lutraman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Finding a Race Condition in Kuberenetes — Like a Detective]]></title>
            <link>https://medium.com/israeli-tech-radar/finding-a-race-condition-in-kuberenetes-like-a-detective-3a538f3b7038?source=rss-25ff3ae28cda------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a538f3b7038</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubernetes-operator]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Tom Klinovsky]]></dc:creator>
            <pubDate>Thu, 19 Feb 2026 06:40:11 GMT</pubDate>
            <atom:updated>2026-02-19T06:40:11.528Z</atom:updated>
            <content:encoded><![CDATA[<h3>Finding a Race Condition in Kuberenetes — Like a Detective</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*vOfz9TZCTxp14Txa6bCiwg.png" /></figure><p>During a project I working on I encountered an unusual problem: Orphaned pods would appear in the cluster throughout the day. These pods were usually managed by jobs and had a volume attached to them, so having them around meant that no replacement could be deployed until they are manually removed. It was a big problem that affected the users greatly.</p><p>My name is Tom, I’m a senior SRE at Tikal Knowledge with over a decade in the field. One of the skills that I developed over my career is the ability to investigate complicated software issues methodically and meticulously. I’ve developed many methods throughout the years and recently came upon a problem so intricate that it gave me the opportunity to share almost all my methods in one coherent — detective style — story.</p><h3>About the setup</h3><p>Now, our setup is somewhat unusual due to the requirements of the product. It runs a service that users connect their phones to and sync messages.</p><ul><li>Every user that logs in will have a pod spin up for them and pair with their phone.</li><li>The pod also has a volume attached to it which it uses to save session data.</li><li>The pods are managed by a job.</li></ul><h3>The Problem</h3><p>On some occasions, users that tried to pair their phones got an infinite loading screen. When they did, I found there are two pods for that user, one of them pending, and the other succeeded.</p><h3>The investigation begins — collecting the first evidence</h3><p>When working a problem, the first thing I always do is to pull out my notebook — I use Katavti, an app I wrote myself — and write down everything I know, and what I suspect. This also comes in handy when I need to hold long strings of data like a uid , and when I need to check multiple theories and want to make sure I don’t forget what I had in mind.</p><p>I start our investigation with the two pods. <strong>The job resource only has a concurrency of one, so that’s already odd.</strong> Inspecting the pods reveals the following:</p><pre>kubectl -n remote-sync-workers get pods<br>NAME                               READY   STATUS    RESTARTS   AGE<br>remote-sync-job-b1g70b-r2mcm    0/1     Pending   0          1d15h<br>remote-sync-job-dfec04-dsewv    0/1     Succeeded 0          2d14h</pre><pre>kubectl -n remote-sync-workers describe pod remote-sync-job-b1g70b-r2mcm<br>...<br>Events:<br>  Type     Reason            Age    From               Message<br>  ----     ------            ----   ----               -------<br>  Warning  FailedAttachVolume &lt;age&gt; attachdetach-controller  Multi-Attach error: volume &quot;xxx&quot; ...<br>  Warning  FailedMount        &lt;age&gt; kubelet     MountVolume.SetUp failed for volume ...</pre><p>The pod in the pending state is waiting for its volume claim (PVC) to become available, which is being held by the pod in the succeeded state. Another thing I discovered it that the pod in the succeeded state doesn’t have an owner reference to a job. <strong>We never create pods directly so that’s another thing that’s odd.</strong></p><p>To track the issue more closely, I used the metrics from kube-state-metrics to follow the number of orphaned pods in the cluster:</p><pre>count(kube_pod_owner{owner_kind!~&quot;.+&quot;, namespace=&quot;remote-sync&quot;})</pre><figure><img alt="Orphaned pods graph" src="https://cdn-images-1.medium.com/max/936/1*RMUkGLnPvo1oYhuEkmyInw.png" /><figcaption>Orphaned pods in Grafana</figcaption></figure><p>The kubernetes cluster runs on AWS, so I turn on audit logs in CloudWatch and waited. Using the aws cli tool, this can be done like so:</p><pre>aws eks update-cluster-config \<br>    --region us-east-2 \<br>    --name use2-prod-1 \<br>    --logging &#39;{&quot;clusterLogging&quot;:[{&quot;types&quot;:[&quot;audit&quot;],&quot;enabled&quot;:true}]}&#39;</pre><p>Not much time has passed and it happened again, this time I have some logs to look at. I query the logs for the name of the pod and see the following:</p><pre>{ $.objectRef.name = &quot;remote-sync-job-b1g70b-r2mcm&quot; }</pre><p>The logs show the pod being created by the job controller, as expected, but it also shows a patch request to delete its owner reference:</p><p><strong>Gotcha!</strong></p><pre>{<br>  &quot;kind&quot;: &quot;Event&quot;,<br>  &quot;apiVersion&quot;: &quot;audit.k8s.io/v1&quot;,<br>  &quot;requestURI&quot;: &quot;/api/v1/namespaces/remote-sync-workers/pods/remote-sync-job-b1g70b-r2mcm&quot;,<br>  &quot;verb&quot;: &quot;patch&quot;,<br>  &quot;user&quot;: {<br>    &quot;username&quot;: &quot;system:serviceaccount:kube-system:generic-garbage-collector&quot;,<br>    &quot;uid&quot;: &quot;xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&quot;<br>  },<br>  &quot;objectRef&quot;: { ... &quot;name&quot;: &quot;remote-sync-job-b1g70b-r2mcm&quot;, ... },<br>  &quot;responseStatus&quot;: { &quot;code&quot;: 200 },<br>  &quot;requestObject&quot;: {<br>    &quot;metadata&quot;: {<br>      ...<br>      &quot;ownerReferences&quot;: [ <br>        {<br>          &quot;$patch&quot;: &quot;delete&quot;,<br>          &quot;uid&quot;: &quot;xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&quot;<br>        }<br>      ]<br>    }<br>  }<br>}</pre><p>Who did it? The service account in the logs says its the kubernetes garbage collector. Well, the mystery isn’t solved yet, why does the garbage collector do that? Where’s the motive?</p><p>I widen the query of our search and limit it to a smaller time-frame of 15 seconds before and after the garbage collector removed the owner reference from the pod, to <strong>any resource related to the job</strong>.</p><pre>{ $.responseObject.metadata.labels.job-name = &quot;remote-sync-job-b1g70b&quot; }</pre><p>I find a few more interesting logs, that show:</p><ul><li>At 19:45:29.797Z: The job is being patched on spec.suspend</li><li>At 19:45:29.984Z: The job being deleted</li></ul><p>In both cases the service account is remote-sync-server-sa— that’s our app.</p><p>I consulted with the developers on the job, who took a look and explained that this happens when the user un-pairs their device, and looking at more logs reveals that’s a common behavior, jobs get deleted all the time, but only sometimes the pod remains running without the job.</p><p>Something else caught my eye in the log showing deletion. <strong>The response object showed a finalizer was added: </strong><strong>orphan</strong>. That looked a bit odd and also implies of our problem, some of the pods remain without the job that created them, orphans. However, looking at more logs showed me that this finalizer gets added to all the jobs that are being deleted, not only the problematic ones. It goes in the notes, but seems it’s a dead end.</p><p>Looking more closely at the logs revealed something very interesting: While most jobs get deleted by the job controller, and the garbage collector does nothing, <strong>in one sequence of logs both the deletion log by the job controller and the patch log by the garbage collector show up at the same time. A-hah! A race condition!</strong></p><h3>But why?</h3><p>Another thing that caught my eye is a deletionGracePeriod: 0. Maybe that what causes the garbage collector to detach the pods? I tried setting the TTL to 5, which sets the deletionGracePeriod to 5 when the job is deleted, and let it run.</p><p>A few days later and I see more orphan pods, our case is re-opened, <strong>I got the wrong guy</strong>.</p><h3>Going back to the basics</h3><p>I go and read the <a href="https://kubernetes.io/docs/tasks/administer-cluster/use-cascading-deletion/">documentation on the kubernetes propagation policy</a>. According to the docs, the orphan finalizer is only added when the propagation policy in the deletion request is orphan.</p><p>Looking at the logs again, I didn’t see any propagation policy, and the default should be background when there is none. Another thing that didn’t work out was that the documentation stated that<strong> regardless of the propagation policy used, it’s the garbage collector that suppose to delete the child resources (the pod)</strong>, but in logs I saw, it’s the job controller:</p><pre>{<br>  &quot;kind&quot;: &quot;Event&quot;,<br>  &quot;apiVersion&quot;: &quot;audit.k8s.io/v1&quot;,<br>  &quot;level&quot;: &quot;RequestResponse&quot;,<br>  &quot;stage&quot;: &quot;ResponseComplete&quot;,<br>  &quot;requestURI&quot;: &quot;/api/v1/namespaces/remote-sync-workers/pods/remote-sync-job-d1ff3nt-v7z74&quot;,<br>  &quot;verb&quot;: &quot;delete&quot;,<br>  &quot;user&quot;: {<br>    &quot;username&quot;: &quot;system:serviceaccount:kube-system:job-controller&quot;, &lt;-- HERE!<br>    &quot;uid&quot;: &quot;xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&quot;<br>  },<br>  &quot;userAgent&quot;: &quot;kube-controller-manager/v1.31.7 (linux/amd64) kubernetes/ca6b377/system:serviceaccount:kube-system:job-controller&quot;,<br>  &quot;objectRef&quot;: {<br>    &quot;resource&quot;: &quot;pods&quot;,<br>    &quot;namespace&quot;: &quot;remote-sync-workers&quot;,<br>    &quot;name&quot;: &quot;remote-sync-job-d1ff3nt-v7z74&quot;,<br>    &quot;apiVersion&quot;: &quot;v1&quot;<br>  }<br>}</pre><p>Time to run some experiments…</p><h3>Running experiments</h3><p>I created a job with kubectl and deleted it. Without stating the propagation policy, I noticed the request in the logs showed a background propagation policy was sent. It appears that kubectl sets it by default. The deletion process also went as expected, the pod was deleted, and the pod deletion was done by the garbage collector.</p><p>Next I tried the same, but with the javascript client used by the remote sync server. This time the request showed up in the logs without a propagation policy, and the orphan finalizer was added. Then, the garbage collector came and removed the owner reference, just like in our original problem.</p><p>It seems that, <strong>unlike what the documentation says, the default propagation policy is actually </strong><strong>orphan.</strong> The default might be referring to what kubectl is doing.</p><h3>The Root Cause</h3><p>One thing remained odd though. In my experiments, the pod remained orphaned every time, but in our live case, it only happened very rarely.</p><p>Turns out, in our app, <strong>we don’t delete the pod when we delete the job, we delete the pod when we suspend it.</strong> And because the server does it just before it sends the request to delete the job, it gets a head start. In most cases it wins, and the pod gets deleted before the garbage collector gets a chance to orphan it — but, if the garbage collector gets to it first (following the delete request with the orphan propagation policy), then the suspend request doesn’t find any pod to delete, and nothing deletes the pod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CQxWL2ePi-y_gtYca_hMEw.png" /><figcaption>Sequence of events for both race condition outcome</figcaption></figure><p>I eventually fixed it by explicitly setting the propagation policy to Foreground during job deletion and removing the suspend call . Now, the fix is deployed, and we don’t see more orphan pods in our metrics.</p><p>Our story reached a happy ending, we found the root cause and fixed the issue. Lets discuss the methods used to crack the case.</p><h3>Methods used</h3><ul><li><strong>Write everything down</strong> — Don’t miss a theory you already thought of because you spent some time investigating another one and forgot about the first one.</li><li><strong>Inspecting the currently configured state </strong>— in our case, that was inspecting the kubernetes objects, but it can also be configuration files, command line arguments, environment variables, etc.</li><li><strong>Looking at metrics, logs and traces</strong> — our go-to places of initial information should always be the logs and metrics, the order depends on the type of problem. If we’re lucky, that’s where we find the answer.</li><li><strong>Comparing to a process that works</strong> —When investigating a system that we did not write ourselves, a quick way to understand how it works is to find an environment where it works as expected.</li><li><strong>Increasing the verbosity of the logs (possibly metrics too)</strong> — logs in production are limited in their verbosity due to resource limitations, and therefore may not provide sufficient details. Temporarily increase the verbosity will help you find the problem faster.</li><li><strong>Reading the docs</strong> —When things get complicated, understanding how the system is designed to behave could go a long way.</li><li><strong>Looking in the code</strong> — In cases where the documentation is lacking, out of date or just flat out wrong, digging deep into the code might become necessary.</li><li><strong>Consulting with the developers </strong>—In some cases, contacting the devs might save days worth of debugging. Even if you would have reached it yourself eventually, in production, speed matters.</li><li><strong>Running experiments</strong> — In very complex issues, you will inevitably get to a point where you have a theory or two about what’s the source of the issue, but can’t go forward without knowing for sure. That’s the point where bringing up a test environment locally, changing some variables to test these theories becomes crucial.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a538f3b7038" width="1" height="1" alt=""><hr><p><a href="https://medium.com/israeli-tech-radar/finding-a-race-condition-in-kuberenetes-like-a-detective-3a538f3b7038">Finding a Race Condition in Kuberenetes — Like a Detective</a> was originally published in <a href="https://medium.com/israeli-tech-radar">Israeli Tech Radar</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What happens when you auto-instrument Python with OpenTelemetry in Kubernetes]]></title>
            <link>https://medium.com/israeli-tech-radar/what-happens-when-you-auto-instrument-python-with-opentelemetry-in-kubernetes-a6ed4ebf514d?source=rss-25ff3ae28cda------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6ed4ebf514d</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Tom Klinovsky]]></dc:creator>
            <pubDate>Thu, 10 Apr 2025 14:39:37 GMT</pubDate>
            <atom:updated>2025-04-10T14:39:37.767Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-mtuik552MOX303MhFGefA.png" /></figure><p>OpenTelemetry is a set of specifications and tools that standardize the transmission of metrics, logs, and traces across all types of applications. One of these tools is the almost magical “zero code” auto-instrumentation.</p><p>By setting up auto-instrumentation — without changing a single line of code — applications can begin sending metrics and traces containing data that couldn’t have been obtained without the application explicitly writing it in code (e.g. parsed endpoints). That seems impossible — so how does it happen?</p><p>My name is Tom Klinovsky, I’m a senior SRE for <a href="https://www.tikalk.com/">Tikal Knowledge</a>. I’ve been an SRE for over a decade, having worked for Google, Microsoft, and Wix. Over the past two years my focus has been on observability, with OpenTelemetry naturally playing a key role in that effort.</p><p>In this post, I’ll dive into the technical details of the steps that take place behind the scenes when using auto-instrumentation in several of its forms.</p><h3>The Setup</h3><p>We start with a Kubernets cluster that has a few pods running a Python app that uses <em>FastAPI</em> and <em>requests</em>. The OpenTelemetry Kubernetes Operator is also installed in the cluster, where a custom resource definition of the type Instrumentation is defined:</p><pre>apiVersion: opentelemetry.io/v1alpha1<br>kind: Instrumentation<br>metadata:<br>  name: my-instrumentation<br>spec:<br>  exporter:<br>    endpoint: http://otel-collector:4317<br>  propagators:<br>    - tracecontext<br>    - baggage<br>    - b3<br>  sampler:<br>    type: parentbased_traceidratio<br>    argument: &quot;0.25&quot;</pre><h3>What the OpenTelemetry Operator is doing</h3><p>With this resource defined, any pod starting up with an annotation instrumentation.opentelemetry.io/inject-python: &#39;true&#39; are caught by the OpenTelemetry operator which will start acting on the pod.</p><p>Once a python pod is created, the operator will attach an open-telemetry instrumentation container as an <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/">init container</a>, creating an emptyDir volume to place the instrumentation libraries. The init container copies the libraries to the emptyDir volume, so when the original container starts the libraries are reachable. It also modifies the PYTHON_PATH environment variable to include the emptyDir volume with the OpenTelemetry libraries.</p><h3>What the OpenTelemetry Python instrumentors are doing</h3><p>Depending on your level of experience with Python, you may be familiar with the sitecustomize file. It’s part of the <a href="https://docs.python.org/3/library/site.html">site module</a> and whenever it’s found in one of the paths listed in the PYTHONPATH environment variable it gets executed <em>before</em> the main module. And sure enough, if we look at the source code of the OpenTelemetry Python instrumentation, there’s a <a href="https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/opentelemetry-instrumentation/src/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py">site customize.py</a> file:</p><pre>➜  opentelemetry-python-contrib git:(main)  tree opentelemetry-instrumentation/src<br>opentelemetry-instrumentation/src<br>└── opentelemetry<br>    └── instrumentation<br>        ├── auto_instrumentation<br>        │   ├── __init__.py<br>        │   ├── _load.py<br>        │   └── sitecustomize.py &lt;--<br>        ├── bootstrap_gen.py<br>        ├── bootstrap.py<br>        ├── dependencies.py<br>        ├── distro.py<br>        ├── environment_variables.py<br>        ├── instrumentor.py<br>        ├── propagators.py<br>        ├── py.typed<br>        ├── _semconv.py<br>        ├── sqlcommenter_utils.py<br>        ├── utils.py<br>        └── version.py</pre><p>From there, it will be looking at the OTEL_PYTHON_DISTRO and OTEL_PYTHON_CONFIGURATOR environment variables to load the specific instrumentation libraries specified — these can be used for custom built auto-instrumentations. If non are specified, OpenTelemetry will load the first one it finds.</p><p>When the configurator is loaded, it loads exporters for metrics, traces and logs by the <a href="https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection">names defined in the environment variables</a> OTEL_METRICS_EXPORTER, OTEL_TRACES_EXPORTER and OTEL_LOGS_EXPORTER respectively. As each exporter gets loaded, they each load their config from environment variables such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. The configurator is also where<a href="https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/"> many other important configurations</a> are loaded.</p><p>Once it did that, it will call load_instrumentors — It’s job is to load specific instrumentation libraries for commonly used libraries and frameworks. First it will load all pre-instrumentation libraries — if there are any. Then, it will start loading the instrumentation libraries — there are <a href="https://github.com/open-telemetry/opentelemetry-python-contrib/tree/v0.52b1/instrumentation">a lot of them</a> — and will take care to skip over anything specified in OTEL_PYTHON_DISABLED_INSTRUMENTATIONS.</p><h3>The FastAPI Instrumentor</h3><p>In our case, our app is running <em>FastAPI</em>, so next we’ll be looking at what the <a href="https://github.com/open-telemetry/opentelemetry-python-contrib/blob/v0.52b1/instrumentation/opentelemetry-instrumentation-fastapi/src/opentelemetry/instrumentation/fastapi/__init__.py#L215"><em>FastAPI</em> instrumentation</a> is doing.</p><p>When the load_instrumentors function loads an instrumentor, it calls the _instrument function, which the <em>FastAPI</em> <a href="https://github.com/open-telemetry/opentelemetry-python-contrib/blob/v0.52b1/instrumentation/opentelemetry-instrumentation-fastapi/src/opentelemetry/instrumentation/fastapi/__init__.py#L319">instrumentation library</a> implements. In it, the important bits are these 2 lines of code:</p><pre>self._original_fastapi = fastapi.FastAPI<br>#...<br>fastapi.FastAPI = _InstrumentedFastAPI</pre><p>First it saves the original <em>FastAPI</em> library to a variable — that will be used for uninstrumentation if necessary — and then it places its own <em>FastAPI</em> library in its place. Remember, none of the app code ran yet, so any <em>FastAPI</em> app is yet to be initiated.</p><h3>The Requests Instrumentor</h3><p>Another library used by the Python app is <a href="https://pypi.org/project/requests/"><em>requests</em></a>. It’s an HTTP client which the app uses to communicate with other apps in the cluster. OpenTelemetry instruments it as well.</p><p>Much like how the <em>FastAPI</em> instrumentation replaced the original <em>FastAPI</em> library with its instrumented version, the <em>requests</em> <a href="https://github.com/open-telemetry/opentelemetry-python-contrib/blob/v0.52b1/instrumentation/opentelemetry-instrumentation-requests/src/opentelemetry/instrumentation/requests/__init__.py#L195">instrumentation</a> replaces the Session.send function of the <em>requests</em> library with a wrapped version of the function, called instrumented_send:</p><pre>wrapped_send = Session.send<br><br>@functools.wraps(wrapped_send)<br>def instrumented_send(<br>    self: Session, request: PreparedRequest, **kwargs: Any<br>):<br>    #...<br><br>#...<br>instrumented_send.opentelemetry_instrumentation_requests_applied = True<br>Session.send = instrumented_send</pre><h3>What happens when the app starts up</h3><p>At this stage, the OpenTelemetry instrumentation libraries have finished doing their job, and the app is ready to start. When it does, it’s initializing a <em>FastAPI</em> app — only that now it calls the __init__ function the OpenTelemetry instrumentation provided it.</p><p>The first thing it does is calling the super().__init__ so <em>FastAPI</em> gets to do its thing before anything else. Then a tracer and a meter are initialized and used to initialize the OpenTelemetryMiddleware. Which in turn, is added as a middleware to the <em>FastAPI</em> app. That means that every request the app gets will go through the OpenTelmetryMiddleware before anything else. Lets see what happens to such a request.</p><h3>What happens when the app receives a request</h3><p>The OpenTelemetryMiddleware used for <em>FastAPI</em> is the <a href="https://github.com/open-telemetry/opentelemetry-python-contrib/blob/v0.52b1/instrumentation/opentelemetry-instrumentation-asgi/src/opentelemetry/instrumentation/asgi/__init__.py#L511">asgi middleware</a>. When a request comes along, its __call__ function will be called. The first thing it does is to start a timer. This timer will be stopped and recorded to a histogram metric just before reply is sent.</p><p>Next, it’s starts a <a href="https://opentelemetry.io/docs/concepts/signals/traces/#spans"><em>span</em></a>, passing the <em>FastAPI</em> request’s scope as the context carrier — which is important because OpenTelemetry needs to check if we already got a trace_id in the headers. It also calls propogator.inject, which sets the trace_id in a <a href="https://docs.python.org/3/library/contextvars.html">Context Variable</a>. In addition, it collects attributes from the scope to add to the span and metrics.</p><p>When the app needs to send a subsequent request to another app in the cluster, it uses <em>requests</em> to do so. As discussed earlier, it also has been instrumented and the instrumented_send is called whenever the app uses its Session.requests. It records the duration, the HTTP method and status code into metric counters.</p><p>The <em>requests</em> instrumentation also starts a new <em>span</em> and injects the trace_id to the headers of any HTTP request it sends out. When it does so, it needs to use the same trace_id that was either set or received by the <em>FastAPI</em> instrumentation. It does so by <a href="https://github.com/open-telemetry/opentelemetry-python/blob/v1.31.1/opentelemetry-api/src/opentelemetry/trace/propagation/__init__.py#L38">obtaining the </a><a href="https://github.com/open-telemetry/opentelemetry-python/blob/v1.31.1/opentelemetry-api/src/opentelemetry/trace/propagation/__init__.py#L38">current_span</a> from the Context Variable set earlier by the the <em>FastAPI</em> instrumentation.</p><p>Eventually the reply is sent out, and all the metrics and traces are recorded to the tracer and metric providers. Depending on the way the configurators defined the exporters (e.g. with OTEL_METRIC_EXPORT_INTERVAL) the metrics traces and logs will eventually send what they recorded to their configured endpoints.</p><h3>Conclusion</h3><p>“What happens when” questions are common in interviews. They are used when the interviewer is interested to know how well the candidate knows the technology they are intended to maintain. From the classical “what happens when you type google.com in the address bar?”, to more region specific questions like “what happens when you apply a kubernetes deployment”. As OpenTelemetry takes its place as the standard for observability, interviewers are certain to look for subject experts.</p><p>Understanding how something works is not just useful for maintaining the thing itself, it also gives you tools to apply in other fields. If you read all the way to here, you now not only know how OpenTelemetry works, but also multiple techniques to inject code into Python.</p><p>If you’re interested in knowing more about OpenTelemetry, Kubernetes, Linux, or any other SRE or DevOps subject, <a href="https://www.linkedin.com/in/tomklino/">follow me on LinkedIn</a>, and let me know what you want me to cover next.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6ed4ebf514d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/israeli-tech-radar/what-happens-when-you-auto-instrument-python-with-opentelemetry-in-kubernetes-a6ed4ebf514d">What happens when you auto-instrument Python with OpenTelemetry in Kubernetes</a> was originally published in <a href="https://medium.com/israeli-tech-radar">Israeli Tech Radar</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Win an SRE’s Heart — A developer’s guide to sturdy production code]]></title>
            <link>https://medium.com/israeli-tech-radar/how-to-win-an-sres-heart-a-developer-s-guide-to-sturdy-production-code-0861798ac8f4?source=rss-25ff3ae28cda------2</link>
            <guid isPermaLink="false">https://medium.com/p/0861798ac8f4</guid>
            <category><![CDATA[trends-and-insights]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[Tom Klinovsky]]></dc:creator>
            <pubDate>Sun, 02 Mar 2025 14:00:25 GMT</pubDate>
            <atom:updated>2025-03-02T14:00:25.796Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Win an SRE’s Heart — A developer’s guide to sturdy production code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SXaIlzXlEIeK-6Z5ryEfFg.jpeg" /></figure><p>My name is Tom. Over the course of my career I’ve mostly been an SRE - or some variation of it. Most recently for Google, before that at Microsoft and Wix, and currently at <a href="https://www.tikalk.com/">Tikal Knowledge</a>. When I’m not working however, I often write fullstack pet-projects.</p><p>Being an SRE, I always do my best to write my projects with what I learned to be reliable, easy-to-maintain, scalable code (even though these pet projects were never meant to scale). Nonetheless, I learned quite a few practices for code that I would love to see — but not always do — in the code I support in production during the day (and sometimes at night, when it breaks).</p><p>What I’d like to share with you here, is what I consider to be the best practices for writing code that stays reliable and helps the engineers who are tasked with keeping it running in production.</p><h3>No hard-coded defaults for your configurations</h3><p>Many guides for getting started with web servers will show you an example like that:</p><pre>port = process.env[&#39;PORT&#39;] || 8080<br>server.listen(port)</pre><p>But I would recommend something like so:</p><pre>port = process.env[&#39;PORT&#39;]<br>server.listen(port)</pre><p>“But wait” I hear you say, “that will cause my service to fail if the environment variable “PORT” is not defined!”. To which I’ll respond “Yes! Exactly!”. In that scenario, the service won’t be able to run in production without its configs set externally.</p><p>Here’s the thing, as SREs, we are not always familiar with your codebase, and usually, when we need to interact with your service in production, it’s an emergency and we need to learn things quickly. It’s easy and quick to learn on which port your service is supposed to listen on by looking at the environment variables, or inspecting a configuration file, but if your service can run without them, it might be tricky to find that.</p><p>Yes, we can also figure out the port by looking at netstat -tunap | grep LISTEN, SREs know their way around Linux, but what about the connection string to the database? API endpoints for a 3rd party service? The garbage collector settings or the name of an S3 bucket the service reads from? Please don’t define them in your code where SREs can’t easily find them.</p><h3>Validate ALL your configs before terminating (and fail fast)</h3><p>Getting the configs right is a big part of bringing up a service in production. One scenario I get a lot is this: A service fails to start with an error message:</p><pre>ERROR: Missing &#39;PORT&#39;</pre><p>When I define a port, I then get another one:</p><pre>ERROR: missing database connection string</pre><p>I then spend the following 30 minutes finding where the database is, its permissions, and where to define the configuration file, only to get another error. This cycle then repeats until after several hours I can finally get the service working.</p><p>A helpful service will start up verifying all its configuration, writing out all the errors it found before exiting, along with a helpful message on how one should go about fixing the issue. Like so:</p><pre>ERROR: Missing &#39;PORT&#39; environment variable<br>ERROR: Missing database connection string - definition in /etc/database.yaml - see http://service-documentation.info/database for details<br>WARNING: Heap size might be too low - see http://service-documentation.info/memory_requirements for details</pre><p>In code, that would be the “Bad” version:</p><pre>if(! isValidNumber(process.env[&#39;PORT&#39;]) ) {<br>  console.error(&quot;ERROR: Missing &#39;Port&#39;&quot;)<br>  process.exit(1)<br>}<br>if(! isValidConnectionString(config.get(&quot;dbConnectionString&quot;) ) {<br>  console.error(&quot;ERROR: Missing database connection string&quot;)<br>  process.exit(1)<br>}<br>initServer()</pre><p>And that would be the “Good” version:</p><pre>let isConfigValid = true<br><br>if(! isValidNumber(process.env[&#39;PORT&#39;]) ) {<br>  console.error(&quot;ERROR: Missing &#39;Port&#39;&quot;)<br>  isConfigValid = false<br>}<br>if(! isValidConnectionString(config.get(&quot;dbConnectionString&quot;) ) {<br>  console.error(&quot;ERROR: Missing database connection string&quot;)<br>  isConfigValid = false<br>}<br>if (!isConfigValid) process.exit(1)<br><br>initServer()</pre><h3>Version your APIs (and stay at least 1 version backwards compatible)</h3><p>One of the biggest challenges in maintaining a service in production is version updates. And a big reason for issues during version updates is a change to the behavior of a service’s APIs.</p><p>Let’s see that in an example.</p><p>Suppose you have a service that lists books. The API endpoint /books responds with an array of book names, like so:</p><pre>[<br>  &quot;Dune&quot;,<br>  &quot;Lord of the rings&quot;,<br>  &quot;1984&quot;<br>]</pre><p>That runs okay for a while, until a product manager asks that you’ll also display the author and the book’s cover picture. So now the /books endpoint responds with an array of objects, like so:</p><pre>[<br>  {<br>    &quot;title&quot;: &quot;Dune&quot;,<br>    &quot;author&quot;: &quot;Frank Herbert&quot;,<br>    &quot;cover_image&quot;: &quot;https://service-images.info/dune.png&quot;<br>  },<br>  { … },<br>  …<br>]</pre><p>The problem is that the old frontend of your app is expecting an array of strings, and will break when it gets objects. If your frontend is a web UI, that might not be too bad, only requiring a refresh (though if that’s cached in a CDN, that might still take a few hours), but if your frontend is a mobile app, it might take several months until old clients are updated.</p><p>To allow a smooth transition between versions, your API endpoints should be preceded with version numbers. In the above example, that API endpoint for /books will be /v1/books for the version returning strings, and /v2/books for the version returning objects. The updated version of your backend service should still support both versions. That way, it doesn’t matter how much time it takes for frontend apps to update, they won’t break — at least not because of the API.</p><p>As a rule of thumb, if you change something in the API but keep the default behavior the same, you can do so in the same version, but if you change the default behavior, that’s a breaking change that requires a new version.</p><h3>Continuously verify connection objects</h3><p>Most services have a dependency on an external data source. A database, a cache, sometimes both or something else entirely. A common strategy is to initiate the connection to a database as the service starts up, and use the resulting connection object on demand as necessary. This strategy has some great benefits: It prepares the connection in advance so the first request doesn’t suffer from a “cold start”, it allows re-use of the same connection object, and it also verifies the connection on startup so if it fails it fails fast.</p><pre>// on startup<br>const mysqlClient = mysqlLib.connect(MYSQL_CONNECTION_PARAMS)<br>// request logic<br>app.get(&#39;/:question&#39;, (req, res) =&gt; {<br>  let answer = mysqlClient.query(`…`)<br>  res.send(answer)<br>})</pre><p>But what happens if the connection fails during operation? This could happen for a multitude of reasons: Inactivity on the connection resulting in the database closing the connection, a momentary network issue, an issue with the database itself, etc. If the service only connects to the datasource at startup, it won’t reconnect after a temporary issue with the connection. For SREs that means that every time we tackle an outage caused by a loss of connection to the data source, we also need to restart all the instances of the service.</p><p>Here’s something you can do to avoid this: Verify that the connection to the datasource is alive <em>on every request</em> to the datasource, and reconnect before making the request if the connection is down. This strategy can be abstracted into a data module so as not to affect code readability, and many database libraries already include this strategy if you’re using a connection pool.</p><p>In javascript, it will look like this:</p><pre>// in the data module<br>function initDataModule(connectionParams) {<br>  const dataModule = {}<br>  const mysqlClient = mysqlLib.connect(connectionParams)<br>  dataModule.query = function(query) {<br>    if(!mysqlClient.isConnected()) {<br>      mysqlClient.reconnect(connectionParams)<br>    }<br>    return mysqlClient.query(query)<br>    }<br>  return dataModule<br>}<br><br>// init<br>dataModule = initDataModule(config.DBConfig)<br><br>// in app logic<br>app.get(&#39;/:question&#39;, (req, res) =&gt; {<br>  let answer = dataModule.query(`…`)<br>  res.send(answer)<br>})</pre><h3>Implement health and readiness endpoints</h3><p>The quickest way for an SRE to check if there’s an issue with the service is to check its health status. Usually, a monitoring utility will show green for healthy and red for unhealthy. Additionally, this is a signal for load balancers to send traffic to an instance and auto-scaler/auto-healer to hold off on scaling or restart an instance. Unfortunately, most services don’t implement a health check endpoint. There are two common levels:</p><ul><li>Healthy: Verifies the service is responsive</li><li>Ready: Verifies the service is expected to successfully respond to requests</li></ul><p>Following these rules will allow SREs to quickly zero-in on the source of the issue and find the appropriate way to mitigate it. If the service isn’t healthy, we can try to restart the instance. If the service is healthy but not ready, we’ll check the connection to the database, etc.</p><p>If you want to go the extra mile, when the health-check endpoint fails, log the reason and respond with it too.</p><h3>Structure your logs</h3><p>We all know the value of error messages and logs, and they really do help… If we can find them.</p><p>With millions of log lines, often coming from many different sources, its nearly impossible to find the cause of the problem, often we need to sift through many red herrings (the error messages that look like the problem but actually aren’t) before we find a single log message that relates to the actual issue, and by the time we do that our search query is so narrow that we don’t have the full context. Here are my recommendations for good metadata to go along with each log message that will make the search much more efficient:</p><ul><li>Log level (e.g. info, warn, error)</li><li>Name of the service reporting the log</li><li>Name of the module or function reporting the log</li><li>Request ID — A unique identifier that’s the same across all logs of the same request</li></ul><h3>Code for horizontal scalability (no singletons, no sticky sessions)</h3><p>Imagine a chat server that keeps all connected users in an array in-memory. When a chat message comes in, the server iterates over the array and sends the message to all connected users. What will happen when the scale of the service grows and we need to add more instances? If 2 users of the same chat room happen to connect to different servers, messages from the first user won’t reach the second user. In that situation, we will call the service a singleton, because we can’t run more than 1 instance of the service at a time. Singletons are a problem for SREs because with only one server running for the service, it doesn’t have any redundancy and will require down-time for version updates. In other words, singletons are fragile.</p><pre>let users = []<br><br>app.on(&#39;user_connects&#39;, (user) =&gt; {<br>  users.push(user)<br>})<br>app.on(&#39;message&#39;, (message) =&gt; {<br>  users.forEach(user =&gt; user.send(message))<br>})</pre><p>A similar, yet not quite the same story can happen in the following scenario. A service that keeps notes for users and holds them locally on each instance. In that case, there could be more than one instance for the service, as long as each user connects to the same instance as they did in the first place.</p><pre>let notes = { user: notes[] }<br><br>app.on(&#39;save_note&#39;, (user, note) =&gt; {<br>  users.push(user)<br>})<br>app.on(&#39;retrieve_notes&#39;, (user) =&gt; {<br>  notes[user].forEach(note =&gt; user.send(note))<br>})</pre><p>This case is called “sticky sessions” and has its own set of issues when running in production at scale, naming only some:</p><ul><li>Not all cloud providers have a load balancer that supports it</li><li>Scaling out doesn’t immediately take existing workload off from previously running instances (as they will still need to handle the same users)</li><li>Scaling in will require a redistribution of data, causing a momentary spike in network and I/O usage.</li></ul><p>In both these scenarios, putting a data-broker behind the service will remove the singleton and sticky sessions limitations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/0*M_mzlijlLiVTZD0G" /></figure><p>Data brokers like Redis or Kafka can scale in and out without losing the data they hold. They do the very complicated work of syncing data between them, choosing leaders by quorum, sharding the data when necessary and more — so you won’t have to. Implemented with care, a service can drop the singleton and sticky sessions limitations with only a slight increase in latency.</p><h3>If you can’t fix it, propagate it</h3><p>If my wife asks me to go to the supermarket and buy some cherries, and the supermarket is closed, or they didn’t have cherries, I need to tell my wife that I couldn’t get the cherries so she can plan ahead.</p><p>Take a look at the following example code:</p><pre>function getSomeData(abc) {<br>  response = []<br>  try {<br>    data = dbCall(abc)<br>    response = data<br>  catch(e) {<br>    console.error(&quot;something happened&quot;)<br>  }<br>  return response<br>}</pre><p>This function will return a valid value even when something goes wrong. So if we call that function as part of our API implementation we’ll have no way of knowing if something went wrong and therefore no way to reply with an error code. We will have to reply with a 200 “OK” even when there’s an issue, making it impossible to measure the error rate of this particular case and leaving us without the ability to set alerts that catch it.</p><h3>Disclaimer about best practices</h3><p>The recommendations described in this article are what I consider to be best practices. Best practices are guidelines that you should follow if there’s no good reason to do otherwise. Before using any of the advice described here, think if they fit your needs.</p><p>And with that, I wish you well. Now go out there and win some SRE’s heart!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0861798ac8f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/israeli-tech-radar/how-to-win-an-sres-heart-a-developer-s-guide-to-sturdy-production-code-0861798ac8f4">How to Win an SRE’s Heart — A developer’s guide to sturdy production code</a> was originally published in <a href="https://medium.com/israeli-tech-radar">Israeli Tech Radar</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you should use the plugin abstraction pattern]]></title>
            <link>https://medium.com/@lutraman/why-you-should-use-the-plugin-abstraction-pattern-3c709548734d?source=rss-25ff3ae28cda------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c709548734d</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[code]]></category>
            <dc:creator><![CDATA[Tom Klinovsky]]></dc:creator>
            <pubDate>Mon, 24 Oct 2022 21:57:31 GMT</pubDate>
            <atom:updated>2022-10-24T21:57:31.654Z</atom:updated>
            <content:encoded><![CDATA[<p>Abstractions in code are very important for the readability and maintainability of your code, but they are also very hard to get right. The following piece will present a tool you can use to abstract well in a few specific, yet common cases, such as connecting your application to a database, a memory store, blob storage, a message broker, etc.</p><h3>The General Idea</h3><p>Lets start with a metaphor. I recently had a daughter, Lorelai. Lorelai sometimes breastfeeds and sometimes drinks from a bottle, but to her, the entire concept is abstract, when a nipple (either her mom’s or the bottle’s) is introduced into her mouth, she calls nipple.suck(). As long as the nipple object provides a suck method, it can be used. So in that case, we can provide Lorelai with BreastNipple or BottleNipple and her eating code will work all the same.</p><p>You may already be familiar with the above concept as interfaces and their implementations. The advantage of using that pattern as a plugin for databases, or other external data services, is that <strong>your code will become agnostic to the type of database used, as well as to the entire logic applied to get the data to and from the database.</strong></p><p>Let’s look at some examples:</p><h3>Connecting to a database</h3><p>Let&#39;s say you are writing an app for a farmers&#39; market, and need APIs to add and get information about farmers. For that purpose, you selected MongoDB as your database. After reading the MongoDB documentation, this is the implementation you now have:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8259ef7e246d42713e05366395378fbb/href">https://medium.com/media/8259ef7e246d42713e05366395378fbb/href</a></iframe><p>Let&#39;s see how this would look as a plugin.</p><p>First, you create a file named farmers-data.js. Which will provide the methods to interact with the data. It will expose 3 methods:</p><ul><li>getFarmer</li><li>addFarmer</li><li>listFarmers</li></ul><p>Each of the methods will take care of connecting to the database and implement all the required boilerplate that comes with reading and writing to the database:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d62390560276fa78898fe05fb9cd18e/href">https://medium.com/media/9d62390560276fa78898fe05fb9cd18e/href</a></iframe><p>Now, the express router can use the plugin to implement its APIs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bb505ecf88eec223da7099d4629ee28/href">https://medium.com/media/5bb505ecf88eec223da7099d4629ee28/href</a></iframe><p>Notice that you were able to remove any MongoDB dependencies from the router. By implementing all database actions via the plugin in all files other than the plugin, you allow future changes to the type of database to be made easier, as they are limited in scope. It also increases the readability of the code, because the router now contains less code that isn’t the business logic, and the implementation of the database logic is limited in scope to the plugin files.</p><h3>Connecting to a BLOB storage</h3><p>The first example might have looked trivial to anyone who already has experience writing micro-services with databases. The following example will include a bit more creativity.</p><p>In this example, you will implement a micro-service to upload pictures via HTTP to be stored in a BLOB storage</p><p>You will need to implement 2 HTTP entry points for the service:</p><ul><li>/upload — to upload an image</li><li>/get/:name — to download a previously uploaded image</li></ul><p>This time, the API entry points will need to handle a stream of data, and use it to write the data to a file, or read it back and send it to a stream.</p><p>Here is the code implementing the HTTP entry points, <strong>without</strong> the use of a plugin:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0474ad3364b4535333e5935bd8b84ab3/href">https://medium.com/media/0474ad3364b4535333e5935bd8b84ab3/href</a></iframe><p>Before looking at the storage plugin implementation, let’s discuss the entry points logic, and what it will require from the plugin implementation to provide (i.e. what will the “contract” be).</p><p>The /upload endpoint requires a writable stream and the /get/:name endpoint requires a readable stream. So the methods getWritableStream and getReadableStream respectively can provide such abstractions.</p><p>Since the goal of the plugin is to abstract away the logic of reading and writing, in addition to providing a stream, the getWritableStream method will also need to provide the name it generated for the file (or object, if the implementation will later be changed), so it can later be retrieved with getReadableStream . To do that, the getWritableStream will return an object with a getID method.</p><p>Here is the implementation of the plugin:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89caa6e7c721da96323cf6661fa2a862/href">https://medium.com/media/89caa6e7c721da96323cf6661fa2a862/href</a></iframe><p>And the router using the plugin will be implemented as such:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b81668f18da85af5f60e42a67a124fd9/href">https://medium.com/media/b81668f18da85af5f60e42a67a124fd9/href</a></iframe><p>The main advantage of this approach is that for dev environments, a file system plugin can be used, and in production, as the application grows, the plugin could be replaced with a cloud BLOB storage such as GCS, with no changes to the code outside the plugin.</p><h3>A word about configuration</h3><p>Before moving on to the final example, I want to direct your attention to the configuration of the plugins.</p><p>All plugins will have some sort of configuration which is tightly coupled with the database service they are connecting to. For the first example, this was the MongoDB database and collection name, in the second example it was the directory name to store the files in. In the examples, the configuration was hardcoded for simple reading. In real-world applications, there are a few approaches to loading the configuration into the plugin, each with its advantages and disadvantages.</p><p>I recommend the following:</p><ol><li>Export an initialization function that will accept the config as its arguments and returns the initialized plugin.</li><li>From the main file of your application, load all the configs of your app, and use the relevant config for the plugin to initialize it.</li><li>If needed, pass the initialized plugin to any module that needs to use it.</li></ol><p>That will look like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f805829cf0d55d133e42ba7cf04ea09/href">https://medium.com/media/7f805829cf0d55d133e42ba7cf04ea09/href</a></iframe><p>A disadvantage of this approach is that the tightly coupled config is present in the main file, out of the plugin&#39;s scope. The reason I recommend it despite this is that it makes the code more readable to a newcomer who isn’t familiar with the code (like the SRE who is responsible for this code in production). The newcomer can look at the main file of the code to immediately understand where files are being written, and to which file to go next if they wish to debug the implementation of writing or reading the files.</p><h3>One last example: Video Streaming with Redis</h3><p>The latest product on our agenda is a video streaming service, using Redis as a backend.</p><p>The service will consist of an HTTP server, that on one side, provides a POST endpoint for the content producer to stream the video to, and on the other side, a GET endpoint for consumers to watch the video.</p><p>To distribute the video efficiently and across multiple instances of the application, you chose to use Redis. But, as by that point you can probably predict, this hardly matters. <strong>Your code is going to use a plugin.</strong> The plugin needs to provide a getWritableStream and a getReadableStream, the returned stream from getWritableStream needs to include a getID method so it can be retrieved later as an argument to the getReadableStream. Sound familiar?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b43b07aac97d5f42159518a74723e4cc/href">https://medium.com/media/b43b07aac97d5f42159518a74723e4cc/href</a></iframe><p>Notice that the initializeServer function takes in a streamProvider argument, it does not require nor use anything specifically related to Redis. It then registers a listener for the HTTP GET and HTTP POST requests, pointing them to the functions playbackStream (line 37) and recordStream (line 53) respectively, which in turn use the stream provider to get a readable or a writable stream. For all they care about, the stream could be written to Kafka, Google Pub/Sub, RabbitMQ, etc.</p><h3>To Summarize</h3><p>To create a well-designed data plugin you will need to:</p><ol><li>Define the functions the plugin needs to expose</li><li>Expose an initialization function to configure the plugin</li><li>Return the functions as per the definition</li></ol><p>Depending on your usage, and as your project grows, the plugin will also need to change. The contract of the plugin does not need to be, nor should it be static.</p><p>Using a plugin does not solve everything. If you need a unified way for many microservices to access a specific database, you will still need to create a library, and each microservice will have to use it to implement its plugin. That said, you can and should encourage your teams to build data plugins to perform actions on the database. That way, a developer from one team can easily understand the database interacting code of another team (and where to find it as well!).</p><p>Sometimes a plugin used in one service can be seamlessly used for a completely different purpose in another. Like in the case of the streams plugin, that can be implemented to a file system or Redis Streams by exporting the same functions.</p><p>Viewing the list of the plugin’s functions can help a developer gain a quick understanding of all the interactions of the service with its database: “This service can create a new farmer entry, edit it, and delete it — and these are the functions that do it”</p><p>A well-designed data plugin can simplify your code, make it much more readable for future comers, reduce vendor lock and increase flexibility. Good luck and happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c709548734d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing for Reliability — Using an Internet Door Buzzer as an Example]]></title>
            <link>https://medium.com/@lutraman/designing-for-reliability-using-an-internet-door-buzzer-as-an-example-eb654b7121d0?source=rss-25ff3ae28cda------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb654b7121d0</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <dc:creator><![CDATA[Tom Klinovsky]]></dc:creator>
            <pubDate>Sun, 26 Jan 2020 16:42:37 GMT</pubDate>
            <atom:updated>2020-01-28T09:43:51.405Z</atom:updated>
            <content:encoded><![CDATA[<h3>Designing for Reliability — Using an Internet Door Buzzer as an Example</h3><p>A few months back, I moved in to a new place, an old apartment building. Not too old, however, to have a door at the lobby, which posed a bit of a problem for me.</p><p>You see, every month I order my dogs’ food from the pet store, and their delivery times overlap with my working hours almost entirely. So usually they’ll leave it by my door and I’ll pull it in when I get back. But they can’t really do that if they can’t get passed the lobby door, and if they can’t, I’ll get billed for the delivery and not get the dog food. Not good.</p><p>Lucky for me, that door in the lobby can be buzzed open with an intercom installed in my apartment, and I am the proud owner of two unused Raspberry-Pi computers. Since the buzzer is just a button that closes a circuit, I could hook it up to the raspberry pi via a relay, the raspberry pi to the internet, and then I could buzz the door open from anywhere, at any time. Now all that’s left, is how to implement the program so I’ll be able to control it from my phone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c_CjbfEgQFdH1exUl-13ew.jpeg" /></figure><h3>Option #1 — The Naive Approach</h3><p>Thinking of the problem in the simplest terms, I can have the Raspberry-Pi trigger the buzzer in response to a POST request, and have a small web page with vanilla javascript display a button to send that POST request, and host it on the raspberry pi as well. That would work, but let’s stop for a second to think what could go wrong.</p><p>It’s important to me that my remote door buzzer not only works, but works when I need it to. There’s not much use for a cool IoT device if, when the delivery arrives and I press the button to let them in, I get a “Cannot connect to server” error, because the IP at my home router changed. Or if in a while I’ll switch my router, and I won’t be able to forward a connection to it. Or if the power goes out for a minute and resets my raspberry pi.</p><p>My point being, when you are designing a project, before jumping in to writing code, think of what could go wrong, and see what you could do to prevent it. One way to do so is to mitigate each issue separately, but if you detect many problems that are hard to tackle, try to step back and think if you can change the design of your project, replacing the harder issues with trivial ones.</p><h3>Option #2 — Complexity Is Not All Bad</h3><p>In my case, many of the issues originated from the fact that the Raspberry-Pi will always be behind my router. If I wanted to mitigate those issues in option #1, I would need to set my router to forward packets to its internal IP, I’ll have to set a static or reserved IP for it in my router, I’ll have to set up a service like no-ip to track my changing IP, and if I’ll ever change my router or internet provider, I’ll have to do all of this over again. Considering all that, lets try something else.</p><p>Instead of having the Raspberry-Pi exposed, I’ll have it connect on startup to a remote server on the cloud (which is always exposed), using a plain TCP connection. The server will hang on to the connection and keep it alive by sending periodical heart-beats. On the other side, it will also serve a web page with vanilla javascript, similar to the one from option #1, the only thing different is that upon receiving the POST request, the server will use its connection to the Raspberry-Pi to send a command to open the door. And just like that, no router related issues.</p><p>Now a critic might say that I’ve added a lot of complexity, I now have a server in the way that I didn’t have before, and that also needs to be maintained. And that’s true, I’ve exchanged my router problems with other problems, possibly more, and now I need to think if the deal is worthwhile before I take it.</p><p>Here are my new problems: the server could go down, the connection to the server could be lost, and the Raspberry Pi could still be reset by a power failure (it’s an old building, after all).</p><p>I’ve decided to take the deal and go with Option #2. Here’s why: A server on the cloud is a lot less likely to go down than any of the home internet issues are likely to occur. I can tackle the issue of losing the server connection by programming the Raspberry-Pi to reconnect to the server if it misses too many heart beats, and as for the issue with the power outage resets, I’ll just add my program to systemd using a unit file, and it will also restart it for me if the program crashes for any other reason. Most importantly, unlike option #1, in which if I ever switched my internet provider or my router, I’ll have to solve everything again, once I solve these problems, they will remain solved.</p><h3>What should you take from this?</h3><p>That was a really fun project, but how does this apply to the real life of a Software Engineer or a Site Reliability Engineer?</p><ol><li>Before starting a project, list your potential problems, and try to understand what it will take to mitigate them.</li><li>Sometimes a more complex design can remove hard problems with the price of other, easier to solve ones.</li><li>A problem that can be solved once is better than one that will require you to solve it again if something changes.</li><li>Keep everything else as simple as you can.</li></ol><h3>A few extra words</h3><p>I would like to thank my colleagues, Oleg, Natalie, Shai and Vitaly, for helping me with the hardware part of the project (and apologize to their managers for taking a bit of their time)</p><p>I currently have this buzzer up and running with monitoring, and as of the writing of these lines, it is up 99.78% of the time.</p><p>I did, for a while, had an issue that brought down the buzzer without having it self-heal. I eventually tracked it down to a race condition that caused the server to think the device was not connected to it when it actually was. Guess that goes to show that even with all that good Kubernetes self healing, you still need to write good code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb654b7121d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>