<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Emmanuel Christopher on Medium]]></title>
        <description><![CDATA[Stories by Emmanuel Christopher on Medium]]></description>
        <link>https://medium.com/@krissemmy17?source=rss-a99cbbeff7a4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KF-Opv0Ul8wD1vK0X8IkhA.png</url>
            <title>Stories by Emmanuel Christopher on Medium</title>
            <link>https://medium.com/@krissemmy17?source=rss-a99cbbeff7a4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:58:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@krissemmy17/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Cut a Startup’s Infrastructure Cost Without Self-Hosting Everything]]></title>
            <link>https://medium.com/@krissemmy17/how-i-cut-a-startups-infrastructure-cost-without-self-hosting-everything-9cd1415bce0e?source=rss-a99cbbeff7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cd1415bce0e</guid>
            <category><![CDATA[cloud-cost-optimization]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-infrastructure]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Emmanuel Christopher]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 23:05:38 GMT</pubDate>
            <atom:updated>2026-08-12T23:05:38.370Z</atom:updated>
            <content:encoded><![CDATA[<h4>A short migration from Vercel, Render and Supabase to Dokploy, R2 and a smaller managed stack, and the trade-offs that came with it.</h4><h3>The problem</h3><p>The easy answer was to move everything to one VPS.</p><p>That would also have been the dumb answer.</p><p>A startup brought me in because their infrastructure cost no longer made sense for their current stage. The application was spread across Vercel, Render, MongoDB Atlas and Supabase Storage, with separate production and staging environments.</p><p>The goal was simple:</p><blockquote>Reduce the monthly cost without creating a setup that would become a reliability nightmare later.</blockquote><p>By the end of the migration, the application was running on a Hetzner server through Dokploy, Redis was self-hosted, files had moved to Cloudflare R2, and MongoDB remained managed on Atlas.</p><p>The migration removed about <strong>$62.50/month in existing managed-service charges</strong>, while the replacement server cost around <strong>€8.99/month</strong> at the time.</p><p>But saving money was the easy part.</p><p>The interesting part was deciding <strong>what not to self-host</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ZHAqx8S-V9LiVXP3lI9T0Q.png" /></figure><h3>Before touching infrastructure, I started with the code</h3><p>Before creating the server, I went through the frontend and backend repositories.</p><p>The application was Next.js + NestJS, with Redis, BullMQ, MongoDB, file uploads, private PDFs, OAuth, email and payment integrations.</p><p>That inspection exposed a few things that immediately affected the migration:</p><ul><li>the backend handled both API requests and background jobs;</li><li>production used Redis for both cache and queue workloads;</li><li>some Next.js configuration was compiled during the build;</li><li>MongoDB documents contained full Supabase URLs;</li><li>public and private files needed different access rules;</li><li>production and staging were similar, but not identical.</li></ul><p>This is why I normally do not trust a migration plan based only on “we have these five services.”</p><p>A container can be healthy while the actual application is quietly pointing at the wrong API or generating links to storage you are about to delete.</p><h3>The VPS was for compute, not everything</h3><p>The final server configuration was:</p><ul><li>4 shared vCPU</li><li>8 GB RAM</li><li>80 GB disk</li><li>Dokploy</li><li>Traefik</li><li>production and staging applications</li><li>Redis</li></ul><p>I could have added MongoDB and object storage to that same server.</p><p>I deliberately didn’t.</p><p>If that VPS failed, I wanted to lose application compute, not the application <strong>and</strong> its database <strong>and</strong> its uploaded files at the same time.</p><p>MongoDB therefore stayed on Atlas, while Supabase Storage was replaced with Cloudflare R2.</p><p>That still reduced the bill considerably without turning one cheap server into the entire company.</p><h3>Redis looked simple until I looked at what it was doing</h3><p>Production had two Redis workloads:</p><ul><li>cache;</li><li>BullMQ queue.</li></ul><p>Both were Redis, but they did not have the same failure requirements.</p><h3>Cache data can disappear</h3><p>Cache data can be evicted. The application can rebuild it.</p><h3>Queue state is different</h3><p>Dropping queue keys under memory pressure can mean lost or inconsistent jobs.</p><p>So I kept them separate.</p><p>The cache used an eviction policy appropriate for temporary data, while the queue used noeviction and persistence.</p><p>It costs slightly more memory, but combining services simply because they use the same software is the kind of optimization that looks clever until production gets stressed.</p><h3>Storage migration was more annoying than Docker</h3><p>Moving the files themselves was straightforward once I stopped trying to be clever.</p><p>Cloudflare migration tooling did not cooperate with the Supabase S3-compatible endpoint, so I switched to rclone.</p><p>The migration became:</p><ol><li>inventory Supabase;</li><li>copy objects to R2 without deleting the originals;</li><li>compare object paths and sizes;</li><li>verify the destination.</li></ol><h3>Copying the files was only half the migration</h3><p>The more interesting problem was MongoDB.</p><p>Some records stored complete Supabase URLs.</p><p>So copying every file perfectly still would not have completed the migration. The database would continue sending users back to Supabase.</p><p>I wrote a targeted rewrite script that:</p><ul><li>scanned known fields;</li><li>ran in dry-run mode first;</li><li>confirmed the destination object existed;</li><li>saved rollback data;</li><li>changed only the intended URLs;</li><li>scanned again afterwards.</li></ul><p>In production, that process modified <strong>~100 documents</strong> and rewrote <strong>~200 URLs</strong>.</p><p>That was probably a better reminder than anything else in this project that infrastructure migrations are often <strong>data migrations wearing a Docker costume</strong>.</p><h3>Next.js had one more trap</h3><p>Initially, using one frontend image for production and staging sounded reasonable.</p><p>Then NEXT_PUBLIC_* entered the room.</p><p>Those variables are embedded into the browser bundle during next build.</p><p>Changing them later in Dokploy does not magically update already-built JavaScript.</p><p>So production and staging needed separate frontend builds from the same repository, each compiled against its own API.</p><p>It is something Vercel normally handles quietly for you.</p><p>Once you leave Vercel, congratulations, the abstraction has handed you your homework back.</p><h3>Cutover was more than checking /health</h3><p>I deployed staging first and used it to validate the new architecture.</p><p>Production came afterwards.</p><p>The final checks included more than HTTP health endpoints:</p><ul><li>signup;</li><li>email verification;</li><li>authenticated requests;</li><li>event creation;</li><li>private authorization;</li><li>R2 assets;</li><li>DNS and TLS;</li><li>database state;</li><li>background services.</li></ul><h3>Deployment, validation and deletion were separate stages</h3><p>The old providers stayed available during the validation period.</p><p>I treated: <strong>deployment → validation → deletion </strong>as three different stages.</p><p>That gave me a rollback path while I confirmed the new environment actually behaved like the old one.</p><h3>The result: lower cost, but more responsibility</h3><p>The startup ended up with a much smaller recurring infrastructure bill and a setup it actually controlled.</p><p>But the important trade-off was that some responsibility moved from Vercel and Render to the client:</p><ul><li>deployments;</li><li>backups;</li><li>server updates;</li><li>monitoring;</li><li>Redis persistence;</li><li>SSH access;</li><li>capacity planning;</li><li>rollback.</li></ul><p>That is the part people sometimes miss when talking about moving away from managed platforms.</p><p>The infrastructure gets cheaper because someone else is now responsible for some of the work the provider used to handle.</p><h3>Handover was part of the migration</h3><p>So I treated documentation as part of the migration itself.</p><p>The final handover covered:</p><ul><li>operations;</li><li>architecture;</li><li>backups;</li><li>migration history;</li><li>known risks;</li><li>what another engineer should know before changing anything.</li></ul><p>Because reducing an infrastructure bill is useful.</p><p>Reducing the bill and leaving behind a system only the consultant understands is just vendor lock-in with a human face.</p><p>I write about infrastructure, DevOps, reliability and the things I learn while working on real systems. More of my work is at <a href="https://krissemmy.com/">https://krissemmy.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cd1415bce0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Thought SOC 2 Would Be Checklists. It Turned Into an Infrastructure Engineering Project]]></title>
            <link>https://medium.com/@krissemmy17/i-thought-soc-2-would-be-checklists-it-turned-into-an-infrastructure-engineering-project-36c9a5f95515?source=rss-a99cbbeff7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/36c9a5f95515</guid>
            <category><![CDATA[nebius]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[soc-2-compliance]]></category>
            <category><![CDATA[drata]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Emmanuel Christopher]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 09:54:32 GMT</pubDate>
            <atom:updated>2026-07-10T09:54:32.829Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u8T3h20v5CDOxUhPAnFpVg.png" /></figure><p>For the past few weeks, I have been leading the technical side of our SOC 2 Type II preparation.</p><p>To be honest, when we first started, I expected most of the work to involve policies, screenshots, answering questions inside Drata, and probably chasing people around to approve documents.</p><p>Well, some of it is exactly that.</p><p>But the deeper I got into the infrastructure side, the more I realised that a lot of compliance work is really just infrastructure engineering with a requirement to prove that what you built actually works the way you claim it does.</p><p>It is one thing to say:</p><ul><li>Our backups are enabled.</li><li>Production access is restricted.</li><li>SSH is secure.</li><li>Storage buckets are versioned.</li><li>Infrastructure changes go through review.</li><li>Databases are monitored.</li></ul><p>It is another thing entirely to answer:</p><blockquote>How do you know?</blockquote><p>And even after answering that, there is another question:</p><blockquote>How will you still know three months from now after the infrastructure has changed?</blockquote><p>That second question is where most of my work started becoming less about compliance paperwork and more about automation, infrastructure security, and evidence collection.</p><p>For context, I currently lead Cloud Platform, DevOps, and infrastructure at RestoreAI. We run workloads across AWS and Nebius, use GitHub for version control and CI/CD, and use Drata as the compliance platform.</p><p>Since I already own most of the infrastructure involved, I also took ownership of the technical remediation work.</p><p>Not just answering failed tests.</p><p>Actually fixing the infrastructure behind them.</p><h3>The first lesson: a failed compliance test is usually trying to tell you something useful</h3><p>When we connected AWS to Drata, quite a number of tests failed.</p><p>Some were straightforward:</p><ul><li>VPC Flow Logs were not enabled.</li><li>Some storage buckets did not have versioning.</li><li>Database monitoring alarms were missing.</li><li>Default networking resources were more permissive than they needed to be.</li><li>Public SSH access was still available.</li><li>Threat detection was not enabled.</li></ul><p>Of course, it is easy to look at a long list of failed controls and think:</p><blockquote><em>How do I make all of these green?</em></blockquote><p>But that is probably the wrong first question.</p><p>The better question is:</p><blockquote><em>Why did this fail, and is the finding actually correct?</em></blockquote><p>Most times, it was.</p><p>That does not mean every failed test should be blindly followed. Some controls do not apply equally to every resource, and some resources genuinely need an exception.</p><p>But when a test points at an open SSH rule, a publicly reachable database, missing backup protection, or an overly privileged identity, pretending it is only a compliance problem does not change the fact that it is still an infrastructure problem.</p><p>So I treated the findings as engineering work.</p><h3>Cleaning up AWS networking without breaking anything</h3><p>One of the first areas I worked on was the development VPC.</p><p>The default security group still had rules attached to it, and workloads were depending on networking defaults that were broader than I wanted.</p><p>I removed the rules from the default security group and moved workloads onto dedicated security groups created for their actual purpose.</p><p>The same idea was applied to the default network ACL.</p><p>The goal was simple:</p><blockquote><em>Default resources should not quietly become the place where every workload gets unrestricted access.</em></blockquote><p>Each workload should have networking rules that explain what it needs and nothing more.</p><p>Of course, changing security groups and network ACLs can easily break connectivity.</p><p>So before making any change, I captured the current behaviour first.</p><p>Could the application reach the database?</p><p>Could services communicate with each other?</p><p>Which ports were actually needed?</p><p>After changing the rules, I repeated the same checks.</p><p>This part may sound obvious, but security hardening without before-and-after validation is one of the fastest ways to create an outage while trying to improve security.</p><p>The control becoming green is nice.</p><p>The application still working afterwards is more important.</p><h3>Replacing public SSH access on AWS</h3><p>Another finding was public SSH access.</p><p>Historically, SSH had been available through restricted inbound rules.</p><p>That is already better than opening port 22 to the entire internet, but the better solution was to remove public SSH access completely.</p><p>For AWS workloads, I moved administrative access to AWS Systems Manager Session Manager.</p><p>That involved:</p><ul><li>attaching the required IAM instance profiles,</li><li>confirming the SSM agent was installed and running,</li><li>validating that the instances appeared as managed nodes,</li><li>testing shell access through Session Manager,</li><li>and only then removing the public SSH rules.</li></ul><p>The result was cleaner:</p><ul><li>no public port 22,</li><li>no SSH keys being passed around,</li><li>access controlled through AWS IAM,</li><li>and session activity available for audit.</li></ul><p>This is one of those cases where the compliance requirement pushed the infrastructure towards a better operating model.</p><p>Not just something that looks better inside Drata.</p><p>Something that is actually easier to control.</p><h3>Nebius did not have SSM, so I used Tailscale instead</h3><p>AWS already had a mature answer for private administrative access through SSM.</p><p>Nebius did not.</p><p>The easy option would have been to leave SSH exposed publicly and restrict it to a known IP address.</p><p>That works, but it still leaves a public SSH endpoint, and it becomes inconvenient once people change networks or need access from different locations.</p><p>So I introduced Tailscale.</p><p>Instead of reaching the servers through their public addresses, authorised users connect through the private Tailscale network.</p><p>After validating access through the Tailscale IPs, I removed public SSH exposure from the security groups.</p><p>The result was similar to what I wanted from SSM:</p><ul><li>no public SSH endpoint,</li><li>access limited to approved identities,</li><li>private connectivity between authorised devices and servers,</li><li>and less dependency on static office/home IP addresses.</li></ul><p>Different cloud provider, different implementation.</p><p>Same control objective.</p><p>That is something I have noticed a lot during the SOC 2 work.</p><p>The requirement normally describes what should be true.</p><p>It does not always tell you exactly how to make it true.</p><p>That part is still an engineering decision.</p><h3>Enabling network visibility with VPC Flow Logs</h3><p>Another failed control was VPC Flow Logging.</p><p>So I enabled VPC Flow Logs and configured the logs to be delivered to S3.</p><p>But enabling it manually in the AWS console would only solve the problem for that moment.</p><p>Someone could remove it later.</p><p>Another environment could be created without it.</p><p>Or the configuration could drift without anyone noticing.</p><p>So after validating the flow logs, I moved the configuration into Terraform and imported the existing resources into state.</p><p>That way, the control became part of the infrastructure definition instead of something that only existed because I remembered to click a button.</p><p>This became a pattern for most of the work:</p><ol><li>Fix the actual infrastructure.</li><li>Verify that it works.</li><li>Codify it.</li><li>Let the compliance platform confirm the result.</li></ol><p>I prefer that over uploading a screenshot and hoping nobody changes the resource later.</p><h3>Adding GuardDuty for threat detection</h3><p>Drata also flagged that threat detection was not in place.</p><p>For AWS, I enabled GuardDuty.</p><p>GuardDuty analyses activity from AWS data sources such as CloudTrail events, VPC Flow Logs, and DNS activity to identify suspicious behaviour.</p><p>The important part for me was not simply turning it on.</p><p>I also needed to think about:</p><ul><li>which regions were in use,</li><li>where findings would be reviewed,</li><li>who would own the response,</li><li>and what happens when a real finding appears.</li></ul><p>A security service that generates alerts nobody checks is mostly decoration.</p><p>So the implementation has to include an operational process around it.</p><p>Who receives the finding?</p><p>How is severity handled?</p><p>Does it become a ticket?</p><p>Who investigates it?</p><p>A lot of compliance tooling can prove that something is enabled.</p><p>It cannot prove that your team will respond properly unless the process also exists.</p><h3>S3 versioning, access logging, and the problem with blanket controls</h3><p>Storage was another major area.</p><p>I enabled object versioning and access logging where required and made versioning a standing requirement for new buckets going forward.</p><p>I also removed a couple of unused buckets instead of spending time securing infrastructure we no longer needed.</p><p>One thing I learned quickly is that applying every control blindly can create unnecessary cost.</p><p>For example, audit logging on a high-read storage bucket can generate a lot of extra events.</p><p>So instead of saying:</p><blockquote><em>Every bucket must use the most expensive logging configuration possible.</em></blockquote><p>I reviewed how each bucket was used and selected the level of logging that provided useful evidence without creating avoidable cost.</p><p>Compliance should make infrastructure more controlled.</p><p>It should not remove engineering judgement.</p><h3>Backup enabled does not mean restore works</h3><p>One of the controls focused on backups.</p><p>At first glance, this sounds simple.</p><p>Enable automated backups.</p><p>Set retention.</p><p>Move on.</p><p>But a backup that has never been restored is still an assumption.</p><p>So instead of stopping after confirming backups were enabled, I created a temporary database instance, inserted test data, allowed the automated backup process to run, and then performed a documented restore drill.</p><p>The goal was to prove the entire process:</p><ul><li>backup was created,</li><li>backup was available,</li><li>restore completed,</li><li>database became reachable,</li><li>data was present,</li><li>application-level access could be validated.</li></ul><p>Only after proving the procedure on a disposable instance would I consider using the same process against production.</p><p>This was probably one of the most useful parts of the work.</p><p>Because during an actual database incident, “the backup status says available” is not enough.</p><p>You need to know the restore procedure works.</p><h3>Not every resource should be forced into the same control</h3><p>There was also a backup finding for a table that did not contain business data.</p><p>It was being used for distributed locking.</p><p>Enabling point-in-time recovery on it would not provide meaningful recovery value because the data was temporary by design.</p><p>So rather than enabling the feature just to make the test green, I documented why the resource was excluded.</p><p>That distinction matters.</p><p>An exception is not the same thing as ignoring a finding.</p><p>A proper exception should explain:</p><ul><li>which resource is affected,</li><li>why the control does not provide meaningful value,</li><li>what risk exists,</li><li>why the risk is acceptable,</li><li>who approved it,</li><li>and whether the decision should be reviewed later.</li></ul><p>A green dashboard achieved by pretending every resource is identical is not useful.</p><h3>Separating CI permissions from privileged infrastructure</h3><p>IAM was another area where I wanted to avoid choosing between two bad options.</p><p><strong>The first bad option:</strong></p><p>Give the CI/CD identity AdministratorAccess because Terraform needs to create everything.</p><p><strong>The second bad option:</strong></p><p>Give CI very limited access and then keep manually fixing Terraform failures whenever it reaches IAM, KMS, audit trails, or other sensitive resources.</p><p>Instead, I designed a two-tier Terraform model.</p><p>The first tier contains infrastructure that CI can safely apply through GitHub Actions and Terragrunt.</p><p>The second tier contains privileged resources that require admin approval and are applied separately.</p><p>For example:</p><ul><li>IAM identities,</li><li>sensitive key-management resources,</li><li>audit configuration,</li><li>privileged security controls.</li></ul><p>The automation identity remains least-privileged.</p><p>Highly sensitive changes remain explicit.</p><p>And both sets of infrastructure are still defined as code.</p><p>This also solved one of the common arguments around Terraform permissions.</p><p>The answer does not always have to be:</p><blockquote><em>Give Terraform admin access because infrastructure as code needs everything.</em></blockquote><p>Sometimes the better answer is to separate the trust boundaries.</p><h3>The bigger problem: Nebius had no native Drata integration</h3><p>AWS integration was straightforward because Drata already had a connector and built-in tests.</p><p>Nebius was different.</p><p>We had production infrastructure running there, but Drata could not see any of it.</p><p>No compute instances.</p><p>No disks.</p><p>No VPCs.</p><p>No security groups.</p><p>No managed PostgreSQL configuration.</p><p>No object storage.</p><p>No IAM posture.</p><p>From the compliance platform’s point of view, that environment did not exist.</p><p>The easiest workaround would have been manual evidence.</p><p>Run commands.</p><p>Export JSON.</p><p>Upload files.</p><p>Repeat every few weeks.</p><p>I did not like that approach.</p><p>It would technically work, but it would turn evidence collection into recurring manual work and leave too much room for stale evidence.</p><p>So I built the integration myself.</p><h3>Building an evidence collector</h3><p>The first version was a Python evidence collector.</p><p>It authenticated using a dedicated read-only service account and collected configuration from several infrastructure areas:</p><ul><li>compute instances,</li><li>disks,</li><li>VPC networking,</li><li>security groups and rules,</li><li>managed PostgreSQL,</li><li>object storage,</li><li>IAM.</li></ul><p>The service account was intentionally limited to read-only access.</p><p>It did not need permission to modify infrastructure.</p><p>It only needed enough access to describe the current state.</p><p>The collector generated evidence artifacts and uploaded them into Drata’s Evidence Library on a schedule.</p><p>That already improved the process because the evidence could refresh automatically.</p><p>But it still had one limitation.</p><p>A human still needed to open the uploaded evidence and interpret it.</p><p>So I went one step further.</p><h3>Moving from uploaded evidence to structured records</h3><p>Instead of only uploading files, I built a Drata Custom Connection.</p><p>The collector now normalises infrastructure data into structured records and sends those records to Drata.</p><p>Drata can then evaluate the records using custom tests.</p><p>The difference is important.</p><p>The first version looked like this:</p><pre>Nebius → Python collector → Evidence file → Drata → Human review</pre><p>The newer version looks more like this:</p><pre>Nebius → Python collector → Structured records → Drata custom tests → Pass/Fail</pre><p>Now the compliance platform can evaluate the infrastructure directly.</p><p>For example:</p><ul><li>Are disks encrypted?</li><li>Is public SSH exposed?</li><li>Is public RDP exposed?</li><li>Is the managed database publicly reachable?</li><li>Is object versioning enabled?</li><li>Is storage audit logging enabled?</li><li>Is database backup retention configured?</li></ul><p>Instead of uploading a JSON file and asking someone to inspect it, the platform receives structured data and owns the verdict.</p><p>Green when the configuration passes.</p><p>Red when it drifts.</p><p>That is much closer to what I mean by evidence as code.</p><h3>Automating the pipeline with GitHub Actions</h3><p>The entire evidence pipeline runs through GitHub Actions.</p><p>Structured records refresh frequently.</p><p>Full evidence artifacts run on a separate schedule.</p><p>Credentials are stored in a protected GitHub environment.</p><p>The service account is read-only.</p><p>Nothing sensitive is committed into the repository.</p><p>I also added cleanup around Drata upload sessions because the evidence API required uploads to follow a complete lifecycle.</p><p>Create session.</p><p>Upload artifact.</p><p>Complete session.</p><p>Clean up properly.</p><p>One incomplete upload could leave unnecessary state behind, so the workflow had to handle failures properly too.</p><p>There were also some interesting integration issues along the way.</p><p>At one point, requests using a default client were blocked at the edge.</p><p>There were strict schema requirements.</p><p>Some fields looked optional but were effectively required.</p><p>And a few test records failed because the payload structure was technically valid JSON but not valid for the API contract.</p><p>Normal integration work, basically.</p><p>The difference is that the output was not a customer-facing feature.</p><p>It was a compliance system evaluating cloud infrastructure.</p><h3>The custom tests found real problems</h3><p>The good thing about automated controls is that they do not care who wrote them.</p><p>They still fail when the infrastructure is wrong.</p><p>Some of the custom tests I created immediately surfaced issues such as:</p><ul><li>storage buckets without versioning,</li><li>missing audit logging,</li><li>security group exposure,</li><li>and a managed database that was publicly reachable.</li></ul><p>The database case was interesting.</p><p>The provider did not support the networking setup we originally expected, and the application and database were on different networks.</p><p>The database therefore needed public reachability for the application to connect.</p><p>It would have been easy to mark the test green manually and move on.</p><p>Instead, I documented the infrastructure limitation, created an interim exception, and planned a proper migration so both workloads can exist on the same network and public database access can be removed.</p><p>A red control with a real remediation plan is better than a fake green control.</p><h3>Compliance as code is not about making every dashboard green</h3><p>After spending weeks inside Drata, Terraform, AWS, Nebius, GitHub Actions, IAM, networking, storage, and databases, this is probably the biggest lesson I have taken from the work:</p><p>Compliance automation is not about finding creative ways to make failed tests disappear.</p><p>The useful part is reducing the difference between:</p><blockquote><em>We believe this is configured properly.</em></blockquote><p>and:</p><blockquote><em>We can continuously prove that this is configured properly.</em></blockquote><p>Sometimes the proof comes from a native connector.</p><p>Sometimes it comes from Terraform.</p><p>Sometimes it comes from logs.</p><p>Sometimes it comes from a restore drill.</p><p>And sometimes the provider does not support the integration you need, so you build the bridge yourself.</p><h3>What changed in the way I think about SOC 2</h3><p>Before starting this work, I mostly associated SOC 2 with policies and audits.</p><p>Now I see it differently.</p><p>A lot of the technical work comes down to normal infrastructure questions:</p><ul><li>Who can access production?</li><li>How is that access approved?</li><li>Can administrative ports be removed from the internet?</li><li>Are infrastructure changes reviewed?</li><li>Can the environment detect suspicious activity?</li><li>Are backups actually recoverable?</li><li>Can configuration drift be detected?</li><li>Can the company prove all of this without collecting screenshots every quarter?</li></ul><p>Those are not only audit questions.</p><p>They are reliability and security questions.</p><p>The audit just forces you to answer them clearly.</p><h3>Final thoughts</h3><p>We are still deep in the preparation work, so I am sure there will be more findings, policies, exceptions, controls, and probably more unexpected engineering problems.</p><p>But so far, the part I have enjoyed most is turning compliance from something passive into something the infrastructure participates in.</p><p>The infrastructure reports its state.</p><p>The tests evaluate it.</p><p>The evidence refreshes automatically.</p><p>And when something changes, the control should go red honestly instead of staying green because the last screenshot looked good.</p><p>That is the direction I want to continue pushing:</p><p>Less manual evidence.</p><p>Less “trust me.”</p><p>More infrastructure that can prove what it is doing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36c9a5f95515" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Helped a Startup Move From Managed Cloud to a Self-Hosted Coolify Setup on Hetzner]]></title>
            <link>https://medium.com/@krissemmy17/how-i-helped-a-startup-move-from-managed-cloud-to-a-self-hosted-coolify-setup-on-hetzner-ff7bec8cb35d?source=rss-a99cbbeff7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff7bec8cb35d</guid>
            <category><![CDATA[cloud-migration]]></category>
            <category><![CDATA[hetzner]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[coolify]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Emmanuel Christopher]]></dc:creator>
            <pubDate>Wed, 29 Apr 2026 20:05:04 GMT</pubDate>
            <atom:updated>2026-04-29T20:32:16.990Z</atom:updated>
            <content:encoded><![CDATA[<h4>A practical breakdown of moving a startup application to Hetzner and Coolify, including Docker issues, PostgreSQL migration, Redis auth, backups, and handover documentation.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xwlibXDWPsPzq-2juXSojw.png" /></figure><p>Recently, I worked with a startup that wanted to move its production application from a mostly managed cloud setup to a more controlled, self hosted deployment environment.</p><p>The biggest business driver was cost.</p><p>Their cloud hosting cost was around $300/month on AWS. After the migration, the core hosting layer moved to a Hetzner server costing about €8/month.</p><p>That is a big difference for an early-stage team.</p><p>But the goal was not just to “move the app to a cheaper server.” That would have been the easy way to describe it, but not the real work.</p><p>The real goal was to reduce infrastructure cost while also giving the team more control over deployments, databases, backups, and day to day operations without forcing them to manually manage every container, SSL certificate, reverse proxy rule, and server process by hand.</p><p>The setup we moved toward was Hetzner for the server layer and Coolify as the self hosted platform for deployments and service management.</p><p>For privacy reasons, I am leaving out the company name, domains, exact infrastructure values, and sensitive implementation details.</p><h3>The Starting Point</h3><p>The startup already had an application running with a frontend, backend, PostgreSQL database, Redis, object storage, and several third party integrations.</p><p>Like many early stage products, the infrastructure had grown around speed. That is normal. At the MVP stage, the priority is usually to ship the product, validate the idea, and keep the team moving.</p><p>But once a product gets closer to production usage, the infrastructure needs to become easier to reason about.</p><p>The team wanted:</p><ul><li>A much lower hosting bill than the existing AWS setup</li><li>A production environment</li><li>A test or staging environment</li><li>GitHub based deployments</li><li>Dockerized frontend and backend services</li><li>PostgreSQL and Redis managed inside Coolify</li><li>Migration from an existing managed PostgreSQL database</li><li>S3 compatible backup storage</li><li>Clear handover documentation so the team could operate the system without depending on me for every small change</li></ul><p>That last point mattered a lot.</p><p>A migration is not complete just because the app is running. It is complete when the team can understand, operate, debug, and recover the system after handover.</p><h3>Why Coolify Made Sense</h3><p><a href="https://coolify.io/"><strong>Coolify</strong></a> was a good fit because the team needed something in between fully managed platforms and raw server management.</p><p>Running everything manually on a VPS gives you control, but it also gives you more things to maintain. You have to think about Docker networking, reverse proxies, certificates, environment variables, deployment commands, logs, backups, service restarts, and health checks.</p><p>Coolify gives you a self hosted PaaS style workflow while still letting you own the server.</p><p>With Coolify, the team could manage:</p><ul><li>Applications</li><li>Databases</li><li>Redis</li><li>Environment variables</li><li>Domains</li><li>Deployment logs</li><li>Webhook based deploys</li><li>Backups</li><li>Service health checks</li></ul><p>For a small startup team, that tradeoff made sense. They got more control than a typical managed app platform, but avoided turning every deployment into manual server work.</p><h3>Setting Up the Infrastructure</h3><p>The server was provisioned on Hetzner and later resized to a more suitable instance size for the expected production workload.</p><p>After that, Coolify was installed and configured as the deployment control plane.</p><p>DNS records were created for the Coolify console, the frontend app, and the backend API. The frontend and backend were deployed as separate services.</p><p>Where possible, internal service URLs were used for private communication between services instead of exposing everything publicly.</p><p>That matters because not every service needs to be public. A cleaner DNS surface reduces unnecessary exposure and makes the setup easier to reason about.</p><h3>Containerizing the Backend</h3><p>The backend was deployed with a production ready Dockerfile using a multi stage build.</p><p>The idea was simple:</p><ul><li>Install dependencies</li><li>Build the application</li><li>Generate the required runtime artifacts</li><li>Copy only what was needed into the final runtime image</li><li>Run the application from a leaner production container</li></ul><p>But one issue came up during deployment.</p><p>The app used Prisma, and Prisma Client was generated during the build stage. The final runtime image initially used production dependencies from another stage where Prisma had not been generated correctly with the actual schema.</p><p>The result was a runtime failure when the application tried to import Prisma enums.</p><p>The fix was to explicitly copy the generated Prisma Client artifacts from the build stage into the runtime image.</p><p>This is the type of bug that can be annoying because the Docker image can build successfully while still failing at runtime.</p><p>That is one of the biggest lessons from container work: a successful build does not mean the container is production ready. You still need to run it as the final artifact and verify that the application actually starts with the same files, environment variables, and dependencies it will have in production.</p><h3>Deploying the Frontend</h3><p>The frontend was a Vite based app and was served through Nginx on port 80.</p><p>One important detail with frontend deployments is that environment variables often behave differently than backend variables.</p><p>For Vite and similar frontend build tools, public environment variables are usually injected at build time. That means changing an environment variable in the platform does not always affect the already built frontend bundle.</p><p>So for frontend changes involving environment variables, the team needed to understand when to rebuild, when to redeploy, and when to use a no cache deployment.</p><p>This was documented clearly because it is one of those small things that can waste hours during production debugging.</p><h3>Migrating PostgreSQL</h3><p>The production database was migrated from the existing managed PostgreSQL instance to a Coolify managed PostgreSQL deployment.</p><p>The source database was running PostgreSQL 17, so the dump needed to be created with a compatible PostgreSQL 17 client.</p><p>Instead of wasting time fighting local PostgreSQL client versions, I used the official PostgreSQL Docker image to run the migration tooling.</p><p>The migration flow looked like this:</p><ol><li>Create a compressed custom format dump from the source database</li><li>Validate the dump contents with pg_restore -l</li><li>Restore the dump into the new PostgreSQL database</li><li>Confirm tables, row counts, and important records</li><li>Update the backend database connection URL</li><li>Redeploy the backend</li><li>Verify the application behavior against the migrated database</li></ol><p>The important part here is verification.</p><p>A restore command can complete successfully and still leave you with questions. Did the expected tables come over? Are key records present? Are row counts reasonable? Does the app work after pointing to the new database?</p><p>For production migrations, I do not like treating “command completed” as proof that the migration is done. The application needs to prove it too.</p><h3>Redis Configuration</h3><p>Redis was deployed through Coolify as a managed service.</p><p>The backend was updated to prefer a full REDIS_URL when available instead of assuming separate host and port values.</p><p>This made the application easier to configure across environments because Coolify provides service connection URLs directly.</p><p>A Redis authentication issue also came up during deployment:</p><pre>WRONGPASS invalid username-password pair or user is disabled</pre><p>The root cause was a mismatch between the Redis URL used by the application and the actual Redis credentials from Coolify.</p><p>The practical fix was to test the connection from inside the running backend container and confirm the exact URL format the application needed.</p><p>This is another simple but important habit: when debugging service connectivity, test from the same network and runtime context as the application. Testing from your laptop is not always enough.</p><h3>Third Party Integrations</h3><p>The app depended on several external services, including logistics, email, authentication, payments, maps, Firebase, and object storage.</p><p>During the migration, each integration had to be reviewed carefully.</p><p>Some services were no longer needed after the move. Some still had to remain external. Some needed new environment variables. Some required files to be included correctly in the backend image.</p><p>For example:</p><ul><li>AWS database usage was removed after the database migration</li><li>S3 compatible object storage remained useful for backups and media related workflows</li><li>Firebase service account files had to be included correctly in the backend container</li><li>External logistics APIs had to be tested carefully after deployment</li></ul><p>One production issue involved a logistics provider rejecting a delivery pricing request because the app was sending an invalid package type payload.</p><p>The fix was not an infrastructure fix in the narrow sense. It required normalizing the shipment payload before calling the external API.</p><p>This is common during migrations. You touch infrastructure, but you also expose application assumptions that were hidden before.</p><h3>Backups</h3><p>Database backups were configured through Coolify and connected to an S3 compatible bucket.</p><p>The goal was simple: the database should not depend only on the server disk.</p><p>The backup documentation covered:</p><ul><li>Where backups are stored</li><li>How S3 storage is configured in Coolify</li><li>How daily PostgreSQL backups are scheduled</li><li>How to restore from a backup</li><li>Why restores are destructive and should be handled carefully</li></ul><p>This part is not glamorous, but it matters.</p><p>A backup that nobody knows how to restore is not a real backup strategy. It is just a file sitting somewhere.</p><h3>Deployment Workflow</h3><p>GitHub webhooks were connected to Coolify so pushes to the main branch could trigger deployments automatically.</p><p>That is convenient, but it also introduces risk.</p><p>If every push to main deploys to production, then main needs to be treated carefully. Developers should avoid pushing unfinished work directly to main. Feature branches and pull requests are safer, especially when the app is already production facing.</p><p>The handover covered how to:</p><ul><li>View deployment logs</li><li>Restart services</li><li>Redeploy applications</li><li>Force deploy without cache</li><li>Update environment variables</li><li>Check whether changes were actually picked up by the running container</li></ul><p>This gives the team a practical workflow for normal operations and basic debugging.</p><h3>Handover Documentation</h3><p>One of the most important deliverables was the handover document.</p><p>It covered:</p><ul><li>How to save and use SSH keys</li><li>How to connect to the server</li><li>Fallback SSH commands if local SSH config fails</li><li>How to access Coolify</li><li>How to upgrade Coolify</li><li>How to resize the Hetzner server</li><li>How to view application deployments</li><li>How to update environment variables</li><li>How to change application domains</li><li>How to connect to PostgreSQL</li><li>How backups and restores work</li><li>Files changed during the migration</li><li>Common troubleshooting notes</li></ul><p>This is the part many engineers underrate.</p><p>Good infrastructure work should reduce dependency on the person who set it up. If the only person who understands the system is the consultant, then the system is fragile.</p><p>My goal was not just to make it work. It was to leave the team with something they could operate.</p><h3>What I Watch Closely After a Migration Like This</h3><p>Self hosting can reduce cost and increase control, but it is not magic.</p><p>After a move like this, I usually watch a few things closely:</p><ul><li>Backup success and restore readiness</li><li>Server disk usage</li><li>CPU and memory pressure</li><li>Database growth</li><li>Deployment failures</li><li>Application logs</li><li>SSL and DNS issues</li><li>Redis connectivity</li><li>Third party API failures</li><li>Whether developers are pushing directly to main too often</li></ul><p>The biggest mistake would be assuming the move is finished forever after the first successful deployment.</p><p>Infrastructure needs operational habits around it.</p><h3>Lessons From the Project</h3><p>A few lessons stood out.</p><p>First, self hosting reduces some costs, but it increases responsibility. Someone now needs to care about backups, upgrades, monitoring, server access, deployment safety, and recovery.</p><p>Second, Docker images need to be tested as runtime artifacts. A clean build is not enough. The final container must start correctly with the same files and environment it will have in production.</p><p>Third, frontend environment variables need special attention. In Vite and similar tools, they are usually baked into the build, so changing them later may require a rebuild.</p><p>Fourth, database migrations should be verified with row counts, table checks, and real application behavior. Do not trust the migration only because the restore command succeeded.</p><p>Finally, handover documentation is part of the infrastructure. If the team cannot operate the system after the migration, the work is unfinished.</p><h3>Final Result</h3><p>The startup ended up with a self hosted deployment environment on Hetzner managed through Coolify.</p><p>The final setup gave the team:</p><ul><li>A reduction in core hosting cost from around $300/month on AWS to about €8/month on Hetzner</li><li>More control over infrastructure cost</li><li>Docker based frontend and backend deployments</li><li>PostgreSQL and Redis managed through Coolify</li><li>GitHub triggered deployments</li><li>S3 compatible database backups</li><li>Cleaner operational documentation</li><li>A deployment workflow the team could continue using after handover</li></ul><p>The work was not just about moving containers from one place to another.</p><p>It involved database migration, Docker debugging, Redis auth fixes, third party integration checks, backup setup, DNS configuration, deployment workflow design, and handover documentation.</p><p>That is usually what real infrastructure work looks like.</p><p>Not just “the app is live.”</p><p>The better question is: can the team deploy it, debug it, back it up, restore it, and keep it running without panic?</p><p>That was the real goal of this migration.</p><p>#DevOps #Coolify #Hetzner #Docker #PostgreSQL #StartupInfrastructure #Self-Hosting #Infrastructure #CloudMigration #PlatformEngineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff7bec8cb35d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I upgrade blockchain nodes with near-zero downtime in production]]></title>
            <link>https://medium.com/@krissemmy17/how-i-upgrade-blockchain-nodes-with-near-zero-downtime-in-production-fefd601bd6e7?source=rss-a99cbbeff7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/fefd601bd6e7</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[blockchain-node]]></category>
            <category><![CDATA[rpc]]></category>
            <category><![CDATA[nodeops]]></category>
            <dc:creator><![CDATA[Emmanuel Christopher]]></dc:creator>
            <pubDate>Tue, 20 Jan 2026 14:13:13 GMT</pubDate>
            <atom:updated>2026-01-20T14:14:05.824Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gbuqc62BI3bainlu" /></figure><p><em>There’s nothing worse than waking up to find your node lagging behind…..not because it crashed, but because you hit “upgrade” at the wrong time.</em></p><p>Most blockchain node outages don’t happen during sync.</p><p>They happen during upgrades.</p><p>Client releases drop, teams rush to upgrade, traffic breaks, data locks, or worse, corrupted state.</p><p>Here’s the boring but reliable way we handled node upgrades in production, repeatedly, with minimal downtime.</p><h4>Before we proceed</h4><p>My approach assumes: You are running <strong>more than one node instance</strong> behind a load balancer.</p><p>If you are running a single node in production, this is your sign to stop reading and fix that first(for real).</p><h3>The setup</h3><p>My setup looked like this:</p><ul><li>2–3 blockchain node instances</li><li>All behind an AWS Load Balancer</li><li>Traffic only goes to <strong>healthy</strong> nodes</li></ul><p>Each node has:</p><ul><li>Its own local data directory</li><li>Independent health checks</li></ul><h3>The real problem upgrades cause</h3><p>Upgrading a node is not always just:</p><blockquote><em>“</em>Download new binary and restart<em>”</em></blockquote><p>It might involve disk migrations, database rewrites (for example, LevelDB to PebbleDB on Geth), state compatibility checks, or unpredictable startup times.</p><p>In other words, an “upgrade” is often a controlled failure plus a recovery plan. The only question is whether you spread that failure across nodes, or concentrate it on all of them at once.</p><h3>The solution I did</h3><p>I have 2 scripts.</p><h4>First script</h4><p>Checks the client’s GitHub repo tags and compares them with the version currently running on the node.</p><p>If a newer release exists, it sends an event to my backend DB (I used Timestream, but any DB works) and triggers a Grafana alert that notifies me on Slack.</p><p>Once I receive the alert, I decide when to trigger the upgrade process.</p><h4>Second script</h4><p>This script carries out the upgrade process for each node behind the load balancer:</p><ol><li>Drain traffic from the node (via LB health check)</li><li>Stop the node process</li><li>Upgrade the client binary</li><li>Start the node</li><li>Wait until:</li></ol><ul><li>Node is fully synced</li><li>RPC is responsive</li><li>Health checks pass</li></ul><p>Only then…it moves to the next node</p><p>This script has two trigger modes.</p><p>The first is manual, based on a decision from the first script, because some upgrades introduce deprecated or new flags required for the node to start successfully. For example, when upgrading my Arbitrum Nitro node from <strong>v3.6.5</strong> to <a href="https://github.com/OffchainLabs/nitro/releases/tag/v3.6.6"><strong>v3.6.6</strong></a>, I had to include a new startup flag.</p><p>The second is automatic. If a node panics and remains unhealthy for more than 5–10 minutes, the script automatically upgrades the client and restarts the node. Around 95% of the time, this fixes the issue. As for the other 5%, well thats a story for another day.</p><h3>Why did I consider this approach?</h3><p>This approach</p><ul><li>Avoids full outages.</li><li>Avoids rushed rollbacks.</li><li>Avoids corrupted state.</li><li>Let&#39;s you observe new client behavior safely.</li><li>Gives you time to react if something looks off.</li></ul><h4>This matters <em>even more</em> during</h4><ul><li>Hard forks</li><li>Client-breaking releases</li><li>Performance-heavy upgrades</li><li>New pruning or DB formats</li></ul><blockquote>In conclusion, my setup isn’t clever. It’s just disciplined. If you run blockchain nodes in production and don’t have rolling upgrades, you’re one bad release away from downtime.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fefd601bd6e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Production‑Grade Monitoring for Starknet Pathfinder with Prometheus & Grafana]]></title>
            <link>https://medium.com/@krissemmy17/production-grade-monitoring-for-starknet-pathfinder-with-prometheus-grafana-83be6e6a73e4?source=rss-a99cbbeff7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/83be6e6a73e4</guid>
            <category><![CDATA[rpc]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[starknet]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[pathfinder]]></category>
            <dc:creator><![CDATA[Emmanuel Christopher]]></dc:creator>
            <pubDate>Mon, 12 Jan 2026 15:27:42 GMT</pubDate>
            <atom:updated>2026-01-12T15:27:42.154Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rIglcVR9V7IedRt5.png" /></figure><p>If you run a Starknet node and you are not monitoring it properly, you are flying blind.<br>Most guides stop at “node is running” and ignore the question that actually matters: <em>is this node healthy, synced, and reliably serving traffic?</em></p><p>This article fills that gap for the Pathfinder client by walking through:</p><ul><li>Installing Prometheus and Grafana</li><li>Wiring Pathfinder’s metrics into Prometheus</li><li>Adding Grafana as a frontend</li><li>Importing a ready‑made dashboard JSON that visualizes sync, RPC, and gateway health end‑to‑end</li></ul><p>My aim is simply to give you a dashboard where you can answer “is this node okay?” in one glance instead of grepping logs for 20 minutes.</p><h3>1. Important point to take note of</h3><p>Pathfinder’s own installation docs are already solid(kudos to the team), so my article would <strong>not</strong> repeat them. I assume you:​</p><ul><li>Already have a Pathfinder node running (e.g. on mainnet), installed via the official docs</li><li>Have enabled its Prometheus metrics endpoint (for example, on 127.0.0.1:9000)</li><li>Are comfortable running services with systemd or Docker.</li></ul><p>From here, you will:</p><ul><li>Stand up Prometheus to scrape Pathfinder metrics</li><li>Install Grafana</li><li>Add Prometheus as a data source</li><li>Import a dashboard JSON file that you can keep under version control and reuse across nodes​</li></ul><h3>2. Installing Prometheus</h3><p>Prometheus will scrape Pathfinder’s metrics endpoint and store time‑series data. You can install it either as a native package or via Docker.​</p><h3>Option A: Native install (Ubuntu‑style)</h3><p>On the same server as your node(or a dedicated monitoring instance, but for the purpose of this article i will keep it simple and have the monitoring in the same server), you would:</p><ul><li>Install <a href="https://prometheus.io/docs/introduction/first_steps/"><strong>Prometheus</strong></a> from your distribution’s repository</li><li>Ensure the service is enabled and starts on boot</li><li>Open only the ports you actually need (often just Grafana’s UI port, not Prometheus itself)​</li></ul><pre># Install Prometheus<br># Visits https://prometheus.io/download/ and gets the latest version<br>PROM_VERSION=$(curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep tag_name | cut -d &#39;&quot;&#39; -f 4 | cut -c 2-)<br>wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz<br>tar xvfz prometheus-*.tar.gz<br>rm prometheus-*.tar.gz<br><br>#Create a file for Prometheus<br>sudo tee /etc/systemd/system/prometheus.service &gt; /dev/null &lt;&lt;EOF<br>[Unit]<br>Description=Prometheus<br>Wants=network-online.target<br>After=network-online.target<br><br>[Service]<br>User=node<br>Type=simple<br>WorkingDirectory=$HOME/prometheus-${PROM_VERSION}.linux-amd64<br>ExecStart=$HOME/prometheus-${PROM_VERSION}.linux-amd64/prometheus \<br>  --config.file=prometheus.yml<br><br>[Install]<br>WantedBy=multi-user.target<br>EOF</pre><h3>Option B: Docker / Docker Compose</h3><p>If you already use containers, you can:</p><ul><li>Run Prometheus in a container</li><li>Mount a local prometheus.yml configuration</li><li>Mount a persistent volume for Prometheus data</li></ul><p>Either way, the important part for Pathfinder is the scrape_configs section.</p><h3>Minimal prometheus.yml for Pathfinder</h3><p>At a minimum, you want Prometheus to scrape your node and attach a job label that matches how the Grafana dashboard is parameterized, so run this on your CLI to update the prometheus.yml file:</p><pre>echo &#39;  - job_name: &quot;starknet&quot;<br>    static_configs:<br>      - targets: [&quot;127.0.0.1:9000&quot;]&#39; | sudo tee -a $HOME/prometheus-${PROM_VERSION}.linux-amd64/prometheus.yml &gt;/dev/null</pre><ul><li>job_name: &quot;starknet&quot; will be used later as template variables in Grafana, so you can switch between nodes from a dropdown instead of hardcoding addresses.</li></ul><p>Once Prometheus is running, visit its UI (typically http://&lt;host&gt;:9090) and search for a metric like current_block or pathfinder_build_info to confirm that Pathfinder is being scraped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3CAB34hg_bfQyhughetWbA.png" /></figure><h3>3. Installing Grafana and wiring Prometheus</h3><p>Grafana is the visualization layer. It queries Prometheus and renders the charts.</p><h3>3.1 Install Grafana</h3><p>You can:</p><ul><li>Install Grafana via your distribution’s package manager</li><li>Or run it in Docker, exposing port 3000 on a safe interface​</li></ul><pre># Install Grafana (ubuntu 22.04)<br>sudo mkdir -p /etc/apt/keyrings<br><br>curl -fsSL https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg<br><br>echo &quot;deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main&quot; \<br>  | sudo tee /etc/apt/sources.list.d/grafana.list &gt; /dev/null<br>sudo apt-get update<br>sudo apt-get install -y grafana<br><br># Start Grafana<br>sudo systemctl start grafana-server</pre><p>After installation:</p><ul><li>Start the Grafana service</li><li>Log in at http://&lt;host&gt;:3000 (default user/password is admin/admin, which you should immediately change)</li></ul><h3>3.2 Add Prometheus as a data source</h3><p>Inside Grafana:</p><ol><li>Go to <strong><em>Settings -&gt; Data sources -&gt; “Add data source”</em></strong>.</li><li>Choose Prometheus.</li><li>Set the URL to your Prometheus instance, for example:</li></ol><ul><li>http://localhost:9090 if Grafana and Prometheus run on the same host(if not then u add the IP address of server prometheus is hosted)</li><li>http://prometheus:9090 if they run in the same Docker network</li></ul><p>4. Click Save &amp; test to confirm Grafana can query Prometheus.​​</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0VFry9ksT7lt9BulhKXAhg.png" /></figure><blockquote>In the dashboard JSON, this data source is referenced via a variable ${ds} so you can easily point the same dashboard at different Prometheus instances without editing every panel.</blockquote><h3>4. Importing the Starknet Pathfinder dashboard</h3><p>To avoid clicking through Grafana’s UI for every panel, the dashboard is defined once as dashboard.json and stored in a <a href="https://github.com/krissemmy/starknet-node-monitoring/blob/main/grafana_dashboard.json"><strong>GitHub repo</strong></a>. You can then import that JSON into any Grafana instance.​​</p><h3>4.1 Import steps</h3><ol><li>Download <a href="https://github.com/krissemmy/starknet-node-monitoring/blob/main/grafana_dashboard.json"><strong><em>dashboard.json</em></strong></a> from the repo.</li><li>In Grafana, go to <strong><em>Dashboards -&gt; New -&gt; Import</em></strong>.</li><li>Upload/Paste the JSON file content, and then click on <strong><em>Load</em></strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZE6kN-_X-K-XGit9RtSdmg.png" /></figure><p>Grafana will create all panels and variables defined in the JSON, including:</p><ul><li>A datasource variable ${ds}</li><li>A job variable (e.g. &quot;starknet&quot;)</li><li>An instance variable (e.g. &quot;127.0.0.1:9000&quot;)</li></ul><p>This lets you switch between nodes or environments with dropdowns instead of editing queries.</p><h3>5. Dashboard overview: what the panels show</h3><p>The dashboard is organized into logical rows so you can move from “basic sanity” to “deep debugging” in a few seconds.</p><h3>5.1 Overview row</h3><h4>Pathfinder version (Stat panel)</h4><ul><li>Query:</li></ul><pre>pathfinder_build_info{job=&quot;$job&quot;, instance=&quot;$instance&quot;, network=&quot;mainnet&quot;}</pre><ul><li>Shows the current binary version for the selected node.</li><li>You can glance at this during upgrades to confirm the instance you are looking at actually runs the version you expect.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PnnzacQ1FvTqlCgcLWRgvg.png" /><figcaption>Pathfinder Version</figcaption></figure><h4>Uptime percentage (Gauge)</h4><ul><li>Query:</li></ul><pre>100 * avg_over_time(up{job=&quot;$job&quot;, instance=&quot;$instance&quot;}[$__range])</pre><ul><li>Shows how often Prometheus considered the node “up” over the selected time window.</li><li>This catches crash loops, flapping processes, or scrape issues; if the gauge is not close to 100% over a day, something is off.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gE_okTct2fKDgmPUWQ7zNw.png" /><figcaption>Uptime percentage</figcaption></figure><p>These two panels give you a quick “is it alive and which version is it?” snapshot.</p><h3>6. Sync progress: following your node vs the network</h3><p>The next row focuses on how well the node is keeping up with Starknet.</p><h4>Current block number</h4><ul><li>Query:</li></ul><pre>current_block{job=&quot;$job&quot;}</pre><ul><li>Time series showing how the node’s current block advances over time.</li><li>A flat line here during active network periods usually means the node is stuck.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OZLzlJJYFspDnBoJaWMirQ.png" /><figcaption>Current Block Number</figcaption></figure><h4>Blocks behind</h4><ul><li>Query:</li></ul><pre>highest_block{job=&quot;$job&quot;, instance=&quot;$instance&quot;} - current_block{job=&quot;$job&quot;, instance=&quot;$instance&quot;}</pre><ul><li>Computes how many blocks the node is behind the highest known block. When it&#39;s synced and healthy, it will be at 0(fluctuates to at most 1 or 2)</li><li>Color overrides highlight healthy vs unhealthy lag (for example: green when ≤5 blocks behind, red when ≥50). This gives you an at‑a‑glance “sync gap” metric.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEtARRywsErjaTB3afjFEw.png" /><figcaption>Blocks behind showing a starknet node that is still syncing</figcaption></figure><h4>Current vs latest block</h4><ul><li>Two series in one chart:</li></ul><pre>current_block{job=&quot;$job&quot;}</pre><pre>highest_block{job=&quot;$job&quot;, instance=&quot;$instance&quot;}</pre><ul><li>Visualizes how closely your node tracks the network tip.</li><li>When these lines diverge and stay apart, you know the node is lagging independently of network block production rate.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VvykoHwW8NdwnC5VVvM76Q.png" /><figcaption>Current block on node vs Latest block on the network</figcaption></figure><h4>Block age (seconds)</h4><ul><li>Query:</li></ul><pre>block_latency{job=&quot;$job&quot;, instance=&quot;$instance&quot;}</pre><ul><li>Represents how old the latest on‑chain block is (from the network’s perspective).</li></ul><p>Helps distinguish:</p><ul><li>Network is quiet (blocks are genuinely old) vs</li><li>Your node is slow (blocks are recent but your current block lags).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4XhR1q9JBA8XJee--wodrQ.png" /><figcaption>Block Age on the network</figcaption></figure><p>Together, these charts answer “Is this node caught up?” much better than a single block number.</p><h3>7. Block processing: download vs processing time</h3><p>Sometimes the node is behind not because of the network but because of limited compute, storage, or gateway performance. This row helps you see where the time goes.</p><h4>Block download vs processing time</h4><ul><li>Queries:</li></ul><pre>block_download{job=&quot;$job&quot;, instance=&quot;$instance&quot;}</pre><pre>block_processing{job=&quot;$job&quot;, instance=&quot;$instance&quot;}</pre><p>Plotted on the same chart but on two queries to compare:</p><ul><li>Time spent downloading a block</li><li>Time spent processing that block</li></ul><p>Typical patterns:</p><ul><li>Download dominates -&gt;likely gateway or network bottleneck, or too aggressive timeouts.</li><li>Processing dominates -&gt; likely CPU, memory, or disk constraints on the node itself.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NZWQze26PxPTqBeWSVmXdw.png" /><figcaption>Block download vs Processing time</figcaption></figure><p>When something feels slow, this chart is often the first place I take a look before resizing hardware or blaming the client.</p><h3>8. RPC and gateway metrics: traffic, errors, and latency</h3><p>Sync is necessary but not sufficient. A node can be fully synced and still be a terrible RPC server if methods are slow or failing. This section focuses on the JSON‑RPC and gateway layers.</p><h4>RPC requests per second (RPS)</h4><ul><li>Query:</li></ul><pre>sum by(method) (rate(rpc_method_calls_total{job=&quot;$job&quot;, instance=&quot;$instance&quot;}[1m]))</pre><ul><li>Shows per‑method request rates over time.</li></ul><p>Useful for:</p><ul><li>Understanding read vs write load</li><li>Seeing which methods are “hot”</li><li>Planning capacity around real usage</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eubm8JkEeYvRcf6sQXfuOg.png" /></figure><h4>RPC error rate (failed/sec)</h4><ul><li>Query:</li></ul><pre>sum by(method) (rate(rpc_method_calls_failed_total{job=&quot;$job&quot;, instance=&quot;$instance&quot;}[1m]))</pre><ul><li>Shows how many failed calls per second each method returns.</li><li>Spikes for a single method usually mean a breaking change, a misconfigured client, or upstream issues.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Klou4fEygI9YOlkwQwPBvQ.png" /><figcaption>RPC error rate showing per method basis</figcaption></figure><h4>Average RPC latency per method</h4><ul><li>Query (5‑minute window):</li></ul><pre>sum by (method)(<br>  rate(rpc_method_calls_duration_milliseconds_sum{job=&quot;$job&quot;, instance=&quot;$instance&quot;}[5m])<br>)<br>/<br>sum by (method)(<br>  rate(rpc_method_calls_duration_milliseconds_count{job=&quot;$job&quot;, instance=&quot;$instance&quot;}[5m])<br>)</pre><ul><li>Computes average latency per method.</li><li>Let&#39;s you catch regressions (a new release increasing latency) or sudden spikes tied to specific consumers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5AgTv-R6wYcvSHbLVXRbSA.png" /><figcaption>Average RPC latency per method</figcaption></figure><h4>Gateway RPS by method</h4><ul><li>Query:</li></ul><pre>sum by (method) (rate(gateway_requests_total{job=&quot;$job&quot;, instance=&quot;$instance&quot;}[1m]))</pre><ul><li>Shows how much upstream traffic your node is sending to Starknet’s gateway.</li><li>If this graph spikes at the same time as block download time or gateway failures, the bottleneck is likely upstream rather than on your machine.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S0EGHVKzsdjVCOUgOUzz9g.png" /><figcaption>Gateway RPS by method</figcaption></figure><h4><strong>Gateway failures per second by reason</strong></h4><ul><li>Query:</li></ul><pre>sum by(reason) (rate(gateway_requests_failed_total{job=&quot;$job&quot;, instance=&quot;$instance&quot;}[5m]))</pre><ul><li>Breaks failures down by reason (timeouts, rate limits, etc., depending on labels).</li><li>This is directly useful when you correlate with logs complaining about gateway timeouts or deserialization errors.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hPXYtOqUI0EFQmOkhQfEgA.png" /><figcaption>Gateway failures/sec by reason</figcaption></figure><p>This row effectively helps me answer questions like: “Are my users’ requests being served reliably?” and “Is the upstream gateway healthy?”</p><h3>9. Troubleshooting: timeouts and stuck sync</h3><p>While syncing the node, was it smooth sailing for me? Nope, I just happened to experience a Pathfinder bug in the v0.21.4 release(which was released a few days ago) that causes latency issues with Pathfinder syncing. I noticed the issue from my <strong><em>Gateway RPS by method</em></strong> chart, from there when i checked the node logs i saw the below messages:</p><pre>ERROR failed to deserialize message err=unexpected `error` field in subscription notification<br>INFO state_update_with_block: Request failed, retrying. Fetching the response or parts of it timed out. Try increasing request timeout by using the `--gateway.request-timeout` CLI option.</pre><p>Often, these mention specific feeder gateway URLs such as:</p><ul><li>.../feeder_gateway/get_state_update?...</li><li>.../feeder_gateway/get_signature?...</li><li>.../feeder_gateway/get_block?...</li></ul><p>In practice, this tells me one of a few things:</p><ul><li>The node is still far behind the tip and needs a lot of data from the gateway</li><li>Responses are slow or large enough to hit request timeouts</li></ul><h3>9.1 Increasing --gateway.request-timeout</h3><p>Pathfinder exposes a CLI flag --gateway.request-timeout to control how long it waits before treating a gateway request as failed. If you see frequent timeout logs and stalled progress, increasing this value can help:</p><ul><li>Start modestly (e.g. 60 seconds)</li><li>If timeouts persist under normal network conditions, you can raise it further (e.g. 300 seconds)</li></ul><p>There is a trade‑off:</p><ul><li>Too low -&gt; many retries, wasted work, and slow progress</li><li>Too high -&gt; stuck requests linger, and you wait a long time to see failures</li></ul><p>Use the dashboard to <em>guide</em> your tuning:</p><ul><li>When you see timeout logs:</li><li>Check Gateway failures/sec by reason for spikes in timeout‑like reasons.</li><li>Check Block download vs processing time; if download spikes while processing stays low, the bottleneck is upstream.</li><li>Check Blocks behind; if it flatlines while failures rise, progress has effectively halted.</li></ul><p>If the node remains stuck after increasing the timeout, doing a graceful shutdown and restart can help it reset any problematic state and resume sync more cleanly, especially when it is still many blocks behind.</p><blockquote>Okay I got carried away and made the article long😅. Time to wrap it up</blockquote><h3>10. Operational routines</h3><p>A dashboard is only useful if you actually look at it. Here is a simple routine you can adopt:</p><p>Daily (or per shift, 30 minutes):</p><ul><li>Check Blocks behind for each instance.</li><li>Check RPC error rate and gateway failures for unexpected spikes.</li></ul><p>During incidents:</p><ul><li>Use Current vs latest block and block age to see if the issue is your node or the network.</li><li>Use RPC latency per method to identify which calls are impacted.</li><li>Use Gateway RPS / failures to determine whether to adjust timeouts or investigate upstream issues.</li></ul><p>Because the dashboard is defined in JSON, you can keep it:</p><ul><li>In Git, alongside your infra code</li><li>Imported automatically by your provisioning (Terraform, Ansible, etc.)</li><li>Reused across environments (testnet, mainnet, multiple regions)</li></ul><h3>11. Conclusion</h3><p>Currently, Pathfinder’s documentation covers how to expose Prometheus metrics, but it does not ship a Grafana dashboard. The intention with this setup is to ensure new operators get production‑grade monitoring on day one instead of reinventing panels from scratch.</p><p>Feel free to adapt this dashboard to your own needs(by all means, please) by:</p><ul><li>Adding system metrics (CPU, memory, disk) from a node exporter.</li><li>Creating additional panels for alerts you care about.</li><li>Reusing the same structure (Overview -&gt; Sync -&gt; Processing -&gt; RPC/Gateway) for other chains and clients.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83be6e6a73e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to run a Tempo Testnet RPC Node on Ubuntu (with Monitoring)]]></title>
            <link>https://medium.com/@krissemmy17/how-to-run-a-tempo-testnet-rpc-node-on-ubuntu-with-monitoring-d6447fce197f?source=rss-a99cbbeff7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6447fce197f</guid>
            <category><![CDATA[rpc]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[blockchain-node]]></category>
            <category><![CDATA[nodeops]]></category>
            <dc:creator><![CDATA[Emmanuel Christopher]]></dc:creator>
            <pubDate>Wed, 24 Dec 2025 12:16:28 GMT</pubDate>
            <atom:updated>2025-12-24T12:16:28.539Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6WNVUl-adiMcrpyF" /></figure><p>Running your own RPC node is one of the fastest ways to actually understand how a blockchain network behaves. Not from docs. Not from abstractions. From watching blocks sync, metrics move, and services fail and recover.</p><p>In this article, I’ll walk through how I set up a <strong>Tempo testnet RPC node</strong> on Ubuntu, including:</p><ul><li>a dedicated, non-root runtime user</li><li>a systemd-managed Tempo node</li><li><a href="https://medium.com/u/19f6b7f36d21">Prometheus Monitoring</a> scraping node metrics</li><li>Grafana for visibility</li><li>and a lightweight status page to track sync progress and health</li></ul><p>This setup is intentionally simple. No Kubernetes. No Docker. Just a clean Linux server, systemd, and observability you can reason about.</p><h3>What you’ll end up with</h3><p>By the end of this guide, you’ll have:</p><ul><li>A running Tempo testnet RPC node</li><li>HTTP and WS JSON-RPC exposed on port 8545 and 8546 respectively</li><li>Metrics exposed on port 9000</li><li>Prometheus scraping Tempo metrics</li><li>Grafana dashboards for visibility</li><li>A small status page showing whether the node is live or still syncing</li></ul><p>Everything runs under a dedicated node user.</p><h3>System requirements</h3><ul><li>Ubuntu 24.04 or newer</li><li>amd64 architecture</li><li>For an extended list of system requirements, check out the <a href="https://docs.tempo.xyz/guide/node/system-requirements">official docs</a></li><li>For the purpose of this article, I used a <a href="https://portal.cherryservers.com/deployment?plan=832&amp;cycle=37&amp;hide_plans=1">Premium VDS 8</a> from <a href="https://medium.com/u/e5feb1337c27">Cherry Servers</a></li></ul><p>Open ports:</p><ul><li>30303 – P2P</li><li>8545 – RPC</li><li>9000 – Metrics</li><li>9090 – Prometheus</li><li>3000 – Grafana</li></ul><blockquote><em>If you’re on Ubuntu 22.04, be aware of glibc compatibility when running prebuilt binaries of v0.8.0 (Check out this </em><a href="https://github.com/tempoxyz/tempo/issues/1726"><em>issue</em></a><em> on the Tempo repo)</em></blockquote><h3>Create a dedicated node user</h3><p>Never run blockchain nodes as root.</p><p>Create a dedicated user and give it sudo access:</p><pre>sudo adduser node<br>sudo usermod -aG sudo node</pre><p>This gives you isolation, cleaner permissions, and fewer foot-guns later.</p><h3>Enable SSH access for the node user (temporary)</h3><p>For Bootstrap, I enabled password SSH. This is fine temporarily, but you should switch to SSH keys afterward.</p><pre>sudo tee -a /etc/ssh/sshd_config &gt; /dev/null &lt;&lt;EOF<br>PasswordAuthentication yes<br>KbdInteractiveAuthentication yes<br>UsePAM yes<br>EOF</pre><pre>sudo systemctl restart ssh</pre><p>Then log in as the new user:</p><pre>ssh node@&lt;SERVER_IP&gt;</pre><h3>Verify OS and glibc</h3><p>This is useful later when debugging binary compatibility issues.</p><pre>cat /etc/os-release<br>ldd --version | head -n 1</pre><h3>Installing base utilities</h3><pre>sudo apt-get update &amp;&amp; sudo apt-get upgrade -y<br>sudo apt-get install -y ccze build-essential pkg-config libssl-dev</pre><p>ccze is optional, but it makes journald logs much easier to read.</p><h3>Install Prometheus</h3><p>I’m using Prometheus directly from upstream releases.</p><p>First, fetch the latest version:</p><pre>PROM_VERSION=$(curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \<br>  | grep tag_name | cut -d &#39;&quot;&#39; -f 4 | cut -c 2-)</pre><pre>wget <a href="https://github.com/prometheus/prometheus/releases/download/v$%7BPROM_VERSION%7D/prometheus-$%7BPROM_VERSION%7D.linux-amd64.tar.gz">https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz</a><br>tar xvfz prometheus-*.tar.gz<br>rm prometheus-*.tar.gz</pre><h3>Run Prometheus via systemd</h3><p>Create a systemd service:</p><pre>sudo tee /etc/systemd/system/prometheus.service &gt; /dev/null &lt;&lt;EOF<br>[Unit]<br>Description=Prometheus<br>Wants=network-online.target<br>After=network-online.target<br><br>[Service]<br>User=node<br>Type=simple<br>WorkingDirectory=/home/node/prometheus-${PROM_VERSION}.linux-amd64<br>ExecStart=/home/node/prometheus-${PROM_VERSION}.linux-amd64/prometheus \<br>  --config.file=prometheus.yml<br><br>[Install]<br>WantedBy=multi-user.target<br>EOF<br></pre><h3>Configure Prometheus to scrape Tempo</h3><p>Append this to prometheus.yml:</p><pre>echo &#39;  - job_name: &quot;tempo&quot;<br>    static_configs:<br>      - targets: [&quot;127.0.0.1:9000&quot;]&#39; \<br>  | sudo tee -a /home/node/prometheus-${PROM_VERSION}.linux-amd64/prometheus.yml &gt;/dev/null</pre><p>Start Prometheus:</p><pre>sudo systemctl daemon-reload<br>sudo systemctl enable prometheus<br>sudo systemctl start prometheus</pre><p>Prometheus UI (optional):</p><pre>http://&lt;SERVER_IP&gt;:9090</pre><h3>Install Grafana</h3><p>Add Grafana’s repository using the modern keyring approach:</p><pre>sudo mkdir -p /etc/apt/keyrings<br>curl -fsSL https://apt.grafana.com/gpg.key \<br>  | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg<br><br>echo &quot;deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main&quot; \<br>  | sudo tee /etc/apt/sources.list.d/grafana.list &gt; /dev/null</pre><p>Install and start Grafana:</p><pre>sudo apt-get update<br>sudo apt-get install -y grafana<br>sudo systemctl enable --now grafana-server</pre><p>Grafana UI:</p><pre>http://&lt;SERVER_IP&gt;:3000</pre><p>Default login is admin / admin.</p><blockquote><em>For monitoring, you can import an existing Tempo dashboard or use the </em><a href="https://github.com/paradigmxyz/reth/blob/main/etc/grafana/dashboards/overview.json"><strong><em>dashboard</em></strong></a><em> from the rethteam(since tempo was built using reth). I link dashboards separately rather than embedding screenshots.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*q4I1NubGTi9eYjuL" /></figure><p><em>Grafana Dashboard of Tempo node syncing to the latest block</em></p><p>It&#39;s normal to have a low count of peers. As the public testnet was only released in early December 2025)</p><h3>Install Tempo</h3><p>Install Tempo using the official installer:</p><pre>curl -L https://tempo.xyz/install | bash<br>source ~/.bashrc<br>tempo --version<br><br># Create a data directory<br>mkdir -p ~/tempo-data</pre><h3>Download Snapshot data</h3><p>If you want your node to get synced to the latest block quickly, then Irecommend you download the tempo testnet snapshot data before running the node. Below is the command to download the snapshot data;</p><pre>tempo download - datadir tempo-data/</pre><p>Once it gets downloaded completely, it will automatically extract the downloaded snapshot database, and you can then proceed to the next step of running your node</p><h3>Run the Tempo node (manual test)</h3><p>Before systemd, always test manually.</p><pre>tempo node -vvv \<br>  --follow \<br>  --datadir ~/tempo-data \<br>  --port 30303 \<br>  --discovery.addr 0.0.0.0 \<br>  --discovery.port 30303 \<br>  --http \<br>  --http.addr 0.0.0.0 \<br>  --http.port 8545 \<br>  --http.api eth,net,web3,txpool,trace \<br>  --ws \<br>  --ws.addr 0.0.0.0 \<br>  --ws.port 8546 \<br>  --ws.api eth,net,web3,txpool,trace \<br>  --metrics 9000</pre><p>Let it run for a bit and confirm it starts syncing.</p><h3>Run Tempo as a systemd service</h3><p>Once you’re satisfied, create a systemd unit:</p><pre>sudo tee /etc/systemd/system/tempo.service &gt; /dev/null &lt;&lt;EOF<br>[Unit]<br>Description=Tempo RPC Node<br>After=network.target<br>Wants=network.target<br><br>[Service]<br>Type=simple<br>User=node<br>Group=node<br>Environment=RUST_LOG=info<br>WorkingDirectory=/home/node<br>ExecStart=/home/node/.tempo/bin/tempo node \<br>  --datadir /home/node/tempo-data \<br>  --follow \<br>  --port 30303 \<br>  --discovery.addr 0.0.0.0 \<br>  --discovery.port 30303 \<br>  --http \<br>  --http.addr 0.0.0.0 \<br>  --http.port 8545 \<br>  --http.api eth,net,web3,txpool,trace \<br>  --ws \<br>  --ws.addr 0.0.0.0 \<br>  --ws.port 8546 \<br>  --ws.api eth,net,web3,txpool,trace \<br>  --metrics 9000 \<br>Restart=always<br>RestartSec=10<br>LimitNOFILE=infinity<br><br>[Install]<br>WantedBy=multi-user.target<br>EOF</pre><p>Enable and start:</p><pre>sudo systemctl daemon-reload<br>sudo systemctl enable tempo<br>sudo systemctl start tempo</pre><p>Follow logs:</p><pre>sudo journalctl -fu tempo | ccze</pre><h3>Verify RPC and sync status</h3><p>Check the current block:</p><pre>curl -H &#39;Content-Type: application/json&#39; \<br>  --data &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:1,&quot;method&quot;:&quot;eth_blockNumber&quot;,&quot;params&quot;:[]}&#39; \<br>  http://127.0.0.1:8545</pre><p>Check sync state:</p><pre>curl -H &#39;Content-Type: application/json&#39; \<br>  --data &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:1,&quot;method&quot;:&quot;eth_syncing&quot;,&quot;params&quot;:[]}&#39; \<br>  http://127.0.0.1:8545</pre><p>Attached is an image of what your Grafana Dashboard will look like once your node is synced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nlifwW10GMhSOUccnmSAZg.png" /></figure><p><em>Synced Tempo Testnet RPC Node</em></p><h3>Add a simple status page</h3><p>To make node health obvious at a glance, I added a small Rust-based status page that:</p><ul><li>checks if the node is reachable</li><li>compares local block height to a public Tempo RPC</li><li>shows whether the node is syncing or live</li></ul><p>It’s optional, but very useful in practice.</p><pre># Install Rust<br>curl --proto &#39;=https&#39; --tlsv1.2 -sSf https://sh.rustup.rs | sh<br>source $HOME/.cargo/env<br><br># clone the repo<br>git clone https://github.com/krissemmy/tempo-status.git<br><br># Create a new tmux session<br>tmux<br><br># Run this in the tmux session<br>export NODE_RPC=&quot;http://127.0.0.1:8545&quot;<br>export REF_RPC=&quot;https://rpc.testnet.tempo.xyz&quot;<br>export MAX_LAG=&quot;3&quot;<br>export BIND=&quot;0.0.0.0:8080&quot;<br><br># Run the status page<br>cargo run --release<br><br><br>It&#39;s optional, but very useful in practice.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M7fipwahSox_F2tfuVVpGw.png" /></figure><h3>Final notes</h3><ul><li>Disable password SSH after setup</li><li>Don’t expose Prometheus or Grafana publicly without auth</li><li>This is a testnet. Expect resets and experimentation</li></ul><h3>Closing</h3><p>Running your own node, even on testnet, changes how you think about infrastructure. Metrics stop being abstract. Sync issues stop being theoretical.</p><p>If you’re experimenting with Tempo nodes, monitoring setups, or node operations in general, feel free to reach out</p><p>#RPC #Blockchain #NodeOps #DevOps #Web3 #Infrastructure #Monitoring #Tempo</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6447fce197f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hosting Fizzy on Coolify (A Practical, Reproducible Guide)]]></title>
            <link>https://medium.com/@krissemmy17/hosting-fizzy-on-coolify-a-practical-reproducible-guide-20c21c8fdd34?source=rss-a99cbbeff7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/20c21c8fdd34</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[fizzy-tool]]></category>
            <category><![CDATA[resend]]></category>
            <category><![CDATA[coolify]]></category>
            <category><![CDATA[hetzner]]></category>
            <dc:creator><![CDATA[Emmanuel Christopher]]></dc:creator>
            <pubDate>Mon, 22 Dec 2025 16:01:22 GMT</pubDate>
            <atom:updated>2025-12-22T16:01:22.053Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ev1CZzQol2xkVxGf6jdaBw.png" /><figcaption>Fizzy Sign In Page</figcaption></figure><p>When I recently consulted on deploying Fizzy, I wanted a setup that was:</p><ul><li>Private</li><li>Simple</li><li>Self-hosted</li><li>Easy to reason about</li></ul><p>This article walks through how Fizzy can be hosted on Coolify with SQLite, Resend, and Caddy-based authentication…without unnecessary complexity.</p><h3>Why Fizzy + Coolify?</h3><p>Fizzy is intentionally lightweight.<br>Coolify complements that by removing most of the operational burden.</p><p>Together, they allow you to deploy an internal team tool with:</p><ul><li>One application</li><li>One server</li><li>Minimal moving parts</li></ul><h3>Final Architecture</h3><ul><li><strong>App</strong>: <a href="https://github.com/basecamp/fizzy/pkgs/container/fizzy">Fizzy (official container)</a></li><li><strong>Platform</strong>: <a href="https://coolify.io/">Coolify</a></li><li><strong>Database</strong>: SQLite (persistent volume)</li><li><strong>Email</strong>: <a href="https://resend.com/">Resend</a> (SMTP)</li><li><strong>Auth</strong>: HTTP <a href="https://coolify.io/docs/knowledge-base/proxy/caddy/basic-auth">Basic Auth via Caddy</a></li><li><strong>Domain</strong>: Custom domain with HTTPS</li></ul><p>No Kubernetes.<br>No managed cloud services required.</p><h3>Database Choice: SQLite (On Purpose)</h3><p>SQLite is often underestimated.</p><p>For a single-instance internal app:</p><ul><li>It’s fast</li><li>It’s durable</li><li>It’s easy to back up</li><li>It requires zero maintenance</li></ul><p>Coolify persists the database file via Docker volumes, so data survives restarts and deployments.</p><p>Unless you need horizontal scaling, SQLite is a solid choice.</p><h3>Email Delivery with Resend</h3><p>Fizzy relies on email for authentication, so email deliverability matters.</p><p><strong>In Resend:</strong></p><ul><li>Add your domain</li><li>Follow DNS instructions</li></ul><p><strong>Example if using Route53:</strong></p><ul><li>Add SPF and DKIM TXT records</li><li>Wait for verification</li></ul><p>This step is mandatory.</p><h4><strong>What you need from Resend</strong></h4><p>You don’t need to understand Resend’s internals. You just need:</p><ul><li>SMTP host</li><li>SMTP port</li><li>Username</li><li>Password</li><li>A verified sending domain</li></ul><h3>Environment Variables</h3><pre>MAILER_FROM_ADDRESS=no-reply@yourdomain.com<br>SMTP_ADDRESS=smtp.resend.com<br>SMTP_PORT=587<br>SMTP_USERNAME=&lt;smtp username&gt;<br>SMTP_PASSWORD=&lt;smtp password&gt;<br>SMTP_AUTHENTICATION=plain<br>SMTP_ENABLE_STARTTLS_AUTO=true</pre><p>Once set in Coolify and redeployed, email delivery works immediately.</p><h3>Step 7: Web Push Notifications (Optional)</h3><p>Fizzy supports browser notifications via Web Push.</p><p>Generate a VAPID key pair once:</p><pre>vapid_key = WebPush.generate_key<br>puts vapid_key.private_key<br>puts vapid_key.public_key</pre><p>Then add to Coolify:</p><pre>VAPID_PRIVATE_KEY=&lt;private key&gt;<br>VAPID_PUBLIC_KEY=&lt;public key&gt;</pre><p>The private key stays server-side; the public key is shared with browsers.</p><h3>Why This Setup Works Well</h3><ul><li>Minimal infrastructure</li><li>Clear operational boundaries</li><li>Easy to back up</li><li>Easy to explain to non-DevOps teams</li><li>Simple to migrate later if requirements change</li></ul><p>Most importantly, it avoids premature complexity.</p><h3>When Would I Change This?</h3><p>I’d revisit parts of this setup if:</p><ul><li>Multiple app replicas were required</li><li>Write volume increased significantly</li><li>Deeper access control or SSO was needed</li></ul><p>Until then, this setup is more than sufficient.</p><h3>Final Thoughts</h3><p>Not every production deployment needs a complex architecture.</p><p>For internal tools, the best setup is often:</p><ul><li>Boring</li><li>Predictable</li><li>Easy to operate</li></ul><p>Coolify, Fizzy, SQLite, and Resend make that kind of deployment achievable without unnecessary friction.</p><p>#SelfHosting #Coolify #Rails #InternalTools #DevOps</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20c21c8fdd34" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>