Back in the day it was nice, apt get update && apt get upgrade and you were done.

But today every tool/service has it’s own way to being installed and updated:

  • docker:latest
  • docker:v1.2.3
  • custom script
  • git checkout v1.2.3
  • same but with custom migration commands afterwards
  • custom commands change from release to release
  • expect to do update as a specific user
  • update nginx config
  • update own default config and service has dependencies on the config changes
  • expect new versions of tools
  • etc.

I selfhost around 20 services like PieFed, Mastodon, PeerTube, Paperless-ngx, Immich, open-webui, Grafana, etc. And all of them have some dependencies which need to be updated too.

And nowadays you can’t really keep running on an older version especially when it’s internet facing.

So anyway, what are your strategies how to keep sanity while keeping all your self hosted services up to date?

  • kossa@feddit.orgEnglish
    1·
    4 days ago

    Ansible. Basically if I need to upgrade something for the first time, I write or extend an Ansible script an run those periodically.

  • eodur@piefed.socialEnglish
    1·
    4 days ago

    Renovate couple with FluxCD if you’re in k8s land, or noco-cd if you’re on docker. GitOps is the way.

  • Richard Wonka@slrpnk.netEnglish
    1·
    5 days ago

    Proxmox helper scripts - at least the ones i use - come with a tag updateable. Those tagged have a command update that runs everything necessary on containers, VMs whatever.

    Makes life simple, mostly.

    The only manual interaction I’ve had was upgrading some VMs Debian from 12 to 13.

  • ThunderComplex@lemmy.todayEnglish
    4·
    7 days ago

    Since all my services are dockerized I just pull new images sporadically. But I think I should invest some time into finding automatic update reminders, especially when I have to hear about critical security updates from some random person on mastodon.

  • uenticx@lemmy.worldEnglish
    1·
    5 days ago

    Snapshots and for i in $hosts;do ssh -tt "sudo apt update -y && sudo apt upgrade -y";done

    For docker/k8s: argocd, helm, etc.

    • tuhriel@discuss.tchncs.deEnglish
      21·
      6 days ago

      That’s the theory…I usually have to first replace some packages that have been removed / renamed then homemanager is acting up because some stuff now is called differently. And every time it just shows me the one error that failed…

  • ken@discuss.tchncs.deEnglish
    1·
    6 days ago

    A dedicated Forgejo instance f.example.com.

    For a small set of trusted “base” images (e.g. docker.io/alpine and docker.io/debian): A Forgejo Action on separate small runner, scheduled on cron to sync images to f.example.com/dockerio/ using skopeo copy.

    Then all other runners have their docker/podman configuration changed to use that internal forgejo container registry instead of docker.io.

    Other images are built from source in the Forgejo Actions CI. Not everything needs to be (or even should) be fully automated right off. You can keep some workflows manual while starting out and then increase automation as you tighten up your setup and get more confident in it. Follow the usual best practices around security and keep permissions scoped, giving them out only as needed.

    Git repos are mirrored as Forgejo repo mirrors, forked if relevant, then built with Forgejo Actions and published to f.example.com/whatever/. Rarely but sometimes is it worth spending time on reusing existing Github Workflows from upstreams. More often I find it easier to just reuse my own workflows.

    This way, runners can be kept fully offline and built by only accessing internal resources:

    • apt/apk repo mirror or proxy
    • synced base container images
    • synced git sources

    Same idea for npm or pypi packages etc.

    Set up renovate1 and iterate on its configuration to reduce insanity. Look in forgejo and codeberg infra repos for examples of how to automate rebasing of forked repo onto mirrors.

    I would previously achieve the same thing by wiring together more targeted services and that’s still viable but Forgejo makes it easy if you want it all in one box. Just add TLS.

    1: Or anyone have anything better that’s straightforward to integrate? I’m not a huge fan of all the npm modules it pulls in or its github-centric perspective. Giving the same treatment to renovate itself here was a little bit more effort and digging than I think should really be necessary.

  • FlowerFan@piefed.blahaj.zoneEnglish
    2·
    7 days ago

    Arcane docker server checks for updates, notifies me when they’re available

    for security relevant stuff I just get notifications of new github releases

  • quick_snail@feddit.nlEnglish
    1·
    7 days ago

    Unattended upgrades 11 months out of the year.

    Very attended apt upgrades 2 weeks out of the year.

  • corsicanguppy@lemmy.caEnglish
    22·
    7 days ago
    1. Avoid anything with bad supply chains that fail iso27002
    2. Yum via cron
    3. Huh. That’s all of it.
  • mlfh@lm.mlfh.orgEnglish
    28·
    8 days ago

    Everything I run, I deploy and manage with ansible.

    When I’m building out the role/playbook for a new service, I make sure to build in any special upgrade tasks it might have and tag them. When it’s time to run infrastructure-wide updates, I can run my single upgrade playbook and pull in the upgrade tasks for everything everywhere - new packages, container images, git releases, and all the service restart steps to load them.

    It’s more work at the beginning to set the role/playbook up properly, but it makes maintaining everything so much nicer (which I think is vital to keep it all fun and manageable).

    • Jeena@piefed.jeena.netOPEnglish
      7·
      8 days ago

      Yeah, For some reason I didn’t think of ansible even though I use it at work regularly. Thanks for pointing it out!

      • SayCyberOnceMore@feddit.ukEnglish
        5·
        7 days ago

        Just a word of caution…

        I try to upgrade 1 (of a similar group) manually first to check it’a not foobarred after the update, then crack on with the rest. Testing a restore is 1 thing, but restoring the whole system…?

    • BlackEco@lemmy.blackeco.comEnglish
      2·
      8 days ago

      I guess auto merge isn’t enabled, since there’s no way to check if an update doesn’t break your deployment beforehand, am I right?

        • BlackEco@lemmy.blackeco.comEnglish
          2·
          7 days ago

          Yes, but usually when you use automerge you should have set up a CI to make sure new versions don’t break your software or deployment. How are you supposed to do that in a self-hosting environment?

          • tofu@lemmy.nocturnal.gardenEnglish
            1·
            7 days ago

            Ideally, you have at least two systems, test updates in the dev system and only then allow it in prod. So no auto merge in prod in this case or somehow have it check if dev worked.

            Seeing which services are usually fine to update without intervening and tuning your renovate config to it should be sufficient for homelab imho.

            Given that most people are running :latest and just yolo the updates with watchtower or not automated at all, some granular control with renovate is already a big improvement.