Everything I run, I deploy and manage with ansible.
When I’m building out the role/playbook for a new service, I make sure to build in any special upgrade tasks it might have and tag them. When it’s time to run infrastructure-wide updates, I can run my single upgrade playbook and pull in the upgrade tasks for everything everywhere - new packages, container images, git releases, and all the service restart steps to load them.
It’s more work at the beginning to set the role/playbook up properly, but it makes maintaining everything so much nicer (which I think is vital to keep it all fun and manageable).
A partial solution to this evil-maid attack vector is Heads firmware (a replacement for the bios/uefi itself), which lets you sign the contents of your unencrypted boot partition using a gpg key on a hardware token, and verify the integrity of the firmware itself using a totp/hotp key stored in the tpm.
All the benefits of secure boot, but you get to control the signing keys yourself instead of relying on a vendor. It’s great stuff.