All webhooks will automatically be retried on failed delivery with exponential backoff. We hope this will increase the success rate of the requests reaching their destinations and offer higher reliability to your services. However, this does not guarantee the delivery of the webhook requests.

You can read more about Expo EAS webhooks in our docs. If you have unexpected problems with your webhooks, please get in touch with us.

Adding retries to EAS Webhooks was originally published in Exposition on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’m now on my fourth Kubernetes release team and the conference is more managable. Virtual or in-person, it’s as hectic as ever with up to 20 talks happening at the same time (I counted) and everyone leaves with their own version of the conference, here is mine.

Glossary
CNCF = "Cloud Native Computing Foundation"
SIG = significant interest group, a Kubernetes subgroup
TAG = technical advisory group, a CNCF subgroup

New Release Process Reflections (Contributor Summit)
The Contributor Summit is my favorite event as I get to meet with folks I’ve been working with for the past year. I hosted a round-table discussion in the Contributor Summit discussing an initiative I’m part of, the migration of enhancements collection from sheets to Github project board. We’ve received good feedback from the community and are working to maintain this traction.

Prow Github Actions (Contributor Summit)
Prow is a job-based service cluster project. A popular feature from prow is powering Github tasks through/foo commands. Robert is working on a proof-of-concept for a Github Actions version of prow, aptly named prow Github Actions, that would make it faster for other projects to adopt this feature in their repo. The project is in its infancy (don’t mind the repo name) and is looking for help.

ContribFest (in-toto)
This is the first time Kubecon introduced ContribFest and I thoroughly enjoyed it. ContribFest is a sub-event throughout the week of Kubecon where CNCF projects would host 1.5 hours contribution sessions. I went to in-toto sessions and submitted two documentations pull requests, one of which improves the security policy. However, my favourite part of it was to say hi to the core maintainers of the project and received guidance on how to start. Being reminded that they are lovely humans who are ready to help made it a lot less intimidating.

Security
Most of the talks I attended was security related, on top of attending the first ever sigstorecon (their swag is very soft.) Here’s a few quick recap:

• Security TAG: checkout the Cloud Native security whitepaper v2 and the TAG’s roadmap.
• DEFCON CoC lawsuit: not security related at all, but I’ll chuck it in here since it’s about DEFCON. A legal take on managing communities with the case study of the DEFCON defamation lawsuit.
• SIG-security deep dive: an official CVE feed is live! Cluster-api was the first subproject security assessment and vSphere CSI driver is next.
• SIG-auth deep dive: KMSv2 is in alpha as of the 1.25 release. They are looking for help with test coverage and cleanups for stable features.
• Fuzzing integration: an introduction into coverage-guided fuzzing currently undergone by many CNCF projects.

No One Is Saving Us But Us
My personal favorite. A serious and hearty conversation about the sustainability and intentionality of open-source, or more specifically, open-source maintainers. Paris and Tabitha offered parallels to running a mutual aid project (and a book recs.) We are the community, we have to take care of each other. This talk reminded me of one that Savitha and Divya did about burn-out in open-source at Kubecon EU 2022 (I’m a serial skipper with Netflix shows. I watched this whole thing end to end.) If you are interested in the anthropology of open-source, I also recommend Nadia Eghbal’s whitepaper “Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure”.

I owe it to the CNCF’s Dan Kohn Scholarship to be able to attend this Kubecon, unite with my fellow contributors, and meet new folks.

That’s a wrap! Talks recordings will be up sometimes in late November. If you found this helpful, I’d appreciate a like. You can find me on Twitter @gracenng.

Riskfuel trains neural network models on datasets consisting of hundreds of millions of data points. Running at such a scale presents its own set of unique challenges. We have more and more pods which need to access tokens for storage, metrics, you name it. Previously, we already have our secrets in 1password but to use them in deployments, we had to manually include them as plaintext in our YAMLs. This slowed our developers’ efficiency and posed a potential security flaw.

In April 2021, 1password rolled out its Secret Automation workflow to address its CLI limited capability in distributed systems. This workflow includes plugins into backend systems like Hashicorp Vault, Kubernetes operator, Ansible collection and more. This allows Riskfuel to expand our existing usage of 1password and utilize it as a single source of truth for our Hashicorp Vault secrets. We are now able to automatically inject secrets into our deployments with ease.

There is documentation for this plugin with a manual Vault server workflow. This blog will be focusing a holistic deployment of the 1password-Vault plugin in Kubernetes with Helm.

For our scope, I’m going to assume the following:

• You have a 1password business account
• You have a Kubernetes cluster ready
• You are familiar with Vault, Kubernetes and Helm CLI

Getting Started

Step 1. Obtain 1password-credentials.json and access token

Each team plan will get 3 free credits for Secret Automation with a subscription for more.

Only an owner or admin of the 1password business account can set up Secret Automation and obtain the configuration json and access token. Follow the on screen instruction here to do so.

Step 2. Deploy Connect server

If you have used Helm before, this step is straightforward; use the credentials file you just obtained.

helm repo add 1password https://1password.github.io/connect-helm-charts/ 

helm install connect 1password/connect --set-file connect.credentials=1password-credentials.json

Step 3. Configure plugin in Vault’s Helm chart

We want to bake in this plugin, so let’s modify Vault’s Helm chart. We’re using /usr/local/libexec/vault in this example but this would be the path to your plugin binaries.

• If you are using vault-helm default chart, in your values.yaml under standalone or ha depending on your mode, under config, add:

plugin_directory = “/usr/local/libexec/vault”

• If you have a slimmed down vault Helm chart like we do, you would want to add this to the server-config-configmap.yaml under extraconfig-from-values.hcl

Step 4. Add plugin installation initContainer

We will install the plugin binaries and designate a file to be the config file later, via an initContainer . Don’t forget to modify the release version to match with your system and create a volume match the volumeMount here. Similar from above:

• If you are using vault-helm default chart, in the extraInitContainers in the values.yaml file, add:

• If you are using your own Helm chart, add this under initContainer in server-statefulset.yaml.

Step 5. Configure plugin inside Vault pod

Once the changes above are made, deploy the chart and exec into the vault-0 pod. We will then configure Vault for sidecar injection with k8s auth, register and enable the plugin.

Step 6. Insert access tokens

Staying inside the vault-0 pod, we now need to configure the plugin with the access token from step 1 to be able to make API request to the connect server. Let’s add to the /usr/local/libexec/vault/op-connect-config.json file we created in the initContainer the following:

{
"op_connect_host": "http://<IP_address_of_Connect_server_step_2>:8080",
"op_connect_token": "<access_token_from_step_1>"
}

Be mindful of the http and port number here, it is important. Then write this file to Vault.

vault write op/config @op-connect-config.json

Step 7. Test Connection

Now that the plugin is set up, we should be able to send and retrieve data from 1password.

• Get a list of Vault your token has access to: vault list op/vaults
• List items in a 1password vault: vault list op/vaults/<1PW_vault_name>/items
• Read item: vault read op/vaults/<1PW_vault_name>/items/<item_name>

Step 8. Deployment Usage

Before we can inject these secrets into our deployment, please ensure that your Vault policy is tied to your 1password path (i.e op/vaults/<1PW_vault_name>/items/*)and Vault role is configured with the proper service account and namespace.

In your deployment.yaml , under spec, add the annotation:

spec:
  ...
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: '<Vault_role>'
        vault.hashicorp.com/agent-inject-secret-config: 'op/vaults/<1PW_vault_name>/items/<item_name>'

Step 9. Profit

If you made it this far, congratulations! Once the deployment with annotation is running, you can exec into the pod and go to /vault/secret/config to see the injected contents. You can also leverage the Consul template annotation to have your secrets as environment variables.

Voila!

Conclusion

While the product is in its infancy, it boasts a powerful pipeline to connect 1password’s reliability with Hashicorp Vault’s infrastructure integration.

If you are interested in furthering your secret workflow, I also recommend checking out IBM’s argocd-vault-plugin. This allows us to streamline Kubernetes secrets by setting desired secrets states and have ArgoCD manage them, without revealing the value of these secrets in Github.

A small plug, Riskfuel is innovating the capital markets industry using cutting-edge machine learning and hardware. If you like solving problems like this, check out our careers page.

That’s it! Thanks for reading :)

Reference

• Deploy a 1Password Connect server on Kubernetes
• GitHub - 1Password/vault-plugin-secrets-onepassword: Hashicorp Vault plugin that allows for the retrieval, creation, and deletion of items stored in a 1Password vault accessed by use of the 1Password Connect.
• Injecting Secrets into Kubernetes Pods via Vault Agent Containers | Vault - HashiCorp Learn