<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mike Arpaia on Medium]]></title>
        <description><![CDATA[Stories by Mike Arpaia on Medium]]></description>
        <link>https://medium.com/@marpaia?source=rss-d12efd302743------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*L-hJf2jM8brNslV-WPxx0w.jpeg</url>
            <title>Stories by Mike Arpaia on Medium</title>
            <link>https://medium.com/@marpaia?source=rss-d12efd302743------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 19:09:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marpaia/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using a Kubernetes Operator to Manage Tenancy in a B2B SaaS App]]></title>
            <link>https://medium.com/kolide/using-a-kubernetes-operator-to-manage-tenancy-in-a-b2b-saas-app-250f1c9416ce?source=rss-d12efd302743------2</link>
            <guid isPermaLink="false">https://medium.com/p/250f1c9416ce</guid>
            <category><![CDATA[b2b]]></category>
            <category><![CDATA[osquery]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Mike Arpaia]]></dc:creator>
            <pubDate>Tue, 15 May 2018 18:54:28 GMT</pubDate>
            <atom:updated>2020-09-28T18:53:42.159Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uJKOlK1CpcEb1VOSt8NSFw.png" /></figure><p><strong>Please Note: This article is referring to a former iteration of our paid offering.</strong></p><p>At Kolide, we recently launched our <strong>Kolide Cloud</strong> product, which is a Software-as-a-Service application that aims to help macOS administrators leverage the power of <a href="https://osquery.io/">osquery</a> to understand the security and integrity of the devices which they’re responsible for managing.</p><p>Now that Kolide Cloud is public, I’m excited share some details on how we build and operate Kolide Cloud. In this article, I want to focus on our multi-tenant deployment model and how we use a Kubernetes Operator to deploy and manage many single-tenant instances of our application. Most Operators reason about the lifecycle of a single instance of a service or application, but at Kolide, we’ve written an Operator to manage hundreds of instances of an application which is itself composed of several component services.</p><p>Once we’ve discussed the Operator and how Kolide uses it to deploy and manage software, we’ll briefly discuss how it’s built and some things that we’ve learned programming against the Kubernetes API in Go.</p><h3>Application Tenancy</h3><p>Companies that create products for other companies or teams often have to reason about how to deal with the tenancy of each team. The two ends of the spectrum are:</p><ul><li>Deploy one monolithic application that handles multi-tenant data isolation via application logic</li><li>Deploy and proxy to many instances of smaller, more isolated single-tenant applications</li></ul><p>When faced with these two options, most companies choose to build the multi-tenant monolith. While endeavoring to find Product-Market Fit, we rapid-prototyped a monolithic Rails application which used the <a href="https://github.com/influitive/apartment">Apartment</a> Rails Gem to isolate customer-specific data in PostgreSQL <a href="https://www.postgresql.org/docs/8.2/static/ddl-schemas.html">schemas</a>.</p><p>As we on-boarded more and more beta customers, however, we reasoned about the following two facts that are somewhat unique to Kolide:</p><ul><li>Kolide builds operating system security and analytics products, so the data we store for our customers is rather sensitive</li><li>An instance of our app may be used by 2–10 humans, but those humans may enroll thousands of devices which are effectively chatty API users</li></ul><p>Because of this, product developers were spending an unfortunate amount of time reasoning about scalability and data isolation when they should have been able to focus more fully on creating features that add value for customers. As a result, we decided to remove tenancy from the Rails application completely and start deploying and independently scaling single-tenant instances of the app on Kubernetes.</p><h3>Kubernetes Introduction</h3><p><a href="https://kubernetes.io/">Kubernetes</a> is a container-orchestration system that was created at Google as a open-source analog of their internal workload orchestration tool called <a href="https://ai.google/research/pubs/pub43438">Borg</a>. If you buy-in to packaging and distributing software in containers, then tools like Kubernetes, <a href="http://mesos.apache.org/">Mesos</a>, <a href="https://docs.docker.com/engine/swarm/">Docker Swarm</a>, etc. are there to help you articulate how containerized workloads should be launched and managed, how these workloads communicate with each other, how deploys should work, etc.</p><h4>Kubernetes Operators</h4><p>The Operator pattern was <a href="https://coreos.com/blog/introducing-operators.html">introduced by CoreOS in 2016</a> to describe a way to embed operational knowledge into software. In the article, <a href="https://twitter.com/brandonphilips">Brandon Phillips</a>, the CTO of CoreOS, explains the pattern:</p><blockquote>An Operator is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex stateful applications on behalf of a Kubernetes user.</blockquote><p>It’s not always clear what exactly it means for a piece of software to be a “Kubernetes Operator”, but I define it as the combination of a Custom Resource Definition (CRD) and a Custom Controller which reacts to changes in instances of the CRD. In this article, I will refer to the Operator when I am describing the capability provided by the combination of the CRD and the Controller.</p><h4>Custom Resources</h4><p>In Kubernetes, “resources” are defined via some declarative configuration. Usually these configurations are articulated via yaml:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dab39c3d59b878115643fecb8dfeaabd/href">https://medium.com/media/dab39c3d59b878115643fecb8dfeaabd/href</a></iframe><p>In this example, the resource is a Deployment. If you “apply” this resource, Kubernetes will endeavor to run 3 replicas of the nginx:1.7.9 container. Custom Resources allow users to extend the domain language of Kubernetes to map more closely to their environment.</p><h4>Custom Controllers</h4><p><a href="https://github.com/ngtuna">Tu Nguyen</a> has written <a href="https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html">an excellent blog article</a> on Kubernetes Controllers on the Bitnami blog. Further, the <a href="https://kubernetes.io/docs/admin/kube-controller-manager/">Kubernetes Documentation</a> explains controllers in the following words:</p><blockquote>In Kubernetes, a controller is a control loop that watches the shared state of the cluster through the API server and makes changes attempting to move the current state towards the desired state.</blockquote><p>So if the set of Custom Resources is the “desired state”, then it is the responsibility of the Custom Controller to observe the actual state and endeavor to converge actual state and desired state by taking actions against the running system.</p><h4>Prior Work: Multi-Tenancy Working Group</h4><p>The astute reader may already be aware of the fact that there is a Kubernetes Working Group focused on Multi-Tenancy which is lead by <a href="https://github.com/davidopp">David Oppenheimer</a> (Google) and <a href="https://github.com/jessfraz">Jessie Frazelle</a> (Microsoft). This working group deals with many different kinds of multi-tenancy and considers a wide variety of threat models. For a more complete overview of the work being done by this Working Group, I suggest David’s KubeCon Europe 2018 talk on <a href="https://www.youtube.com/watch?v=xygE8DbwJ7c">Multi-Tenancy in Kubernetes</a>.</p><p>At KubeCon Europe 2018, during the Multi-Tenancy WG Deep Dive, <a href="https://github.com/christianhuening">Christian Hüning</a> gave a <a href="https://youtu.be/Vi3ptlNjZJE?t=19m35s">talk on a Tenant Operator</a> that he is working on for his use-case of multi-tenant access to a Kubernetes cluster by potentially hostile users.</p><p>While related in several obvious way, the Operator that is described here is biased towards the deployment and on-going management of somewhat homogenous tenants that all come from the same distributor (Kolide Engineering). For our use-case, per-tenant data isolation is critical. Container and network isolation is important as well, but something we strive to improve over time. Resource metering is a non-issue for us because we don’t bill based on cluster utilization.</p><h3>Kolide’s Tenant Operator</h3><p>At Kolide, we have written an Operator which is responsible for managing the lifecycle of tenants in our Cloud product. This Operator intentionally embeds our domain and technology decisions into the software itself. The SRE team at Kolide endeavors to have the Operator manage as much of the daily operations of Kolide Cloud as possible.</p><h4>The Tenant CRD</h4><p>The main interface for interacting with the system is the “Tenant” Custom Resource Definition. Consider the following example of a Tenant CRD:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d98acf232712b335e54dd9e34f688036/href">https://medium.com/media/d98acf232712b335e54dd9e34f688036/href</a></iframe><p>Each Tenant resource maps to a single customer’s instance of the application. This approach allows us to boil down a tenant to just the attributes that change on an ongoing basis, such as:</p><ul><li>What databases should be reserved for each tenant?</li><li>What versions of each component of each tenant should be deployed?</li><li>How large should the database connection pool be?</li></ul><p>Details like initial customer email and organization are included so that internal systems can reach out to customers during the account provisioning process.</p><p>Replicas are explicitly not managed via this CRD because we manage replicas via external systems that react based on observed resource utilization and customer size.</p><h4>Control Loop</h4><p>Whenever a Tenant resource is updated, the resource is queued for processing on a work queue in the controller. When a tenant is processed by the controller, the following general steps occur:</p><ul><li>Create the Kubernetes namespace</li><li>Reserve and configure access to all required data infrastructure</li><li>Make sure that the relevant application secrets are available in the tenant’s namespace</li><li>Deploy the correct version of each component of the tenant</li><li>Apply network and container isolation primitives to secure workload execution</li></ul><p>It’s important for each of these steps to be idempotent. It’s also critical for the happy path through these steps (ie: nothing changed since the last synchronization) to happen extremely quickly. A global cluster event (like a code deploy) may enqueue literally every tenant for processing. Our product is only just out of beta, but this is already several hundred tenants for us and growing every day as new customers <a href="https://kolide.com/">sign up on our website</a>.</p><h4>Data Infrastructure</h4><p>Right now, we manage data infrastructure by interacting with <a href="https://godoc.org/cloud.google.com/go">GCP APIs</a> directly. Each tenant resource enumerates it’s own database requirements and the controller takes care of creating the requisite databases and user accounts. The controller will then save the per-tenant credentials in our secret storage mechanism which is described in more detail in the next section.</p><p>For future versions of our operator, we are actively exploring provisioning infrastructure services in a more generic way by taking advantage of the <a href="https://www.openservicebrokerapi.org/">Open Service Broker API</a>. The OSB API defines a <a href="https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md">specification</a> for provisioning and binding to infrastructure services. The ecosystem around the OSB API is extensive and the folks that are working on adding Service Catalog support to Kubernetes are doing excellent work.</p><h4>Secret Management</h4><p>In Kubernetes, workloads can only access secrets in their namespace. Because we run each customer’s workloads in their own namespace, we must copy shared secrets into each tenant namespace. Further, each tenant has several unique secrets for accessing isolated data infrastructure.</p><p>Being somewhat loyal Google Cloud customers at the moment, we use <a href="https://cloud.google.com/storage/">Google Cloud Storage</a> and <a href="https://cloud.google.com/kms/">Google Cloud Key Management Service</a> to store secrets encrypted at rest. SREs and infrastructure services are granted access to encrypt and decrypt secrets in different environments as necessary using <a href="https://cloud.google.com/iam/">Google Cloud IAM</a>. Since we use Kubernetes for literally everything at Kolide, we store serialized corev1.Secret blobs in GCS, encrypted at rest via KMS. It’s then easy for us to write tools which read secrets from GCS and unmarshal the secret into a Go objects. These tools are authenticated via GCP IAM rules and access can be easily audited and revoked as necessary.</p><p>The interface that we implement using the GCS and KMS APIs is as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5380b87837b4f832986ea91993b6c27a/href">https://medium.com/media/5380b87837b4f832986ea91993b6c27a/href</a></iframe><p>While our exo-cluster secret storage solution provides an effective mechanism for initializing clusters, neither GCS or KMS are low-latency enough to be in the hot path of the control loop. In order to support hundreds, if not thousands of tenants, the happy path of a tenant control loop needs to happen extremely quickly if no changes are required.</p><p>To do this, we first synchronize secrets from storage in a dedicated namespace within the cluster. This namespace is kept up-to-date. We then use <a href="http://borismattijssen.github.io/articles/kubernetes-informers-controllers-reflectors-stores">well-established Kubernetes API primitives</a> to maintain a local copy of all secrets in the cluster. When synchronizing a tenant, the control loop will use this cache to avoid unnecessary network interactions with any external secret storage mechanism whenever possible.</p><p>Optimizing the efficiency of the control loop has largely been an exercise in fast, accurate, secure secret management.</p><h4>Application Components</h4><p>Each application component consists of a templated set of Kubernetes resources and a set of values to interpolate into the resources. Internally, we call our tools to apply values into resource template “varz”. Each component of a tenant is developed in it’s own repository and the Kubernetes configurations for that component are included in that repository.</p><p>The main reason why we include configurations with each repository is so that application configuration can be modified atomically in the same commit that modified application deployment requirements. For example, if a service adds a new required environment variable, the default value can be added to the environment-specific Kubernetes deployment configurations by the developer.</p><p>When we deploy tenant components, we specify the commit hash that we want to deploy. The control loop will then deploy a container built at that ref by applying configurations pulled from the repository at that commit. This makes it easy to atomically maintain configuration while enabling developers to own the configuration for their services.</p><p>We often get asked why we don’t use Helm for this use-case. While our resource templates are similar to Helm Charts, a few features of Helm make it unsuitable for our use-case. Notably, Helm requires a cluster service called Tiller which is an unauthenticated-by-default Cluster Admin service that installs software in a cluster. Since our controller is already a privileged service that can install software, the Tiller is a superfluous service which presents an unnecessary level of attack surface. Fortunately the <a href="https://github.com/kubernetes-helm/community/blob/1e053cb837fa60990916ff359f673c0b19cb9192/helm-v3/000-helm-v3.md">design documents for Helm V3</a> describe a client-side model for installing Helm Charts and the Tiller is no longer a required component. Helm V3 sounds much more suitable for our use-case and we’re looking forward to trying it out once it’s available.</p><h3>Writing Kubernetes Controllers in Go</h3><p>I’ve previously written on the Kolide Blog about <a href="https://blog.kolide.com/using-go-for-scalable-operating-system-analytics-cb170d85b1c5">using Go for scalable operating system analytics</a>. When it comes to writing software in the Kubernetes ecosystem, Go is hands-down the best tool for the job.</p><p>I feel there can be a high barrier to entry in learning to use Kubernetes. Many of the concepts can feel foreign and it’s not always clear why every line of yaml needs to exist. Jason Moiron, a well-respected Go developer, recently wrote <a href="http://jmoiron.net/blog/is-k8s-too-complicated/">an interesting article which asked if Kubernetes is too complicated</a>.</p><p>In my personal experience, while using Kubernetes felt tricky at first, once I got the core concepts down, I realized that everything works together in a really predictable way that actually accelerates your ability to learn about and integrate new concepts.</p><p>Similarly, when programming against Kubernetes APIs, it can feel foreign at first to an experienced Go developer. In my experience however, once you learn the structure of the Go API client as well as how Kubernetes ecosystem projects organize themselves, it becomes incredibly easy to program against mainline Kubernetes APIs as well as external/custom APIs in a consistent, intuitive fashion.</p><h4>Go API Client: “Client Go”</h4><p>The Kubernetes Go API Client is maintained by the API Machinery Special Interest Group (SIG): <a href="https://github.com/kubernetes/client-go">https://github.com/kubernetes/client-go</a></p><p>The Go client has a really consistent look-and-feel for all types (Deployments, Pods, Namespaces, etc) in all API groups (core/v1, apps/v1, etc). For custom resources, there are code generation tools that allow users to generate API Clients from the struct definitions. <a href="https://blog.openshift.com/author/stttsredhat-com/">Stefan Schimanski</a> wrote a great introduction to Kubernetes API client code generation on the <a href="https://blog.openshift.com/kubernetes-deep-dive-code-generation-customresources/">Red Hat Open Shift Blog</a>.</p><h4>Directory Structure</h4><p>Most Kubernetes ecosystem projects follow a somewhat consistent directory layout. From the root of the repository, a controller will usually have the following directories:</p><ul><li>cmd/foo will contain the binary entrypoint</li><li>pkg/apis will contain the typed Kubernetes resources</li><li>pkg/client or pkg/generated will contain generated client code for interacting with that project’s resources</li></ul><p>One project that does a great job following the patterns of a consistent Kubernetes ecosystem project is the <a href="https://github.com/heptio/ark">Ark</a> disaster recovery tool, written by some really helpful engineers at <a href="https://www.heptio.com/">Heptio</a>. Given that <a href="https://twitter.com/jbeda">Joe Beda</a>, the creator of Kubernetes, is the CTO of Heptio, it’s perhaps not surprising that their code sets a good example. We use Ark to backup and provision Kubernetes clusters, but that is a story for another blog.</p><p>Other operators that follow these consistent patterns nicely in my opinion are:</p><ul><li><a href="https://github.com/coreos/etcd-operator">CoreOS’s Etcd Operator</a></li><li><a href="https://github.com/coreos/prometheus-operator">CoreOS’s Prometheus Operator</a></li><li><a href="https://github.com/kinvolk/habitat-operator">Kinvolk’s Habitat Operator</a></li></ul><h4>Repository Size</h4><p><a href="https://youtu.be/4usXJE0EwHo?t=11m22s">Dave Cheney has spoken about his opinion on how large a controller codebase should be</a> and, at the time of writing, our controller is about 12,000 lines of Kolide-authored Go code. We’ve found that a codebase of this size is extremely manageable for us and we are not concerned about accommodating future growth effectively.</p><h4>CoreOS’s Operator Framework</h4><p>On my birthday this year, CoreOS released the <a href="https://coreos.com/blog/introducing-operator-framework">Operator Framework</a>, an open source toolkit designed to manage Operators in a more effective, automated, and scalable way. So far, the Operator Framework contains three components which help with:</p><ul><li>Operator development</li><li>Lifecycle management</li><li>Resource Metering</li></ul><p>For our use-case at Kolide, complicated lifecycle management and resource metering are not problems that we have, but one common problem that all controllers have to solve is the queueing and synchronized processing of resources as they change over time.</p><p>For most new controllers, I think that it probably makes sense to consider using the CoreOS library for this objective. If your use-case is sufficiently complicated or you just want more control, it may be worth using something like the <a href="https://github.com/kubernetes/sample-controller">official Sample Controller</a> as an adaptable starting point instead.</p><h3>Conclusions</h3><p>Kubernetes has been described as a set of APIs for building distributed systems and as a “platform for building platforms”. We’ve enjoyed using Kubernetes to deploy, scale, and secure our product. We’re looking forward to continuing to discuss how we use Kubernetes to deliver Kolide Cloud here on the Kolide Blog.</p><h4>About the Author</h4><p><a href="https://github.com/marpaia">Mike Arpaia</a> is Co-Founder &amp; CTO of <a href="https://kolide.com/"><strong>Kolide</strong></a><strong> </strong>and the creator of <a href="https://osquery.io/"><strong>osquery</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uuFSDRnXkDJmGaPZ.png" /><figcaption>The Kolide Cloud Desktop App gives your users visibility into the state of their device.</figcaption></figure><p><a href="https://www.kolide.com/"><strong>Kolide</strong></a><strong> </strong>is an endpoint monitoring solution which leverages and instruments Facebook’s open-source <a href="https://github.com/facebook/osquery"><strong>osquery</strong></a> project. <a href="https://k2.kolide.com/signup">Try it today; completely free for the first 14 days for your entire fleet</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=250f1c9416ce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kolide/using-a-kubernetes-operator-to-manage-tenancy-in-a-b2b-saas-app-250f1c9416ce">Using a Kubernetes Operator to Manage Tenancy in a B2B SaaS App</a> was originally published in <a href="https://medium.com/kolide">Kolide</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Go for Scalable Operating System Analytics]]></title>
            <link>https://medium.com/kolide/using-go-for-scalable-operating-system-analytics-cb170d85b1c5?source=rss-d12efd302743------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb170d85b1c5</guid>
            <category><![CDATA[osquery]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Mike Arpaia]]></dc:creator>
            <pubDate>Thu, 16 Nov 2017 18:08:16 GMT</pubDate>
            <atom:updated>2020-09-28T19:01:51.811Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I_ceW2OlKb675LQJG0Wylw.png" /></figure><p>At Kolide, part of what we do is build a client-server system that facilitates the management of large, isolated installations of an open-source tool called <a href="https://osquery.io/">osquery</a>. While I’ve written and spoken about osquery in the past, in this post I want to focus on how we use Go to solve many of the challenges we face building software for the osquery community and for internal distributed systems.</p><p>Osquery is a operating system analytics tool that allows you to articulate operating system state using SQL queries. You then provide osquery with a declarative configuration of queries that you want to use to monitor a host. With each query, you use options to define how the results of these queries should be logged (i.e.: log whenever the result-set changes, log a snapshot of the data-set at a set interval, etc). An operator can then use this data to create and annotate a current, evolving data-model of your infrastructure. You can also store this raw data (JSON) and load it into a data warehouse like <a href="https://cloud.google.com/dataproc/">HDFS</a> or <a href="https://cloud.google.com/bigquery/">BigQuery</a>. Analysis of this data can be useful for accomplishing objectives related to security, operations, device administration, and compliance.</p><p>At Kolide, we have a lot of experience with osquery and managing osquery infrastructure. In order for an osquery deployment across your fleet to be successful, you need to have excellent packaging tooling: osquery provides a lot of capabilities, but in-order to take advantage of it’s power, configuration can be rather complex. As of osquery version 2.10.1, there are 115 configurable options. Configuration via the filesystem is most effectively provided with a directory structure of files. You may use custom extensions, which are to be distributed as platform-specific binary executables. To accomplish your specific objectives, you must have <a href="https://github.com/kolide/launcher/tree/master/cmd/package-builder">good tooling for packaging and distributing osquery</a>.</p><p>Managing osquery also requires diligent update hygiene: in order to take advantage of the most reliable software that allows you to monitor for emergent threats, you should really be keeping osqueryd very current. Osquery is a fast-moving project and a lot of great work is happening all throughout the community. Using an osquery manager which is capable of auto-updating and managing osquery as per an “update channel” (i.e.: “beta”, “stable”) makes this process much easier, especially in environments where clients and package management may be unreliable.</p><h3>The Osquery Launcher</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTP6u-llOnFya0VvhQ-pHw.png" /></figure><p>At Kolide, the tool we use for managing osquery instances, autoupdating osquery, and establishing remote communication with a specified server is called “Launcher”. You can read about Launcher on <a href="https://kolide.com/launcher">our website</a> and <a href="https://github.com/kolide/launcher">on GitHub</a>. Launcher is written in Go and includes a number of niceties that make scaling osquery deployments easier. For the Go developer, some of my favorite features / parts of the codebase include:</p><ul><li>An <a href="https://github.com/kolide/launcher/blob/master/osquery/runtime.go">osquery runtime</a> which exposes a programmatic Go interface that takes advantage of the Functional Option API pattern made popular by Dave Cheney (<a href="https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis">blog</a>, <a href="https://www.youtube.com/watch?v=24lFtGHWxAQ">video</a>). Once <a href="https://twitter.com/wikiwalk">@groob</a> introduced me to this pattern, I fell in love with it. It definitely takes more time and boilerplate, but the beauty of the resultant API and the additional type safety make it a must-have for these kinds of APIs.</li><li>A modern, type-safe <a href="https://github.com/kolide/launcher/blob/master/service/internal/launcherproto/launcher.proto">gRPC server specification</a>. <a href="https://grpc.io/">gRPC</a>, a CNCF project created by Google, is an excellent RPC framework that is basically <a href="https://developers.google.com/protocol-buffers/docs/overview">Protocol Buffers</a> over <a href="https://developers.google.com/web/fundamentals/performance/http2/">HTTP2</a> with a whole bunch of fun capabilities. We write a lot of Launcher Servers, so having a well-defined, versioned server specification is useful.</li><li><a href="https://blog.kolide.com/how-we-securely-autoupdate-osquery-at-kolide-b0eda6ad05f6">A secure osquery auto-update system</a> using The Update Framework. TUF, a CNCF project created by Docker, is a specification that outlines a peer-reviewed update mechanism that is robust and complete. The specification articulates a client/server model. Kolide hosts a TUF server using the Notary tool. <a href="https://docs.docker.com/notary/getting_started/">Notary</a>, a CNCF project created by Docker, provides a TUF server and we wrote a purpose-built TUF client called <a href="https://github.com/kolide/updater">Updater</a> at Kolide (in Go of course).</li></ul><p>All of these components are wired together in the Launcher project in the main file of the Launcher repository (<a href="https://github.com/kolide/launcher/blob/master/cmd/launcher/launcher.go">cmd/launcher/launcher.go</a>).</p><h3>Kolide Fleet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fV4RZ7IdUIdH2Zf4yAEaFQ.png" /></figure><p>The Launcher is designed to communicate with a server that fulfills the provided gRPC server specification. Package Builder makes it easy to distribute the Launcher and Osquery but a server is necessary to complete the transport. For this, there is <a href="https://kolide.com/fleet">Kolide Fleet</a>, an open-source osquery fleet manager. <a href="https://blog.kolide.com/kolide-fleet-an-open-source-osquery-fleet-manager-26e8094fab">We have previously written about Fleet</a> on this blog and one of the excellent features of Fleet is that it supports both the existing osquery TLS server API as well as the new osquery gRPC server API. This makes converting easy for users with an existing osquery deployment.</p><h3>Patterns</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TlfoSznEvJhrpApGmnseMQ.png" /></figure><p>The <a href="https://github.com/kolide/launcher/tree/master/cmd/launcher">Launcher</a> and <a href="https://github.com/kolide/launcher/tree/master/cmd/package-builder">Package Builder</a>, and <a href="https://github.com/kolide/fleet/blob/master/docs/development/building-the-code.md">Fleet</a> all follow similar patterns that we like to follow for all Go projects at Kolide:</p><ul><li>The minimal program files should always be in cmd/program/.</li><li>Use Go-Kit whenever you need to map an idiomatic program interface to a remote transport (like gRPC or HTTP).</li><li>Use the new Go Dep tool to manage dependencies.</li><li>Use Dave Cheney’s error package to wrap errors.</li><li>Use the command parsing pattern <a href="https://github.com/oklog/oklog/blob/master/cmd/oklog/main.go">established by Peter Bourgon in OKLog</a> instead of command-parsing libraries if the CLI is not extremely complex.</li><li>Don’t use init and don’t use package-level variables. <a href="https://peter.bourgon.org/blog/2017/06/09/theory-of-modern-go.html">Peter Bourgon has written about this rather convincingly</a>. Peter says:</li></ul><blockquote>tl;dr: magic is bad; global state is magic → no package level vars; no func init.</blockquote><ul><li>Propagate a context.Context throughout your APIs. Use this context to pass things like request UUIDs, which are useful when doing distributed tracing of requests.</li><li>Supply a Dockerfile with each project to facilitate Cloud Native testing and deployment strategies.</li></ul><h4>Kolide Kit &amp; Style Guide</h4><p>We have written about these patterns and more in our <a href="https://github.com/kolide/kit/blob/master/styleguide.md">Go Style Guide</a> which lives in the repository where we keep many Go libraries and helpers that we frequently use across our projects: <a href="https://github.com/kolide/kit">kolide/kit</a>. We have many utilities and helpers that make writing modern TLS servers easier, helpers on top of the <a href="https://godoc.org/github.com/go-kit/kit/log">Go-Kit Logger</a>, and more. We think these libraries can be generically useful and we plan on blogging more about them in the future.</p><h3>A Consistent Developer Experience using GNU Make</h3><p>In addition to Fleet, Launcher, Updater, and Kit, we also have several projects in this domain that are internal. Having clearly not gone the way of the monorepo, having some consistencies across all these projects really adds to the developer experience.</p><p>Whenever a developer at Kolide creates a new Go repo, they will add a Makefile that follows a similar format.</p><p>The basic Makefile structure that every Kolide project uses requires the following commands:</p><ul><li>make deps</li><li>make test</li><li>make build</li><li>./build/template --help</li><li>./build/template version</li></ul><pre># install the dependency manager and invoke it<br># all kolide projects use dep now, but we used to always use glide<br># hiding this behind `make deps` made the transition transparent</pre><pre>make deps</pre><pre># run the full test-suite<br># this is (hopefully) just a light wrapper on top of go test<br># sometimes we also add linters and static analyzers here too</pre><pre>make test</pre><pre># build the binary for your platform<br># you can also run `make xp` to cross-compile a mac and linux binary</pre><pre>make build</pre><pre>## now you can run the program, get help, and version information</pre><pre>./build/template --help<br>./build/template version</pre><p>An example Makefile that implements this above functionality is included here. Note the usage of the version package fromkolide/kit which is used to easily add git-based version tooling to a program.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9122a657c889f256de64955e6eda7ba/href">https://medium.com/media/d9122a657c889f256de64955e6eda7ba/href</a></iframe><p>Additionally, consider the <a href="https://gist.github.com/marpaia/8adc4cf0a08d84824e9d8792567db1cc">minimal main file of this program</a>, which uses the kolide/kit version package and the OKLog command parsing pattern.</p><p>These files should live at Makefile and cmd/template/template.go for this example, but obviously paths and the template string would need to be replaced for your usage.</p><h3>Development Infrastructure: Docker Compose and Minikube</h3><p>Many of our Go (and non-Go) projects require some sort of infrastructure to enable effective local development. For example, Kolide Fleet requires <a href="https://www.mysql.com/">MySQL</a> and <a href="https://redis.io/">Redis</a>. <a href="https://github.com/mailhog/MailHog">MailHog</a> is also useful for testing SMTP features. To solve this requirement for local development, we usually use <a href="https://docs.docker.com/compose/overview/">Docker Compose</a>. For example, check out Fleet’s <a href="https://github.com/kolide/fleet/blob/78b831a6d25f9242953b17b7e40b69aee3f698bf/docker-compose.yml">docker-compose.yml</a>.</p><p>In the future, I would like to start defining all of this as a Kubernetes Deployment or a ReplicaSet and manage local development infrastructure via <a href="https://github.com/kubernetes/minikube">Minikube</a>. Since we use Kubernetes in production at Kolide, this parity with the local environment would be an interesting way to further shrink the gap between development and production.</p><h3>Conclusion</h3><p>At Kolide, we create and manage client-server software to make operating system analytics easy, modern, and scalable. This domain lends itself very well to what Go is really good at. We love writing the best Go that we can and we’re constantly trying to improve. If you think you can help us write great Go, <a href="https://kolide.com/jobs">we’re hiring</a>!</p><h4>About the Author</h4><p><a href="https://github.com/marpaia">Mike Arpaia</a> is Co-Founder &amp; CTO of <a href="https://kolide.com/"><strong>Kolide</strong></a><strong> </strong>and the creator of osquery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uuFSDRnXkDJmGaPZ.png" /><figcaption>The Kolide Cloud Desktop App gives your users visibility into the state of their device.</figcaption></figure><p><a href="https://www.kolide.com/"><strong>Kolide</strong></a> is an endpoint monitoring solution which leverages and instruments Facebook’s open-source <a href="https://github.com/facebook/osquery"><strong>osquery</strong></a> project. <a href="https://k2.kolide.com/signup">Try it completely free for the first 14 days for your entire fleet.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb170d85b1c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kolide/using-go-for-scalable-operating-system-analytics-cb170d85b1c5">Using Go for Scalable Operating System Analytics</a> was originally published in <a href="https://medium.com/kolide">Kolide</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we securely autoupdate Osquery at Kolide]]></title>
            <link>https://medium.com/kolide/how-we-securely-autoupdate-osquery-at-kolide-b0eda6ad05f6?source=rss-d12efd302743------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0eda6ad05f6</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[osquery]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Mike Arpaia]]></dc:creator>
            <pubDate>Thu, 26 Oct 2017 15:08:33 GMT</pubDate>
            <atom:updated>2017-10-26T15:44:03.154Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*c-_mxEyHLLKb_ryHgvdyJg.gif" /></figure><p><a href="https://kolide.com/osquery">Osquery</a> is a fast-moving <a href="https://github.com/facebook/osquery">open source project</a> that allows you to monitor a host using SQL. The tables that are provided by osquery are embedded in the tool itself, so getting the latest capabilities requires keeping your osquery deployment up-to-date. To make this easier for our users, we created an<a href="https://kolide.com/launcher"> osquery launcher</a> that includes the ability to securely manage and autoupdate osquery instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*JnAcwQEMVyeO-IFmflVYyw@2x.png" /></figure><p>After researching available solutions, we decided to implement our autoupdater with<a href="https://theupdateframework.github.io/"> <strong>The Update Framework (TUF)</strong></a>. TUF defines a <a href="https://github.com/theupdateframework/specification/blob/master/tuf-spec.md">specification</a> for secure software update systems. The spec describes a client/server model where the client is the software to be updated and the server is the update server.</p><p>For our implementation, we use<a href="https://github.com/docker/notary"> Docker Notary</a> as our TUF server and a Go client library that <a href="https://github.com/kolide/updater">we built in-house</a> called “Updater”. Through Updater, Launcher uses a mirror like Google Cloud Storage to store update targets, and uses<a href="https://github.com/docker/notary"> Notary</a> to ensure that targets have not been tampered with. We don’t take the security implications of automatically updating osquery lightly, so we knew an independent third-party audit of this architecture was necessary. To this end, we contracted NCC Group to perform a security audit of our in-house TUF client library. The report is publicly available at <a href="https://www.nccgroup.trust/globalassets/our-research/us/public-reports/2017/ncc-group-kolide-the-update-framework-security-assessment.pdf">here</a>. NCC Group has also previously performed assessments on <a href="https://www.nccgroup.trust/us/our-research/docker-notary/">Docker Notary</a> and <a href="https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2016/march/ncc-group-reviews-osquery/">Osquery</a> as well, so this assessment bridges the two together nicely.</p><p>The<a href="https://www.nccgroup.trust/us/our-research/docker-notary/"> Docker Notary assessment</a> was carried out by NCC Group’s “Cryptography Services” which was led by cryptography expert <a href="https://ritter.vg/">Tom Ritter</a>. Docker’s lead on getting this audit done was <a href="https://twitter.com/diogomonica">Diogo Monica</a> who has<a href="https://www.youtube.com/watch?v=2Dyz8huMfIo"> frequently spoken</a><a href="https://www.youtube.com/watch?v=hlzpfNdc6wU"> rather insightfully</a> on this topic.</p><p>Ted Reed and I first discussed the<a href="https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2016/march/ncc-group-reviews-osquery/"> osquery assessment</a> in <a href="https://code.facebook.com/posts/226775617661196/in-pursuit-of-secure-open-source-software/">an article on Facebook’s engineering blog</a>, along with a myriad of other activities the osquery team undertook to maintain a secure open source ecosystem. In that article, I spoke about the idea of “constant vigilance” as a software developer. Public assessments are a point-in-time verification of one aspect of security and are not meant to imply that the software is secure <em>right now</em>. However, it <em>does</em> imply that there are no jarring architectural flaws that the authors missed when first reasoning about the design of their solution.</p><p>The Updater assessment ties these two components together. Keeping osquery up to date via Updater and Notary is noteworthy because <a href="https://www.nccgroup.trust/us/our-research/docker-notary/">Notary</a>, <a href="https://www.nccgroup.trust/us/our-research/the-update-framework-tuf-security-assessment/">Updater</a>, and <a href="https://www.nccgroup.trust/us/our-research/osquery-application-security-assessment/">Osquery</a> have all had security assessment reports publicly published. The Notary server infrastructure that Kolide hosts for this purpose has not been audited yet, but we are looking forward to further investing in this infrastructure as time goes on. Consider the following excerpt from the executive summary of the NCC Group report’s Executive Summary section:</p><blockquote>During the summer of 2017, Kolide engaged NCC Group to conduct a security assessment of their implementation of a client for TUF, The Update Framework. Two NCC Group consultants performed this assessment between August 28 and September 1. Along with a Docker Notary server, this client allows osquery to automatically and securely update itself. This report, in conjunction with NCC Group’s prior assessments of Notary and osquery, completes a trifecta of security assessments for a fully functional suite of interdependent secure endpoint management technologies.</blockquote><blockquote>Though NCC Group did find some issues that could affect users in less-secure environments, the assessment did not uncover any findings that affect the security of the processes for receiving and verifying updates.</blockquote><blockquote>- <a href="https://www.nccgroup.trust/us/our-research/the-update-framework-tuf-security-assessment/">The Update Framework (TUF) Security Assessment</a></blockquote><p>We take the responsibility of providing an osquery autoupdater seriously. If this is interesting to you, I encourage you to consider the following resources and help us by reporting any potential vulnerabilities in our software.</p><ul><li><a href="https://github.com/kolide/updater">The Updater library source code</a></li><li><a href="https://www.nccgroup.trust/globalassets/our-research/us/public-reports/2017/ncc-group-kolide-the-update-framework-security-assessment.pdf">The Updater security audit</a></li><li><a href="https://github.com/kolide/launcher">The Launcher source code</a></li><li><a href="https://github.com/facebook/osquery">The Osquery source code</a></li><li><a href="https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2016/march/ncc-group-reviews-osquery/">The Osquery security audit</a></li><li><a href="https://github.com/docker/notary">The Notary source code</a></li><li><a href="https://www.nccgroup.trust/us/our-research/docker-notary/">The Notary security audit</a></li><li><a href="https://github.com/theupdateframework/specification/blob/master/tuf-spec.md">The Update Framework (TUF) Specification</a></li><li><a href="https://kolide.com/jobs">The Kolide Jobs Page</a></li></ul><p>Please contact us at <a href="/service/mailto:security@kolide.com">security@kolide.com</a> if you believe you have found vulnerabilities in Kolide or osquery software.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0eda6ad05f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kolide/how-we-securely-autoupdate-osquery-at-kolide-b0eda6ad05f6">How we securely autoupdate Osquery at Kolide</a> was originally published in <a href="https://medium.com/kolide">Kolide</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing Osquery with Kolide Launcher and Fleet]]></title>
            <link>https://medium.com/kolide/managing-osquery-with-kolide-launcher-and-fleet-b33b4536acb4?source=rss-d12efd302743------2</link>
            <guid isPermaLink="false">https://medium.com/p/b33b4536acb4</guid>
            <category><![CDATA[osquery]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[kolide]]></category>
            <dc:creator><![CDATA[Mike Arpaia]]></dc:creator>
            <pubDate>Fri, 20 Oct 2017 18:12:09 GMT</pubDate>
            <atom:updated>2020-09-28T19:05:13.560Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*I7ZdZ6EwUNrdBXNUXSLAqQ@2x.png" /></figure><p>In this article, I want to walk-through setting up a local Kolide Fleet server with a local instance of osquery via the Kolide Launcher. Before you dig in, I highly recommend you read our <a href="https://blog.kolide.com/kolide-fleet-an-open-source-osquery-fleet-manager-26e8094fab">Fleet</a> and <a href="https://blog.kolide.com/kolide-launcher-making-it-easier-to-deploy-osquery-439cf814425c">Launcher</a> announcement blog posts. This tutorial will build both tools from source to illustrate how to interact with the open-source repositories.</p><p>First, let’s clone the Fleet repository from GitHub:</p><pre>$ cd $GOPATH/src/github.com/kolide<br>$ git clone git@github.com:kolide/fleet.git<br>$ cd fleet</pre><p>From the root of the repository, let’s install the application dependencies, generate the JavaScript bundle, and build the binary:</p><pre>$ make deps<br>$ make generate<br>$ make</pre><p>Now, we will test this locally. We’re going to have to do something about the infrastructure requirements (database and cache). The Fleet repo includes Docker Compose tooling for this exact reason. Start the infrastructure with the following command:</p><pre>$ docker-compose up</pre><p>To set up the database schema of your new Fleet instance, use the fleet command:</p><pre>$ ./build/fleet prepare db<br>Migrations completed.</pre><p>Now, to run the server locally, the only required flag is --auth_jwt_key which defines the key used for session tokens. For example, you may run:</p><pre>./build/fleet serve --auth_jwt_key=3zqHl2cPa0tMmaCa9vPSEq6dcwN7oLbP</pre><p>Note that in production, this value can also be set via environment variables or a config file.</p><p>Now if you browse to <a href="https://localhost:8080,">https://localhost:8080,</a> you should see the interactive setup flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Urx5kewEyepETQB0E8LJTQ.png" /></figure><p>After you create your account and log in, you will be able to pull up the “Add New Host” screen, which should look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wOHZtPL9wjkr6vlYZnZ1sA.png" /></figure><p>Select “Reveal Secret” and you will be able to see and copy your Fleet server’s enroll secret. You’ll need this to connect a Launcher to Fleet.</p><p>Now let’s connect a host via Launcher. For the sake of this example, we will also build Launcher from source. First, we must clone the repository:</p><pre>cd $GOPATH/src/github.com/kolide/<br>git clone git@github.com:kolide/launcher.git<br>cd launcher</pre><p>Now, we follow a similar flow to Fleet in order to install the dependencies and build the binaries:</p><pre>make deps<br>make<br>./build/launcher --help</pre><p>Assuming you have the osqueryd installed on your local host (<a href="https://osquery.io/downloads/">https://osquery.io/downloads/</a>), you can run the following to connect to Fleet:</p><pre>./build/launcher --debug \<br>  --hostname=localhost:8080 \<br>  --enroll_secret=&quot;MFyZh3+l1VUFNHE5Q3U9DLc6Xab1c6/C&quot; \<br>  --insecure</pre><p>We include --insecure because the TLS certificate we’re using for locahost:8080 does not have a trusted signature. After running this command, you should see the host appear in the web UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V8l68ps478SxUZ9cso6lrw.png" /></figure><p>Feel free to pause here and go play with the host you’ve just enrolled! Click the little database symbol on the host card to run a live query.</p><p>Now, let’s assume you want to get your co-workers enrolled in this too. You need to build a package and send it to them. Luckily, the Launcher includes a tool to generate cross-platform packages for your environment. To build the tool, run the following from the root of the Launcher repo:</p><pre>make deps<br>make package-builder<br>./build/package-builder make --help</pre><p>First, build a package that will install osquery and connect it to the local Fleet instance that running on <a href="https://localhost:8080.">https://localhost:8080</a>. Obviously the hostname will be different when you actually deploy Fleet in production, but it’s easy enough to play with the tools, so let’s give it a try! All you have to do is call ./build/package-builder make with the same flags you passed to the Launcher:</p><pre>./build/package-builder make \<br>  --hostname=localhost:8080 \<br>  --enroll_secret=&quot;MFyZh3+l1VUFNHE5Q3U9DLc6Xab1c6/C&quot; \<br>  --insecure</pre><p>This will print out a list of package paths. Open and install the macOS package.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UnhsB3HfcqrHlHT8LiZa7Q.png" /></figure><p>You should now see another instance of your host in Fleet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_OYw9kivCnX7IY5p2CHnJQ.png" /></figure><p>If this was your actual Kolide Fleet instance, you could easily deploy out this package to all of the macOS users in your company and they would be enrolled with an autoupdating osquery package! For more information on deploying Fleet in production, see the <a href="https://github.com/kolide/fleet/tree/master/docs/infrastructure">Fleet Infrastructure Documentation</a>.</p><h4>About the Author</h4><p><a href="https://github.com/marpaia">Mike Arpaia</a> is Co-Founder &amp; CTO of <a href="https://kolide.com/"><strong>Kolide</strong></a><strong> </strong>and the creator of osquery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uuFSDRnXkDJmGaPZ.png" /><figcaption>The Kolide Cloud Desktop App gives your users visibility into the state of their device.</figcaption></figure><p><a href="https://www.kolide.com/"><strong>Kolide</strong></a><strong> </strong>is an endpoint monitoring solution which leverages and instruments Facebook’s open-source <a href="https://github.com/facebook/osquery"><strong>osquery</strong></a> project. <a href="https://k2.kolide.com/signup">Try it today for free.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b33b4536acb4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kolide/managing-osquery-with-kolide-launcher-and-fleet-b33b4536acb4">Managing Osquery with Kolide Launcher and Fleet</a> was originally published in <a href="https://medium.com/kolide">Kolide</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kolide Launcher: Osquery deployment made easy.]]></title>
            <link>https://medium.com/kolide/kolide-launcher-making-it-easier-to-deploy-osquery-439cf814425c?source=rss-d12efd302743------2</link>
            <guid isPermaLink="false">https://medium.com/p/439cf814425c</guid>
            <category><![CDATA[osquery]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Mike Arpaia]]></dc:creator>
            <pubDate>Wed, 18 Oct 2017 18:35:01 GMT</pubDate>
            <atom:updated>2020-09-28T19:05:36.435Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*coRhUKcncSNJ8N5X1mgixg@2x.png" /></figure><p>At Kolide, we have a lot of experience deploying and using <a href="https://osquery.io/">osquery</a>. We want to use this experience to make deploying and managing osquery fleets easier than ever.</p><p>As an engineer at Facebook, I conceptualized and created osquery. Later, I managed the deployment of osquery to the entire Facebook fleet as the leader of Facebook’s Intrusion Detection Infrastructure team. My cofounder, Zach Wasserman, was a core contributor to osquery at Facebook, and went on to found a consulting company developing (and open-sourcing) custom osquery integrations for Silicon Valley customers.</p><p>Osquery is founded on the idea that instrumentation should be open and audited. It’s not reasonable for a security company (or any company) to ask users to install powerful, opaque rootkits. We founded Kolide to leverage our experience with osquery over a wide-range of enterprises and empower a more open, secure internet.</p><p>Today we are open-sourcing <a href="https://kolide.com/launcher">our osquery launcher</a> because we stand by these principles of openness.</p><p>The <a href="https://kolide.com/launcher">Kolide Launcher</a> is a lightweight manager runtime for osquery which offers extra capabilities and quality of life improvements that make deploying and using osquery easier and more scalable, for organizations of any size.</p><h3>Automatic Updates</h3><p>The first, perhaps most notable capability that the Launcher adds is secure automatic updates of the osquery binary as well as the Launcher binary itself. Osquery is statically linked and that allows for the easy bundling and distribution of the software. But this also requires users to maintain excellent update hygiene in order to take advantage of emerging capabilities. The Launcher includes the ability to securely manage and autoupdate osquery instances. This is implemented using<a href="https://theupdateframework.github.io/"> The Update Framework (TUF)</a>. TUF defines a<a href="https://github.com/theupdateframework/tuf/blob/develop/docs/tuf-spec.md"> specification</a> for secure software update systems. The specification describes a client/server model in which the client is the software to be updated and the server is the update server. For our implementation, we use<a href="https://github.com/docker/notary"> Docker Notary</a> as our TUF server and a Go client library that<a href="https://github.com/kolide/updater"> we built in-house</a>. Because we understand the security implications of an osquery auto-updater, we contracted with NCC Group to perform a security audit of our in-house TUF client library. NCC Group has also performed assessments on<a href="https://www.nccgroup.trust/us/our-research/docker-notary/"> Docker Notary</a> and<a href="https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2016/march/ncc-group-reviews-osquery/"> Osquery</a> itself.</p><h3>New Remote Plugin: gRPC</h3><p>Next, Launcher also adds a new set of remote plugins for osquery that establish a typed server specification. Osquery has a very extensible plugin architecture that allow it to be heavily customized with plugins. The Launcher includes a set of gRPC plugins for remote communication with a gRPC server. The<a href="https://github.com/kolide/agent-api"> server specification</a> is independently published and versioned. An implementation of the gRPC server is included with the<a href="https://github.com/kolide/fleet"> Kolide Fleet</a> osquery fleet manager. Kolide Fleet implements both the gRPC server as well as the existing TLS server API, so it presents an easy migration path for existing TLS API users.</p><h3>Use-Case Driven Tables</h3><p>It can be hard to know questions to ask and what queries will expose the answers. These new tables provide immediate insights into aspects of concern on macOS systems.The Launcher includes a kolide_best_practices table which aggregates useful information in an easy “compliant” vs. “non-compliant” interface. Also included is a kolide_vulnerabilities table which Launcher populates with active vulnerabilities found on the host.</p><h3>Reduced Configuration Surface</h3><p>The launcher also provides a dramatically reduced configuration surface. The Launcher wraps osqueryd configuration and exposes very high-level options that allow you to easily connect osquery to a server that is compliant with the<a href="https://github.com/kolide/agent-api/blob/master/agent_api.proto"> gRPC specification</a> (such as<a href="https://github.com/kolide/fleet"> Kolide Fleet</a>). This means that connecting Launcher to Fleet requires only two flags: server hostname and secret. To learn about The Launcher’s command-line interface, see the<a href="https://github.com/kolide/launcher/blob/master/cmd/launcher/README.md"> documentation</a> for the Launcher command. Compare this with over a dozen flags required to connect raw osquery to a TLS server.</p><h3>Tooling For Custom Package Generation</h3><p>Finally, Deploying osquery and configuring it to communicate with a management server can be complicated, especially if you have to make customized deployment packages. The Launcher includes a tool called package-builder which you can use to create no-fuss Launcher packages for your organization.The options and capabilities are documented on<a href="https://github.com/kolide/launcher/blob/master/cmd/package-builder/README.md"> GitHub</a>.</p><p>We hope that Launcher makes your life easier. An implementation of a compatible server is included with the<a href="https://github.com/kolide/fleet"> Kolide Fleet</a> osquery fleet manager. Please reach out with questions in the #kolide channel in <a href="https://osquery-slack.herokuapp.com/">osquery slack</a>. The source code and documentation is on GitHub at <a href="https://github.com/kolide/launcher">https://github.com/kolide/launcher</a>.</p><h4>About the Author</h4><p><a href="https://github.com/marpaia">Mike Arpaia</a> is Co-Founder &amp; CTO of <a href="https://kolide.com/"><strong>Kolide</strong></a><strong> </strong>and the creator of osquery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uuFSDRnXkDJmGaPZ.png" /><figcaption>The Kolide Cloud Desktop App gives your users visibility into the state of their device.</figcaption></figure><p><a href="https://www.kolide.com/"><strong>Kolide</strong></a><strong> </strong>is an endpoint monitoring solution which leverages and instruments Facebook’s open-source <a href="https://github.com/facebook/osquery"><strong>osquery</strong></a> project. <a href="https://k2.kolide.com/signup">Try it today; completely free for the first 14 days for your entire fleet.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=439cf814425c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kolide/kolide-launcher-making-it-easier-to-deploy-osquery-439cf814425c">Kolide Launcher: Osquery deployment made easy.</a> was originally published in <a href="https://medium.com/kolide">Kolide</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kolide Fleet: An open-source osquery fleet manager.]]></title>
            <link>https://medium.com/kolide/kolide-fleet-an-open-source-osquery-fleet-manager-26e8094fab?source=rss-d12efd302743------2</link>
            <guid isPermaLink="false">https://medium.com/p/26e8094fab</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[osquery]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Mike Arpaia]]></dc:creator>
            <pubDate>Wed, 18 Oct 2017 18:34:22 GMT</pubDate>
            <atom:updated>2020-09-28T19:06:06.746Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*t17v4ZaUn-4Nz7tmAJWElQ@2x.png" /></figure><p><a href="https://osquery.io/">Osquery</a> is a tool that allows users to monitor and ask questions about servers and workstations with an easy and expressive query language. Released <a href="https://www.youtube.com/watch?v=bcfg4VLyIHw">three years ago</a> at Facebook’s “Security @ Scale” conference, osquery is the most powerful open-source host instrumentation agent. Though osquery exposes rich capabilities, it only solves part of the host instrumentation problem. Using osquery on more than one host requires a server deployment in order to orchestrate and interact with the fleet of hosts running the agent. Today, Kolide is open-sourcing our osquery management server: <a href="https://kolide.com/fleet">Kolide Fleet</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8e8qGLz4vmoR5x7I." /></figure><p><a href="https://kolide.com/fleet">Kolide Fleet</a> is a beautiful, minimal, open-source web application for managing a fleet of hosts running osquery. Fleet gives you a place to store and iterate on osquery queries. You can run these queries on any subset of your hosts and instantly get the results flowing back into your browser.</p><p>Fleet has browser-based analytics tools that allow you to interactively filter and search results. For more advanced analytics, you can export the results of a live query, or integrate with the <a href="https://github.com/kolide/fleet/blob/master/docs/api/README.md">Fleet API server</a>: Fleet’s UI is a React app that interacts with a robust Go-based TLS API. You can easily use this API directly in your own tools.</p><p>In addition to running a query once and getting immediate results, Fleet allows you to group those queries into query packs and perform ongoing monitoring. You define how often you want the queries to be executed, whether to track changes to results, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DRy4wDc0FnKFR-0s." /></figure><p>Kolide Fleet also allows grouping of endpoints based on properties defined in queries. Imagine being able to view (and target additional scrutiny towards) machines with processes listening on non-standard ports, or those running out-of-date software. Fleet makes it easy to categorize and target subsets of hosts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mLCHIyZO0ngrbeXL." /></figure><p>Osquery itself exposes a plethora of capabilities to the user: live queries, proactive differential monitoring, periodic state snapshots, and more. Kolide Fleet provides an intuitive UI for all of this, while still exposing the power and flexibility of osquery. Advanced configuration options, decorators and other features are supported.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ujq3yrEjHpFAcqN7." /></figure><p>We’re excited to introduce Kolide Fleet into the osquery ecosystem. If you’re looking to manage your osquery deployment using on-premises, open-source software, we hope you will consider Fleet. Learn more about Kolide fleet <a href="https://kolide.com/fleet">on our website</a>. The source code and documentation is on GitHub at <a href="https://github.com/kolide/fleet">https://github.com/kolide/fleet</a>.</p><h4>About the Author</h4><p><a href="https://github.com/marpaia">Mike Arpaia</a> is Co-Founder &amp; CTO of <a href="https://kolide.com/"><strong>Kolide</strong></a><strong> </strong>and the creator of osquery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uuFSDRnXkDJmGaPZ.png" /><figcaption>The Kolide Cloud Desktop App gives your users visibility into the state of their device.</figcaption></figure><p><a href="https://www.kolide.com/"><strong>Kolide</strong></a> is an endpoint monitoring solution which leverages and instruments Facebook’s open-source <a href="https://github.com/facebook/osquery"><strong>osquery</strong></a> project. <a href="https://k2.kolide.com/signup">Try it today; free for the first 14 days for your entire fleet.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26e8094fab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kolide/kolide-fleet-an-open-source-osquery-fleet-manager-26e8094fab">Kolide Fleet: An open-source osquery fleet manager.</a> was originally published in <a href="https://medium.com/kolide">Kolide</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>