<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Koki - Medium]]></title>
        <description><![CDATA[A complete platform for running applications on Kubernetes - Medium]]></description>
        <link>https://medium.com/kokster?source=rss----60f3ae8e606f---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Koki - Medium</title>
            <link>https://medium.com/kokster?source=rss----60f3ae8e606f---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:06:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/kokster" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Set up a Hyperledger Fabric development environment on Kubernetes]]></title>
            <link>https://medium.com/kokster/set-up-a-hyperledger-fabric-development-environment-on-kubernetes-6428c63e018b?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/6428c63e018b</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[hyperledger]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[hyperledger-fabric]]></category>
            <dc:creator><![CDATA[Kynan Rilee]]></dc:creator>
            <pubDate>Fri, 23 Feb 2018 20:17:29 GMT</pubDate>
            <atom:updated>2018-02-23T20:17:28.666Z</atom:updated>
            <content:encoded><![CDATA[<p>Hyperledger Fabric has some rough edges when it comes to running on Kubernetes. For example, the way peers build and run chaincode isn’t the best fit for Kubernetes, but <a href="https://medium.com/kokster/simpler-setup-for-hyperledger-fabric-on-kubernetes-using-docker-in-docker-8346f70fbe80">we can use Docker-in-Docker to avoid most of the issues</a>. Likewise, there are ways around the other issues as well. Today, we’ll look at how to set up Fabric on Kubernetes so we can develop chaincode and client applications.</p><h3>Overview of Fabric setup</h3><p>Hyperledger Fabric is a modular system that can be configured with different components. Since we’re just interested in a development environment today, there are few simplifications we’ll start with:</p><ul><li>Disable TLS on Fabric components. Everything runs in a single Kubernetes cluster that we control, so we don’t need TLS.</li><li>Solo orderer service. We don’t need high availability for development.</li><li>Leave out CA and MSP services for now. Whether you need these for development depends on your use case. They’re relatively simple to set up.</li></ul><p>In summary, the development environment consists of a solo orderer and a set of peers, all configured to use a static local MSP. Once the environment is set up, we can deploy clients to set up channels, install and run chaincode, etc. Here’s an outline of the setup steps:</p><ul><li>Generate configuration artifacts for the network. e.g. Use cryptogen to generate the static MSP files. Use configtxgen to generate the orderer’s genesis block.</li><li>Upload artifacts to Kubernetes as ConfigMaps and Secrets. e.g. Create a tarball with the MSP files for a peer. Use kubectl create secret --from-file to upload the tarball.</li><li>Deploy the workload (orderer, peers, and clients) on Kubernetes.</li></ul><p>Each step is simple to automate, and it doesn’t require any changes to Fabric or to your Kubernetes cluster. If you don’t have a Kubernetes cluster, creating one on your own machine is as simple as downloading <a href="https://github.com/kubernetes/minikube">Minikube</a> and running minikube start.</p><h3>Generating configuration artifacts</h3><p>If you’ve tried fabric-samples before, you’re probably familiar with cryptogen and configtxgen. The <a href="http://hyperledger-fabric.readthedocs.io/en/latest/build_network.html#run-the-tools">official docs for this step</a> are pretty straightforward.</p><ul><li>Write a crypto-config.yaml for the network. Use it (and cryptogen) to create crypto-config for all components of the network, including the orderer, peers, and users (for clients).</li><li>Write an orderer.yaml and configtx.yaml for the network. Use them (and configtxgen) to create a genesis block for the orderer.</li></ul><h3>Uploading configuration artifacts</h3><p>When you run a Fabric network on your own machine using docker-compose, you can directly mount local directories into the orderer and peer containers. Look at the volumes fields in fabric-samples’ <a href="https://github.com/hyperledger/fabric-samples/blob/02ca1dcf23f746338014d77b862a8b85996e1d8e/first-network/base/docker-compose-base.yaml">docker-compose-base.yaml</a>:</p><pre>volumes:<br>- <strong>../channel-artifacts/genesis.block</strong>:/var/.../orderer.genesis.block<br>- <strong>../crypto-config/.../msp</strong>:/var/hyperledger/orderer/msp<br>- <strong>../crypto-config/.../tls</strong>:/var/hyperledger/orderer/tls</pre><p>The part in boldface is the local path on your machine. The part on the right is the mount path — where the orderer container sees the files. This approach doesn’t work if you’re running Fabric on a Kubernetes cluster because Kubernetes can’t access your local filesystem.</p><p>Fortunately, the Kubernetes API has another solution for configuration artifacts:</p><ul><li>ConfigMap</li><li>Secret</li></ul><p>Both resources can store arbitrary data, and they can be injected into a container as either environment variables or files. Here’s how to upload the orderer genesis block to a secret named orderer-genesis-block:</p><pre>kubectl create secret generic <em>orderer-genesis-block</em> --from-file=<strong>../channel-artifacts/genesis.block</strong></pre><p>This command uploads a generic-type Secret with a single entry, genesis.block. Later, when we deploy the orderer, we can mount the Secret orderer-genesis-block into the orderer’s filesystem.</p><h4>Uploading crypto-config</h4><p>The process is a little more involved for the MSP artifacts, because cryptogen creates an entire directory structure. There are a few more steps, but the general idea is the same:</p><ul><li>tar -zcf orderer.example.com.msp.tar.gz -C ../crypto-config/.../orderer.example.com msp</li><li>kubectl create secret generic ...</li><li>Mount the Secret into the orderer Pod.</li><li>Use an InitContainer to unpack orderer.example.com.msp.tar.gz so the orderer container can use the MSP artifacts.</li></ul><p>The main difference here is that the orderer container can’t directly use the file stored in the Secret, so an InitContainer has to unpack the tarball first. Another approach is to create a composite Secret that contains multiple files rather than a single tarball — then an InitContainer might not be needed.</p><h3><strong>Deploying the workload</strong></h3><p>Deploying the orderer and peers to Kubernetes is primarily a matter of making sure all resources (Services, Deployments, Secrets, ConfigMaps) are configured correctly. The basic architecture looks like this:</p><ul><li>Kubernetes <em>Service</em> and <em>Deployment</em> for the orderer</li><li>Kubernetes <em>Service</em> and <em>Deployment</em> for each peer</li><li>Kubernetes <em>Secrets</em> and <em>ConfigMaps</em> for Fabric configuration artifacts</li></ul><p>Each <em>Deployment</em> is the actual running instance(s) of a service. Each <em>Service</em> is a cluster-internal DNS name for the underlying <em>Deployment</em>. For example, when a peer needs to connect to the orderer, it uses the orderer’s <em>Service</em> name. Likewise, when a client needs to connect to a peer, it uses the peer’s <em>Service</em> name.</p><p>There are a lot of things to get right in the Kubernetes config for our Fabric development environment. Here’s a checklist of concerns:</p><ul><li>Environment variables are set for orderer, peer, and client containers.</li><li>Docker-in-Docker sidecar containers are set up for peer and client containers.</li><li>The correct Secrets and ConfigMaps are attached to the orderer, peer, and client Deployments.</li><li>InitContainers unpack tarballed secrets and configs and initialize the right filesystem structure for the orderer, peers, and clients.</li><li>Orderer and peer Service names match the orderer, peer, and client containers’ environment variables.</li></ul><p>Once you have the right Service and Deployment configurations, deploying them is a simple kubectl create -f.</p><h3>Conclusion</h3><p>Running Hyperledger Fabric on Kubernetes boils down to making sure you’re deploying the right resource configurations. There are a lot of details to figure out the first time, but afterwards, the process is very simple. Good luck, and as always, leave your thoughts and questions in the comments section!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*3n5c7n-B_2dDdj0o8TiS4g.png" /><figcaption>Check out the different Hyperledger projects at <a href="https://www.hyperledger.org/">https://www.hyperledger.org/</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6428c63e018b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/set-up-a-hyperledger-fabric-development-environment-on-kubernetes-6428c63e018b">Set up a Hyperledger Fabric development environment on Kubernetes</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simpler setup for Hyperledger Fabric on Kubernetes using Docker-in-Docker]]></title>
            <link>https://medium.com/kokster/simpler-setup-for-hyperledger-fabric-on-kubernetes-using-docker-in-docker-8346f70fbe80?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/8346f70fbe80</guid>
            <category><![CDATA[hyperledger]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[hyperledger-fabric]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Kynan Rilee]]></dc:creator>
            <pubDate>Thu, 22 Feb 2018 20:12:15 GMT</pubDate>
            <atom:updated>2018-02-23T04:04:26.773Z</atom:updated>
            <content:encoded><![CDATA[<p>Hyperledger Fabric is a distributed blockchain network that allows users to define the behavior of their ledgers using conventional general-purpose programming languages. This user-defined blockchain code is called <em>chaincode</em>. It’s executed by peers to determine what effect a transaction has on the state of the ledger. Currently, Fabric supports chaincode written in Go and Node.js, but the goal is to support other languages as well.</p><h3>How Peers run Chaincode</h3><p>Chaincode is distributed as source code. It’s up to each peer to build and run it. First, the peer uses a builder image for the specified programming language (e.g. Golang) to create an executable image from the chaincode source:</p><ul><li>Make sure the builder image is available to the Docker daemon. Pull the image from Docker hub if necessary.</li><li>Tell the Docker daemon to start a container from the builder image and send the chaincode source code to the new builder container.</li><li>The builder container creates a new image for the built chaincode.</li></ul><p>The built chaincode image is not uploaded to a remote registry, but it is available locally to the Docker daemon that started the builder container. This is fine as long as the peer uses the same Docker daemon to run the builder container and the chaincode container.</p><p>Once the chaincode image is built, the peer can use it to execute the chaincode:</p><ul><li>Tell the Docker daemon to start a container from the chaincode image. It must be the same Docker daemon as before, otherwise the chaincode image won’t be available.</li><li>The chaincode container uses a provided IP address (e.g. localhost) or DNS name (e.g. peer0.org1) to connect to the peer.</li></ul><p>The current implementation of Fabric peers uses the environment variable CORE_PEER_CHAINCODELISTENADDRESS to configure the address chaincode connects to. It’s simplest to use a predetermined value like localhost or the DNS name of the peer, but it may be possible to dynamically determine the correct IP after the peer is deployed.</p><h3>Trouble with Chaincode in Kubernetes</h3><p>The requirements I described above aren’t well-suited to Kubernetes. Kubernetes is a container orchestration framework. At the most basic level, it’s a system for managing Pods — groups of containers that each share a network namespace. As such, Kubernetes expects applications to run their code in Pods. This is the first obstacle for running chaincode on Kubernetes.</p><h4>Running Chaincode using Host Docker</h4><p>The current standard is to mount the host machine’s Docker socket into the peer’s container. The peer then uses the host’s Docker daemon to build and run chaincode. This strategy works, but it has flaws.</p><p>Since the builder container and the chaincode container are created without going through the Kubernetes API, Kubernetes cannot manage them. For example, if the peer crashes or must be rescheduled onto a different machine, Kubernetes won’t clean up the chaincode container.</p><p>Another problem is that each Kubernetes Pod gets its own IP and network namespace. When the chaincode container is created using the host’s Docker daemon, it’s not part of the Pod. This means the chaincode can’t connect to the peer using localhost. This wouldn’t be so bad, but we can’t use DNS names here either. Internally, Kubernetes uses its own DNS, called kube-dns. Pods are set up to use kube-dns, but the chaincode isn’t part of a Pod — it won’t be able to resolve the IP address for peer0.org1.</p><h3>Docker-in-Docker</h3><p>Docker-in-Docker (DinD) is a Docker daemon running inside a Docker container. The DinD daemon is separate from the host’s Docker daemon. Containers created by one daemon are not visible to the other. We can use DinD in a Pod to solve the problems of running chaincode on Kubernetes.</p><h4>No more orphaned chaincode</h4><p>DinD runs as a <em>sidecar</em> in the peer’s Pod. This means it’s a container that runs alongside the main application (the peer) in the Pod. In addition to sharing a network namespace, containers in a Pod also share the same lifetime. If one container fails, the entire Pod fails. If the peer crashes, DinD and the chaincode containers get cleaned up too.</p><h4>Chaincode connects to Peer at localhost</h4><p>Now, the chaincode container is started by a Docker daemon <em>inside</em> the Pod’s network namespace. This means the DinD daemon, the chaincode container, and the peer all have the same IP address. The peer can then be configured with CORE_PEER_CHAINCODELISTENADDRESS as localhost:7052. When the chaincode container starts, it doesn’t need kube-dns to find the peer at localhost:7052.</p><h3>Comparison to Other Solutions</h3><p>The current standard for Fabric on Kubernetes is to mount the host’s Docker daemon into the peer Pod. In order for the chaincode container to find the peer, it needs to use the peer’s DNS name. This means the host’s Docker must be configured to use kube-dns. This isn’t a standard step in setting up a Kubernetes cluster, so users with existing clusters have to reconfigure all their worker nodes. It’s not a very Kubernetes-native approach, and it doesn’t address the issue of cleaning up orphaned chaincode containers.</p><p>Another approach being investigated is to run chaincode as a Pod. This way, Kubernetes can properly manage the chaincode — scheduling based on resource requirements, setting up DNS, etc. It’s much more complex than the DinD approach, but it could be the right approach at some point in the future.</p><h3>Conclusion</h3><p>For now at least, using Docker-in-Docker as a sidecar to the peer solves the main problems with running chaincode in Hyperledger Fabric on Kubernetes. It’s a drop-in solution that doesn’t require changes to Fabric or the user’s Kubernetes cluster.</p><p>As a final note, it makes sense that Docker-in-Docker is useful for building and running chaincode. Building and running arbitrary code is essentially what Continuous Integration does, and CI is a popular use case for DinD. Leave your thoughts and questions in the comments!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*3n5c7n-B_2dDdj0o8TiS4g.png" /><figcaption>Check out the different Hyperledger projects at <a href="https://www.hyperledger.org/">https://www.hyperledger.org/</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8346f70fbe80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/simpler-setup-for-hyperledger-fabric-on-kubernetes-using-docker-in-docker-8346f70fbe80">Simpler setup for Hyperledger Fabric on Kubernetes using Docker-in-Docker</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Hyperledger Sawtooth — Proof of Elapsed Time]]></title>
            <link>https://medium.com/kokster/understanding-hyperledger-sawtooth-proof-of-elapsed-time-e0c303577ec1?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/e0c303577ec1</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[hyperledger]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[intel]]></category>
            <dc:creator><![CDATA[Kynan Rilee]]></dc:creator>
            <pubDate>Tue, 20 Feb 2018 20:41:49 GMT</pubDate>
            <atom:updated>2018-02-20T20:41:49.371Z</atom:updated>
            <content:encoded><![CDATA[<p>Sawtooth, like Fabric, is a permissioned blockchain network technology. The network is called <em>permissioned</em> because prospective participants must identify themselves to the network, and the network can decide whether to let them participate. Once in the network, participants share a view of the blockchain ledger. The network uses a <em>consensus algorithm</em> to make sure all participants see identical ledgers.</p><p>Consensus algorithms come in different forms. For example, Bitcoin uses a <em>proof-of-work</em> system. Members of the Bitcoin network compete to solve a cryptographic puzzle that identifies the solver as the new <em>leader</em> — the authoritative party for the new block on the blockchain. It’s essentially a lottery for choosing who is responsible for adding the new block. <em>Proof-of-stake</em> is another lottery system, except the lottery is based on ownership of coin in the system. The more coin a participant owns, the more likely they are to be chosen as the leader for a new block.</p><p>In addition to lottery-like systems, there are also other kinds of consensus algorithms. For example, permissioned blockchain networks like Sawtooth and Fabric can also use traditional Byzantine fault tolerance (BFT) algorithms, which are often based on <em>voting</em>.</p><p>Sawtooth and Fabric are both designed to allow users to choose the consensus algorithm for their blockchain network. Today, we’ll look at a lottery-like algorithm used with Hyperledger Sawtooth called <em>proof-of-elapsed-time</em>.</p><h3>Proof of Elapsed Time (PoET)</h3><p>At a high level, proof-of-elapsed-time follows this strategy:</p><ul><li>Each participant in the blockchain network waits a random amount of time.</li><li>The first participant to finish waiting gets to be leader for the new block.</li></ul><p>In order for this to work, two requirements must be verified. First, did the lottery winner actually choose a random wait time? Otherwise, a participant could intentionally choose a short wait time in order to win. Second, did the lottery winner actually finish waiting the specified amount of time?</p><p>PoET comes from Intel, and it relies on a special CPU instruction set called Intel Software Guard Extensions (SGX). SGX allows applications to run <em>trusted code</em> in a protected environment. For PoET, the trusted code is what ensures that these two requirements are satisfied — keeping the lottery fair.</p><h4>Intel SGX</h4><p>Before going into what PoET’s trusted code does, let’s understand some key points about SGX.</p><ul><li>A specialized hardware component can create an <em>attestation</em> that some particular trusted code was set up correctly in a protected environment. e.g. An external party can use the <em>attestation</em> to verify that the right code is running the right way.</li><li>Trusted code runs in an environment that is private to the rest of the application. e.g. The rest of the application can’t inspect or interfere with the memory space of the trusted code.</li></ul><p>The first point allows a network participant to prove to other participants that it is running the right trusted code for the network. Without this feature, there’s no way for the network to know whether a participant is actually running the PoET trusted code.</p><p>The second point ensures that a malicious participant can’t cheat by manipulating the PoET trusted code after it’s been set up.</p><h4>PoET’s trusted code</h4><p>The details of the protocol are somewhat complex, but here’s a simplified outline:</p><p><strong>Joining the network</strong></p><ul><li>A new participant downloads the trusted code for the blockchain.</li><li>On initialization, the trusted code creates a new keypair.</li><li>Participant sends a SGX attestation (which includes the trusted code’s public key) to the rest of the network as part of a <em>join</em> request.</li></ul><p><strong>Participating in the lottery</strong></p><ul><li>Participant obtains a signed <em>timer</em> object from the trusted code.</li><li>Participant waits for the time specified by the timer object.</li><li>Participant obtains a certificate (signed by the trusted code’s private key) that the timer has completed. Participant sends this certificate to the rest of the network along with the new block for the blockchain.</li></ul><p>The protocol also involves other protections on top of SGX. For example, the network measures how often a given participant wins the lottery in order to detect participants with a compromised SGX system. Bad actors can then be blacklisted.</p><h3>Conclusion</h3><p>Hyperledger Sawtooth can use a <em>proof-of-elapsed-time</em> (PoET) consensus algorithm that leverages Intel’s Software Guard Extensions (SGX) to implement a leader-election lottery system. Instead of using computational effort to solve cryptographic puzzles (like Bitcoin), the system uses a trusted execution environment to generate random wait times—potentially a much more energy-efficient approach.</p><p>To learn more about Sawtooth, including additional details about the PoET protocol, check out the <a href="https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html">project’s docs</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*3n5c7n-B_2dDdj0o8TiS4g.png" /><figcaption>Check out the different Hyperledger projects at <a href="https://www.hyperledger.org/">https://www.hyperledger.org/</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0c303577ec1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/understanding-hyperledger-sawtooth-proof-of-elapsed-time-e0c303577ec1">Understanding Hyperledger Sawtooth — Proof of Elapsed Time</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Corda—Transactions vs Blocks]]></title>
            <link>https://medium.com/kokster/understanding-corda-transactions-vs-blocks-1f3d42d00cc2?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/1f3d42d00cc2</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[hyperledger]]></category>
            <category><![CDATA[corda]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[hyperledger-fabric]]></category>
            <dc:creator><![CDATA[Kynan Rilee]]></dc:creator>
            <pubDate>Fri, 16 Feb 2018 22:02:07 GMT</pubDate>
            <atom:updated>2018-02-16T22:02:06.620Z</atom:updated>
            <content:encoded><![CDATA[<p>Corda is a blockchain-inspired distributed ledger technology that currently targets finance use cases. Like Hyperledger Fabric, it’s a permissioned network where all participants have verifiable identities — using public-key infrastructure. However, unlike other distributed ledgers, Corda does not use a blockchain to record transactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*23wQfPvUoL1sUDRl4XVSmg.jpeg" /><figcaption>Corda breaks apart the blocks and the blockchain. (<a href="https://commons.wikimedia.org/wiki/File:Lego_dublo_arto_alanenpaa_5.JPG">image source</a>)</figcaption></figure><h3>Why doesn’t Corda use a blockchain?</h3><p>Blockchain networks replicate a ledger across every member of the network. In Bitcoin, the ledger is shared across the entire world. In Hyperledger Fabric, the ledger is shared across every organization in a Fabric channel.</p><p>Corda views this approach as an unnecessary dilution of privacy. The only entities that should be aware of a transaction are the parties directly involved. If a bank issues a transaction to say it owes a customer <em>n</em> dollars, only the bank, the customer, and relevant regulatory organizations need to know anything about the transaction.</p><h3>Direct Communication vs Broadcast</h3><p>If you want to make sure only the involved parties know about a transaction, the first step is to control who the transaction is sent to. In general, blockchain networks heavily use <em>broadcast</em>, where messages are distributed to every node in the network. This makes sense for anonymous networks like Bitcoin. If you don’t know who anyone is, the only way to get a message to a specific someone is to send that message to everyone.</p><p>A Corda network includes a <em>network map</em> service that publishes information about how to reach every identity in the network. This allows any entity to specifically contact any other entity — directly.</p><blockquote>Public identities and direct communication have wide implications for the architecture of a distributed ledger network.</blockquote><p>Anonymous networks use <em>proof-of-work</em> to prevent Sybil attacks, where a single entity creates a large number of users to influence the network. Proof-of-work networks are influenced by computational effort, not number of users. A single entity can still influence the network, but it’s <em>expensive</em>.</p><p>Permissioned networks like Corda and Hyperledger Fabric can skip proof-of-work because the permissioning process can tie in-network identities to real-world identities. There’s much less threat of one entity maliciously flooding the network with fake identities.</p><h4>No more blocks?</h4><p>Corda goes a step further to say that transactions no longer need to be batched together into blocks. One reason blockchains batch transactions together into blocks is that it takes time to distribute a block across the entire network. In that time, many new transactions can occur. Especially in a proof-of-work system where every node is trying to calculate the <em>next</em> block, it’s important that the network have a uniform idea of what the <em>last</em> block was. This means that blocks need to be created slowly enough that each has already been distributed across the network before the next one is created.</p><p>Corda uses direct communication, so it doesn’t have to account for the time it takes to broadcast a message to the entire network. Individual transactions are sent directly to the involved parties, a very quick process. However, because transactions aren’t broadcast to the entire network:</p><blockquote>No entity has an entire history of all the transactions in the network.</blockquote><p>So how does Corda collect enough knowledge in one place to know if a new transaction is valid? How can you tell what the current state is?</p><h3>How do Corda’s transactions work?</h3><p>Corda transactions operate on the notion of <em>consumable states</em>. For example, a state might be <em>“Entity X is the rightful owner of Asset A”</em>. Entity X can <em>spend</em> this state to perform a transaction, which creates a new state <em>“Entity Y is the rightful owner of Asset A”</em>. Note that after the input state is spent, it is no longer valid. After the transaction, Entity X no longer owns Asset A.</p><p>These consumable states are analogous to the world state of a blockchain ledger. Only the latest world state can be used to validate a new transaction on the blockchain. Once a transaction is committed, the old world state will never be used again — it is replaced by a new world state.</p><p>What Corda does is this: It separates the world state into granular components that, instead of being globally visible, are only held by involved parties.</p><p><em>(Side note: Hyperledger Fabric does something similar, though for a different purpose. Fabric splits world state into </em>read-sets<em> and </em>write-sets<em> so it can execute transactions in parallel and apply the results afterward.)</em></p><p>Corda’s approach introduces a new problem. If I’m involved in a transaction that consumes a state I don’t currently know about, how do I know that the transaction is valid? How do I know that Entity X still owns the Asset A that is being transferred to me?</p><p>This is the job of Corda’s <em>notaries</em>. A notary is a service that knows the current state of a chunk of the world. At a high level, creating a new transaction looks like this:</p><ul><li>Suppose a new <strong>transaction T</strong> spends a <strong>state X</strong> to create a <strong>state Y</strong>.</li><li>Send transaction T to the notary that knows about state X.</li><li>The notary checks that state X is still valid and signs transaction T. State X is now invalid, with state Y taking its place.</li></ul><p>If state X above were not valid when the transaction was proposed, the notary would respond that state X had already been spent. The transaction would not occur.</p><p>Corda has shifted the responsibilities of <strong>knowing the world state</strong> and <strong>seeing all transactions</strong> from the entire network onto a special <em>notary</em> service. Members of the network don’t need to trust everyone, just the organizations that run the notary. There can also be multiple notaries, each trusted for a different purpose.</p><h3>Conclusion</h3><p>Because Corda is a permissioned network with public identities, Corda nodes are able to communicate directly with one another instead of broadcasting messages to the entire network. This allows Corda to split knowledge and responsibilities across the network at a more granular level than possible with other distributed ledgers.</p><p>Follow us for more information about blockchain and related technologies, and leave your thoughts in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f3d42d00cc2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/understanding-corda-transactions-vs-blocks-1f3d42d00cc2">Understanding Corda—Transactions vs Blocks</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Hyperledger Fabric — Gossip]]></title>
            <link>https://medium.com/kokster/understanding-hyperledger-fabric-gossip-512a217d5d1e?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/512a217d5d1e</guid>
            <category><![CDATA[hyperledger-fabric]]></category>
            <category><![CDATA[hyperledger]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Kynan Rilee]]></dc:creator>
            <pubDate>Thu, 15 Feb 2018 20:04:17 GMT</pubDate>
            <atom:updated>2018-02-20T03:17:25.584Z</atom:updated>
            <content:encoded><![CDATA[<p>Hyperledger Fabric is a distributed blockchain network where multiple components each keep their own copy of a shared <em>ledger</em>, the definitive history of all transactions. As new transactions occur, they must be distributed across the network, keeping all copies of the ledger in sync.</p><p>At a high level, the process looks like this:</p><ul><li>New transactions are submitted to the orderer (a.k.a. <em>ordering service</em>).</li><li>The orderer creates a new block (containing the new transactions) on the blockchain.</li><li>The orderer delivers the block to all peers.</li></ul><p>The last step, delivery, is where <em>gossip</em> comes in. Even in a blockchain network with relatively few member organizations, there could still be a large number of peers. For example, a network that makes heavy use of IoT (Internet of Things) devices might need many peers to connect these devices to the network. Furthermore, the wider network of peers may have intermittent connectivity — some peers might not always be reachable from the orderer. It’s unnecessary stress for the orderer to have to deliver to every peer, and it may not even be possible in some situations.</p><p>What actually happens in Hyperledger Fabric is that the orderer only delivers new blocks to a single peer in each organization — the <em>leader</em> peer. Through a process called <em>gossip</em>, the peers themselves do the job of spreading the word to each other:</p><ul><li>A peer has a message that needs to be distributed to all other peers. For example, the leader peer has a new block for the blockchain.</li><li>This peer sends the message to a predetermined number of (randomly selected) other peers.</li><li>Each of those peers forwards the message to a predetermined number of other peers.</li><li>And so on, until every peer has received the message.</li></ul><p>The process above is called <em>broadcast</em>, and it’s used by Fabric’s gossip system to distribute multiple kinds of messages to all peers.</p><h3>Membership Gossip</h3><p>A key component of the gossip protocol is that each peer forwards messages to a random selection of the current peers in the network. This implies that each peer knows which peers are in the network so it can choose among them.</p><p>In Fabric, each peer periodically <em>broadcasts</em> a message message that indicates it’s still alive and connected to the network. Every peer maintains its own list of all the peers in the network — those that are “alive”, and those that are “dead”.</p><ul><li>When Peer A receives an <strong>“alive”</strong> message from Peer B, it marks Peer B as <strong>“alive”</strong>. According to Peer A, Peer B is part of the network.</li><li>If after a while, Peer A hasn’t received an <strong>“alive”</strong> message from Peer B, it marks Peer B as <strong>“dead”</strong>. According to Peer A, Peer B is no longer part of the network.</li></ul><p>Just in case Peer B isn’t actually dead, Peer A periodically attempts to connect to dead peers to see if they are actually alive. For example, it’s possible that network issues or failures on other peers prevented Peer B’s “alive” message from reaching Peer A. When Peer A contacts Peer B directly, it can see that Peer B is indeed still alive.</p><h4>Bootstrapping</h4><p>The process described above works only if a peer has a list of other peers to send “alive” messages to. So each peer is bootstrapped into the network with a <em>bootstrap set </em>of peers to start with.</p><h4>Byzantine Fault Tolerance</h4><p>Each peer signs its “alive” messages, which means a bad actor can’t trick the rest of the network by <em>creating</em> messages. All it can do is not forward a peer’s “alive” message. This isn’t much of a threat as long as other peers still forward the “alive” message.</p><h3>Data Dissemination Gossip</h3><p>As described earlier, the basics of data dissemination are simple. Each peer forwards new data to a randomly-selected subset of the peers it knows about. This process is called <em>broadcast</em>, and it’s a <em>push</em>-based way to move information through the network.</p><p>However, if a peer became disconnected from the network and reconnects later, it would’ve missed the <em>broadcast</em> process. In order to catch up with the rest of the network, it needs a <em>pull</em>-based mechanism to request the data it’s missing. In Hyperledger Fabric, peers periodically exchange both membership data (the list of peers, alive and dead) and ledger data (transaction blocks) with each other. This allows peers to stay up-to-date even if they miss the broadcast of an “alive” message or a new transaction block.</p><h3>Conclusion</h3><p>Hyperledger Fabric uses gossip between peers as a fault-tolerant and scalable mechanism for keeping all copies of the blockchain ledger in sync. It reduces the load on the orderer, which only needs to deliver blocks to a single peer per organization, and it allows peers to “catch up” to the current state after being disconnected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*3n5c7n-B_2dDdj0o8TiS4g.png" /><figcaption>Check out the different Hyperledger projects at <a href="https://www.hyperledger.org/">https://www.hyperledger.org/</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=512a217d5d1e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/understanding-hyperledger-fabric-gossip-512a217d5d1e">Understanding Hyperledger Fabric — Gossip</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Hyperledger Fabric — Byzantine Fault Tolerance]]></title>
            <link>https://medium.com/kokster/understanding-hyperledger-fabric-byzantine-fault-tolerance-cf106146ef43?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf106146ef43</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[hyperledger]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[hyperledger-fabric]]></category>
            <dc:creator><![CDATA[Kynan Rilee]]></dc:creator>
            <pubDate>Thu, 15 Feb 2018 04:10:11 GMT</pubDate>
            <atom:updated>2018-02-15T04:10:10.881Z</atom:updated>
            <content:encoded><![CDATA[<p>The architecture of Hyperledger Fabric factors out <em>consensus</em> into its own component — the orderer service. This is an unusual feature because it means Fabric can support different consensus algorithms by just switching out the orderer implementation. Not every blockchain platform supports this.</p><p>For example, Bitcoin’s consensus algorithm is baked in. It makes sense for Bitcoin to only support its proof-of-work system. Bitcoin expects to be a massive untrusted network, and its consensus algorithm works great for that scenario. On the other hand, Hyperledger Fabric is a generic platform — meant to be applied in a wide variety of environments. There isn’t a one-size-fits-all consensus algorithm, so Fabric is designed to let users choose.</p><h3>What is Consensus?</h3><p>Consensus means all parties agree on a particular decision. In the case of a blockchain network, members of the network arrive at a <em>consensus</em> on the contents of the blockchain. It’s telling that consensus is part of the <em>ordering</em> step of Fabric’s <em>execute-order-validate</em> architecture. The <strong>orderer</strong> determines which transactions to add to the blockchain and in what order. All members of the network rely on what the orderer says.</p><p>In a distributed system, there’s a variety of threats that a consensus algorithm must be resilient against. Processes can crash. Machines and devices can fail. Network connections can be interrupted. Furthermore, since components of the system may be controlled by different organizations with different goals, they may act maliciously as well. <em>Crash fault tolerance</em> (CFT) is one level of resiliency, where the system can still correctly reach consensus if components fail. Byzantine fault tolerance (BFT) is more complex and deals with systems that may have malicious actors.</p><h3>The Byzantine Generals Problem</h3><p>The name <em>Byzantine fault tolerance</em> is derived from a <a href="http://lamport.azurewebsites.net/pubs/byz.pdf">1982 paper called <em>The Byzantine Generals Problem</em></a>. (excerpt below)</p><blockquote>We imagine that several divisions of the Byzantine army are camped outside an enemy city, each division commanded by its own general. The generals can communicate with one another only by messenger. After observing the enemy, they must decide upon a common plan of action.</blockquote><p>This analogy fits distributed computation because individual machines (generals) are connected only via the network (messengers). Each is only aware of what it observes and the messages it receives. Likewise, every component is a possible source of failure or malicious activity.</p><ul><li>Networks can fail — messengers can be captured or killed.</li><li>Man-in-the-middle attacks can send forged messages — messengers can be compromised.</li><li>Hardware or software components can break or crash — generals can be killed.</li><li>Malicious components can send malicious messages — generals can be traitorous.</li></ul><p>The analogy also fits the topic of consensus because the generals must act together. The army only succeeds if its divisions act together. Either all divisions attack or all divisions retreat. If only some divisions attack, the city can defeat them, and the retreating divisions will be too weak to resist being chased down. In a blockchain, consensus is also paramount. The power of the blockchain is that it is a shared source of truth. If different parts of the network have different blockchain states, they can no longer work together — they’re operating in different worlds.</p><h4>Byzantine Fault Tolerance</h4><p>The goal of the Byzantine generals is for every (non-traitorous) general to agree on the same decision. In a blockchain network, this means every non-malicious entity has the same blockchain state. The implication for Hyperledger Fabric is that the orderer service should be jointly controlled by the network’s members using a BFT algorithm that resists malicious activities by bad actors. It’s insufficient for one organization to control the orderer, because that organization itself may not be trustworthy. After all, one of the motivations to use blockchain is so that organizations can cooperate while only partially trusting one another.</p><h3>The Big Picture</h3><p>The Hyperledger Fabric architecture lets users choose an orderer service that implements a consensus algorithm that fits their application. One desirable property is Byzantine fault tolerance (BFT), which says the orderer can do its job even in the presence of malicious actors.</p><p>It’s important to note that BFT, however it’s implemented, only applies to the <em>ordering</em> of transactions in Hyperledger Fabric. Its job is to ensure that every peer has the same list of transactions on its ledger.</p><p>In order to get on the ledger in the first place, the transactions first have to get past endorsement policies and the endorsing peers that ran the transactions’ chaincode. (More details in a <a href="https://medium.com/kokster/hyperledger-fabric-endorsing-transactions-3c1b7251a709">previous post</a>.) Then after they’re on the ledger, the transactions are checked a final time by each peer’s <em>validation</em> step. Every step of the way, Fabric has a system in place to prevent bad things from happening.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*3n5c7n-B_2dDdj0o8TiS4g.png" /><figcaption>Check out the different Hyperledger projects at <a href="https://www.hyperledger.org/">https://www.hyperledger.org/</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf106146ef43" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/understanding-hyperledger-fabric-byzantine-fault-tolerance-cf106146ef43">Understanding Hyperledger Fabric — Byzantine Fault Tolerance</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fault Tolerant Redis Architecture]]></title>
            <link>https://medium.com/kokster/fault-tolerant-redis-architecture-35334ef5de0b?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/35334ef5de0b</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sidhartha Mani]]></dc:creator>
            <pubDate>Tue, 13 Feb 2018 23:53:08 GMT</pubDate>
            <atom:updated>2018-02-13T23:53:08.358Z</atom:updated>
            <content:encoded><![CDATA[<p>Redis is an open-source, in-memory data-structure store used as a cache, message broker and database. It can be configured to run in a highly available fashion. High availability can be achieved by removing or reducing the probability of service disruption.</p><p>However, it is impossible to prevent each and every kind of service disruption. When a disruption does occurs, the ability of the system to continue functioning is defined as fault tolerance.</p><blockquote>The ability of a system to continue functioning in the face of failures is known as fault tolerance.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*_Us9pdQs2j8G-kJ-.jpg" /><figcaption>A starfish can survive even if it loses an arm.</figcaption></figure><h4>Fault Tolerance</h4><p>Reasoning about fault tolerance can be done by assessing the surface area of damage in case of failures, which can be mitigated by spanning across fault domains.</p><p>In order to asses the surface area of damages in case of failures, the different types of failures need to be understood. The various failure scenarios include:</p><ul><li>Node failures — Disk crashes, overheating, or accidentally deleting a node on which the service is running can lead to failures</li><li>Network failures — Network device failure, channel overload, or misconfiguration can lead to service disruption</li><li>Application-specific failures — In case of redis, it is possible for a client to try to write to a stale master after a failure scenario where some other instance has taken over as master. In this case, data loss can occur.</li></ul><h4>Tolerating Node and Network failures</h4><p>Running a service on one node, writing data into one network disk, or isolating all Redis instances to one availability zone can all lead to complete service disruption. This is because the fault domain spans across the whole service.</p><p>The best mechanism to survive such failures is to reduce the surface area of damage by spanning across multiple fault domains —</p><ul><li>Run multiple masters spread out across multiple nodes. (A node is one fault domain)</li><li>Nodes should span across availability zones. (Availability zone is one fault domain)</li><li>Less performance-sensitive workloads, such as data warehousing and analytics can even be spread out across regions. (Region is one fault domain)</li></ul><h4>Redis Sentinel</h4><p>Redis Sentinel is the primary mechanism to setup and run Redis in a fault tolerant manner. Redis Sentinel is a distributed service that continuously checks for the health of masters and slaves running in the cluster, and performs a fail-over if needed.</p><p>If it detects a failure or is unable to reach a particular instance, it tries to verify with other sentinels if that instance is actually down.</p><p>When the Sentinel first detects that a master is down, it designates the master a new state known as S_DOWN. This state says that the current master is Subjectively Down.</p><p>Then, it communicates with other sentinels, and if a <em>preconfigured</em> number of sentinels agree that the master is down, then the state is updated to O_DOWN, or Objectively Down. Then it proceeds to “vote” for the new master. The voting process requires a <strong>majority</strong> of the sentinels to agree on the new master.</p><blockquote>The process of electing a new master to take over the tasks of a failed master is known as failover.</blockquote><p>There are important considerations to make when setting up Sentinel for Fault Tolerance —</p><ul><li>The Redis client should support Sentinel.</li><li>The architecture of the Sentinel-based cluster should promote robustness. It is possible to configure the cluster in such a way that sentinels cannot function effectively</li><li>The sentinel itself should be setup in a highly available and fault tolerant manner.</li></ul><h4>Fault Tolerant Architecture — I</h4><p>Consider this setup of Redis Master and Sentinel. The larger box is one node. The smaller box can be considered one container.</p><pre>+------------+<br>|   +---+    |                   M  - Master<br>|   |M1 |    |                   S  - Sentinel<br>|   +---+    |                   SL - Slave<br>|   |S1 |    |<br>+---+---+----+</pre><p>If the master process dies, the sentinel will detect that the process is dead. It will know that the master is down. However, it won’t be able to bring the service back up. <strong><em>This setup is not fault tolerant.</em></strong></p><h4>Fault Tolerant Architecture — II</h4><pre>+------------+  <br>|   +---+    |                   M  - Master<br>|   |M1 |    |                   S  - Sentinel<br>|   +---+    |                   SL - Slave<br>|   |S1 |    |<br>+---+-|-+----+<br>      |<br>+-----|------+  <br>|   +-|-+    |                   <br>|   |SL1|    |                   <br>|   +---+    |                   <br>|   |S2 |    |<br>+---+---+----+</pre><p>There setup consists of two instances — one master and one slave. If the master process dies, then the sentinels can detect it and promote the slave. This setup can tolerate master process failure.</p><p>However, if the node running master dies, then both M1 an S1 will die. In this case, the remaining sentinel will not be able to perform a failover, since it requires a majority of the total sentinels to agree on the new master. <strong><em>This setup will not tolerate master node failure.</em></strong></p><h4>Fault Tolerant Architecture — III</h4><pre>+------------+  <br>|   +---+    |                   M  - Master<br>|   |M1 |    |                   S  - Sentinel<br>|   +---+    |                   SL - Slave<br>|   |S1 |    |<br>+---+-|-+----+<br>      |<br>+-----|------+            +-----|------+<br>|   +-|-+    |            |   +-|-+    |       <br>|   |SL1|    |            |   |SL2|    |   <br>|   +---+----+------------+---+---+    |     <br>|   |S2 |    |            |   |S3 |    |<br>+---+---+----+            +---+---+----+</pre><p>This setup consists of three instances — one master and two slaves instances. Each of the instances are run on different nodes. If the master process dies, or if the master node dies, the majority vote can be done, and one of the slaves can be promoted to be the new master. This setup is tolerant to master process failure, and master node failure.</p><blockquote>This is the minimally fault tolerant Redis Setup that should be considered for production.</blockquote><p>In this setup, if one master and any one of the slaves fail, then the consensus cannot be reached to promote the left over slave to master. <strong><em>This setup cannot tolerate the failure of one master and one slave node.</em></strong></p><h4>Beyond bare-minimum fault tolerance</h4><p>The above architectures serve as reference architectures for fault tolerance and reasoning about them. The other important fault tolerance mechanisms are:</p><ul><li>Restart processes that die unexpectedly</li><li>Do not couple the life-cycles of two components.</li></ul><p>If any of the instances (process) dies for any reason, using a system like Docker or Kubernetes to automatically restart those processes will ensure that this architecture is always satisfied. Otherwise, this architecture might be honored during setup, but not after the failure of one of the processes.</p><p>In the fault-tolerant architecture III presented above, if any one of the sentinels were run on separate node, then the lifecycle of the sentinel and the lifecycle of the master will not be tied to the same nodes. This will ensure that the above setup can tolerate the failure of one master and one slave, making it more fault tolerant.</p><h4>Next steps</h4><p>This blog post serves as an architecture reference for fault tolerant Redis setup.</p><p>Stay tuned for more architectural deep dives into Redis and announcements of automation technologies for production ready architectures!</p><h4>Legal</h4><p>Redis and the Redis logo are the trademarks of Salvatore Sanfilippo in the U.S. and other countries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35334ef5de0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/fault-tolerant-redis-architecture-35334ef5de0b">Fault Tolerant Redis Architecture</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Highly available Redis Architecture]]></title>
            <link>https://medium.com/kokster/highly-available-redis-architecture-613c89f887b4?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/613c89f887b4</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[koki]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sidhartha Mani]]></dc:creator>
            <pubDate>Mon, 12 Feb 2018 20:17:05 GMT</pubDate>
            <atom:updated>2018-02-12T22:18:11.095Z</atom:updated>
            <content:encoded><![CDATA[<p>Redis is one of the most widely used container images in the world<a href="https://www.datadoghq.com/docker-adoption/">¹</a>. It is an open-source, in-memory, data-structure store. It is used as a cache, message broker, and as a database. <em>It is classified as a stateful service.</em></p><p>Running a stateful service such as Redis (with or without Kubernetes) requires careful architectural planning. This article goes over the architectural concerns that need to be addressed to confidently and correctly run Redis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*0NYxHJsEKUEiK6LF.png" /><figcaption>Stateful services may require complex orchestration</figcaption></figure><h3>Production Planning for Stateful Applications</h3><p>Planning to run stateful services in production requires carefully addressing <strong>at least</strong> the following set of concerns. These concerns do not just apply to Redis — an instance of this <em>framework-of-thinking</em> has been applied to PostgreSQL in one of our previous articles — <a href="https://goo.gl/G1QW4C">https://goo.gl/G1QW4C</a></p><p><strong><em>High Availability</em></strong></p><ol><li>Write Consistency guarantees</li><li>Replication</li><li>Clustering &amp; Sharding</li><li>Load Balancing</li></ol><p><strong><em>Fault Tolerance</em></strong></p><ol><li>Failover</li><li>Backups and Restore</li></ol><p><strong><em>Observability</em></strong></p><ol><li>Monitoring — Performance, and Uptime</li></ol><p><strong><em>Management Concerns</em></strong></p><ol><li>Migration</li><li>Upgrades and Downgrades</li></ol><p><strong><em>Infrastructure Planning</em></strong></p><ol><li>Network Partitioning</li><li>Service Distribution across Failure domains</li></ol><blockquote>Once these concerns are addressed and tested, any Stateful service can be confidently deployed in production.</blockquote><h4>The Redis Architecture</h4><p>The Redis 3.0 release consists of the following components :</p><ul><li>Redis Server</li><li>Redis CLI (and other clients)</li><li>Redis Sentinel</li><li>Redis Cluster</li></ul><h4>High Availability</h4><p>Reasoning about high availability can be done by addressing the various ways high availability can be disrupted.</p><blockquote>High availability can be described as the ability to serve requests even in the face of failures.</blockquote><p>The possible mechanisms of disruption of availability with respect to stateful services include —</p><ol><li>Data Inconsistency — What is written is what should be available to read (with reasonable guarantees). Stale, older data should not be a served. Redis has an inbuilt mechanism of client acknowledgement to ensure Data consistency.</li><li>Write load — Since writes are significantly more expensive than reads, failures happen at much lower request rate. Mechanisms to distribute write loads are mandated in such situations.</li><li>Data loss and Data Corruption — Node failures, Process crashes, and Disk failures can lead to data loss. Mechanisms to combat these are important.</li></ol><h4>Addressing Write Loads</h4><p>Redis Cluster is the primary mechanism for high availability. It provides mechanisms for sharding. A Redis shard is a subset of 16384 buckets that run in a particular failure domain. For sharding purposes, failure domains are generally <em>hosts</em>.</p><p>In sharded clusters, writes will be distributed among various shards and the failure of one shard will be isolated and will not lead to the failure of the entire cluster.</p><blockquote>Containerized platforms such as Kubernetes require careful consideration to ensure that two containers, each representing a shard, do not run on the same failure domain.</blockquote><h4>Tolerating Data Loss and Corruption via Replication</h4><p>Replication requires running the Redis cluster in a Master-Slave fashion.</p><p>The Redis Server can run in two modes:</p><ul><li>Redis Master</li><li>Redis Slave</li></ul><p>Redis Master is the <em>source of truth </em>and the write-head for the Redis cluster. The Redis slave (also known as <em>replica</em>) acts as a standby server for redundancy purposes. Redis slaves can be used to serve read-requests when run in a replicated fashion.</p><p>The Redis slaves connect to Redis masters to receive data and continuously synchronize the data with the master. This provides redundancy through replication. Redundancy addresses the above concern of <em>Data loss and corruption.</em></p><p>There are limits to the tolerance of data loss and corruption. This is partially due to the nature of replication between master and slaves. The master-slave replication is done asynchronously, and therefore an acknowledged write may not be available in the slave. A master failure before asynchronous streaming of data to slave would lead to data loss.</p><p>Mechanisms such as <em>multipath writes, disk level replication </em>etc. can be used to prevent the data loss scenario described above.</p><p>The other mechanism of data loss and corruption is where all of the masters and slaves fail. Mechanisms such as distribution of replication across failure domains (across availability zones) can be implemented to combat this failure.</p><h4>Quick Notes on Kubernetes</h4><p>Kubernetes provides excellent mechanisms to orchestrate the setup and provisioning of Highly available Redis clusters that address sharding, and replication.</p><p>When sharding, it is important to ensure that two shards do not run on the same host. This can be done using anti-affinity rules. Secondly, using node selector or node affinity, it is possible to select different availability zones for masters and slaves.</p><p>The configuration of this cluster can be managed as one config map, and a controller can be used to orchestrate the whole process. This will lead to a 1-click-install experience for setting up and managing a HA Redis cluster.</p><h4>Next Steps</h4><p>This article describes architectural planning for running highly available Redis Clusters. We still haven’t addressed how to make this cluster fault tolerant —</p><blockquote>Fault tolerance can be described as the ability to recover from failures of operation of a service.</blockquote><p>The next blog post will discuss fault tolerance in Redis. Soon-to-be-released Koki Installer for Redis will provide highly available and fault tolerant Redis for your Kubernetes cluster. Stay tuned for more updates!</p><h4>Legal</h4><p>Redis and the Redis logo are the trademarks of Salvatore Sanfilippo in the U.S. and other countries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=613c89f887b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/highly-available-redis-architecture-613c89f887b4">Highly available Redis Architecture</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Hyperledger Fabric — Endorsing Transactions]]></title>
            <link>https://medium.com/kokster/hyperledger-fabric-endorsing-transactions-3c1b7251a709?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/3c1b7251a709</guid>
            <category><![CDATA[hyperledger]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[hyperledger-fabric]]></category>
            <dc:creator><![CDATA[Kynan Rilee]]></dc:creator>
            <pubDate>Fri, 09 Feb 2018 19:40:23 GMT</pubDate>
            <atom:updated>2018-02-14T03:19:10.334Z</atom:updated>
            <content:encoded><![CDATA[<p>One key difference between Hyperledger Fabric and many other blockchain platforms is the lifecycle of a transaction. In other platforms, the lifecycle of a transaction is usually this:</p><ul><li><strong>Order:</strong> Transactions are added to the ledger in some order and disseminated to all peers.</li><li><strong>Execute:</strong> Transactions are sequentially executed (e.g. using <em>smart contract</em> code) on all peers.</li></ul><p>In order for all peers to end up with the same state, transactions must always execute deterministically — i.e. the same transaction must always create the same result, no matter where or when it is executed. This requirement places strong limitations on what a smart contract can do, which is one reason why smart contracts often use a special <em>domain-specific language</em>. It’s generally not possible to enforce determinism in a general-purpose language (e.g. Java or Go).</p><p>In Hyperledger Fabric, the lifecycle of a transaction is different:</p><ul><li><strong>Execute:</strong> Transactions are executed (using <em>chaincode</em>) in any order, possibly even in parallel.</li><li><strong>Order:</strong> When enough peers agree on the results of a transaction, it’s added to the ledger and disseminated to all peers. This step is where the transactions are first given an ordering — until transactions are added to the ledger, there’s no concept of one transaction happening before or after another.</li><li><strong>Validate:</strong> Each peer validates and applies the ledger’s transactions in sequence. Now that the transactions have an ordering, the peers can check whether a later transaction was invalidated by an earlier transaction. For example, this prevents one item from being sold two times (called <em>double-spending</em>).</li></ul><p>The first note to make here is that there’s a separation between the <em>execution</em> of a transaction (<em>Execute</em> step) and actually updating the ledger (<em>Validate</em> step). This separation has useful effects:</p><ul><li>All peers need to update the ledger, so all peers do the <em>Validate</em> step. But not every peer needs to <em>Execute</em> the smart contract. Hyperledger Fabric uses <em>endorsement policies</em> to define which peers need to execute which transactions. (More details later in this post.) This means that a given chaincode (smart contract) can be kept private from peers that aren’t part of the endorsement policy.</li><li>Transactions can be executed before they are put in order. This allows peers to execute transactions in parallel, which can improve throughput.</li><li>In Fabric’s three-step <em>execute-order-validate</em> model, the results of executing chaincode for a transaction are <em>explicitly</em> agreed upon (according to the endorsement policy) <em>before</em> the transaction is added to the ledger. The two-step <em>order-execute</em> model uses deterministic “chaincode”, which <em>implies</em> that peers will agree on the result of executing a smart contract. Making agreement explicit allows Fabric to use non-deterministic chaincode, which is why you can write Fabric chaincode in Go, Java, and Node.js.</li></ul><h4>Endorsement Policy</h4><p>Hyperledger Fabric allows users to define policies around the execution of chaincode. These <em>endorsement policies</em> define which peers need to agree on the results of a transaction before it can be added to the ledger. Fabric includes a small domain-specific language for specifying endorsement policies. Example endorsement policies might be:</p><ul><li>Peers A, B, C, and F must all endorse transactions of type T</li><li>A majority of peers in the channel must endorse transactions of type U</li><li>At least 3 peers of A, B, C, D, E, F, G must endorse transactions of type V</li></ul><p>The endorsement policy isn’t responsible for making sure the right chaincode is installed on the right peers. However, there is a similar mechanism for “endorsing” and installing chaincode packages — a topic for another post.</p><h4>How Endorsement Works</h4><p>So far, I’ve been using the term <em>transaction</em> somewhat loosely. In the <em>order-execute</em> model, the concepts of executing chaincode and updating the ledger are combined into one idea — <em>transaction</em>. In Fabric, these two concepts are separated, so the idea of <em>transaction</em> is split as well.</p><p>Fabric starts with a <em>transaction proposal</em>. It’s a bundle of information used to trigger a specific chaincode. The transaction proposal is sent to some peers for endorsement. An endorsing peer <em>executes</em> the chaincode, which (if it succeeds) yields an actual transaction for the ledger. The endorsing peer then signs the <em>transaction</em> and returns it to the proposer. This is the <em>Execute</em> step in <em>execute-order-validate</em>.</p><p>Once the creator of the proposal receives enough signatures to satisfy the endorsement policy, it can submit the transaction (and the signatures) to be added to the ledger. This is the <em>Order</em> step.</p><h3>Conclusion</h3><p>Hyperledger Fabric takes a novel approach to blockchain transactions. Separating the process of executing smart contracts from the process of updating the ledger makes it possible to improve transaction throughput, support more granular privacy controls, and implement more flexible and powerful smart contracts. One of the key ingredients for making this possible is a system for <em>explicitly</em> endorsing transactions before they are added to the ledger.</p><p>If you have thoughts or questions, please post them in the comments! I’m excited to continue providing information about Hyperledger Fabric and other blockchain technologies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*3n5c7n-B_2dDdj0o8TiS4g.png" /><figcaption>Check out the different Hyperledger projects at <a href="https://www.hyperledger.org/">https://www.hyperledger.org/</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c1b7251a709" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/hyperledger-fabric-endorsing-transactions-3c1b7251a709">Understanding Hyperledger Fabric — Endorsing Transactions</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to setup NGINX ingress controller on AWS clusters]]></title>
            <link>https://medium.com/kokster/how-to-setup-nginx-ingress-controller-on-aws-clusters-7bd244278509?source=rss----60f3ae8e606f---4</link>
            <guid isPermaLink="false">https://medium.com/p/7bd244278509</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[koki-short]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sidhartha Mani]]></dc:creator>
            <pubDate>Fri, 09 Feb 2018 18:47:30 GMT</pubDate>
            <atom:updated>2018-02-09T18:47:50.218Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>TL; DR Simply copy paste the commands to get a fully functional NGINX controller on any AWS Kubernetes cluster</blockquote><p>Ingress controllers are the gateway managers for network traffic entering into services running on your Kubernetes clusters. They decide how traffic is routed into the cluster and how it is morphed as it enters the cluster.</p><p>It is important to note the first statement I made above.</p><blockquote>Ingress controllers are the gateway <strong>managers</strong> for network traffic</blockquote><p>The reason I emphasized manager is because it only <em>manages</em> how traffic enters through the gateway. It is not the gateway itself.</p><p>Let us understand this with an example.</p><p>Consider an AWS setup with one EC2 instance backing a public-facing Elastic Load Balancer (ELB). The gateway for the traffic in this case would be the ELB. The gateway manager is the entity that configures the ELB and runs it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*KSSaN971JfLsvHVC.png" /><figcaption>Nginx is a great choice of reverse proxy for Kubernetes</figcaption></figure><p><em>Note: I’ve used the Short format to represent Kubernetes resources. It is easier on the eyes. You can find more information about it </em><a href="https://github.com/koki/short"><em>here</em></a><em>.</em></p><h4>Load Balancer Planning</h4><p>There are many ways to provision a load balancer for a cluster. Kubernetes lets you automate and program it to fit your needs precisely. In general, one of the following two models are used:</p><ol><li>Setup a cluster-level load balancer and use SNI or path-based routing to services</li><li>Setup a load balancer for each service that needs it</li></ol><p>The first approach is the one followed in this blog post. I find that most teams maintaining the load balancer are not the same as the ones managing applications. The first approach works better for such scenarios.</p><h4>Management on AWS Kubernetes clusters</h4><p>In this blog post, we’ll use the official nginx ingress controller residing at <a href="https://github.com/kubernetes/ingress-nginx">this repository</a>. This is a highly versatile and configurable version of the nginx ingress controller. It can be configured in any way that nginx itself can be configured.</p><p>This ingress controller works by first creating a ELB instance for the cluster. This property allows us to easily connect the nginx ingress controller to route53.</p><p>Let’s understand this by setting up a fully functional nginx ingress controller and ingress instances.</p><h4>Setup of NGINX ingress controller</h4><p>Start by creating a namespace for the ingress controller to run in</p><pre>$ kubectl create -f <a href="https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml">https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml</a></pre><p>Then create the cluster role and role for the nginx ingress controller</p><pre>$ kubectl create -f <a href="https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml">https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml</a></pre><p>This will create the cluster role and role for the ingress controller. Then, write a configuration file for configuring your nginx proxy</p><pre>config_map:<br>  data:<br>    client-body-buffer-size: 32M<br>    <strong>hsts: &quot;true&quot;</strong><br>    proxy-body-size: 1G<br>    proxy-buffering: &quot;false&quot;<br>    proxy-read-timeout: &quot;600&quot;<br>    proxy-send-timeout: &quot;600&quot;<br>    server-tokens: &quot;false&quot;<br>    <strong>ssl-redirect: &quot;false&quot;</strong><br>    upstream-keepalive-connections: &quot;50&quot;<br>    <strong>use-proxy-protocol: &quot;true&quot;</strong><br>  labels:<br>    app: ingress-nginx<br>  name: nginx-configuration<br>  namespace: ingress-nginx<br>  version: v1<br>---<br>config_map:<br>  name: tcp-services<br>  namespace: ingress-nginx<br>  version: v1<br>---<br>config_map:<br>  name: udp-services<br>  namespace: ingress-nginx<br>  version: v1</pre><p>Note the configuration set above. This is a standard config that has worked well for us.</p><p>It is recommended to use HTTP Strict Transport Security (hsts). This, along with SSL redirects, are used to redirect HTTP requests to HTTPS. This is generally considered to be more secure. We recommend it unless your service specifically needs to terminate SSL itself.</p><p>The other important option above is use-proxy-protocol. Since I’ll be setting up a L4 ELB loadbalancer (which does not forward SRC IP, SRC Port, SRC proto and other possibly important information to the services behind it) this option provides a mechanism to forward those L7 headers.</p><p>Let’s create this config map</p><pre>$ short -k -f configmap.short.yaml &gt; configmap.yaml<br>$ kubectl create -f configmap.yaml</pre><p>Now that the configuration is ready to be used, we can start by creating the default backend. The default backend acts as a catch-all service. It is routed to whenever an unknown URL is requested from this <em>proxy</em> (i.e. the nginx proxy we’ll be running).</p><pre>$ kubectl create -f  <a href="https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml">https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml</a></pre><p>This creates a pod and a corresponding service that acts as a “catch-all” when a request for an unavailable URL is made.</p><p>Now, we shall launch the nginx proxy pods that will route traffic to the default backend until an ingress rule is added.</p><pre>kubectl create -f <a href="https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml">https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml</a></pre><p>We’re almost done. The proxy is present and running. However, to route traffic from outside the cluster using a single URL, we need to create a ELB instance. This can be done by creating a Kubernetes Service of type Loadbalancer.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  annotations:<br>    <strong>service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp</strong><br>    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: &quot;3600&quot;<br>    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: &#39;*&#39;<br>    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <strong>arn:aws:acm:us-east-1:1234567:certificate/1234294-232-4f89-bca8</strong><br>    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https<br>  labels:<br>    k8s-addon: ingress-nginx.addons.k8s.io<br>  name: ingress-nginx<br>  namespace: ingress-nginx<br>spec:<br>  externalTrafficPolicy: Cluster<br>  ports:<br>  - name: https<br>    port: 443<br>    protocol: TCP<br>    targetPort: http<br>  - name: http<br>    port: 80<br>    protocol: TCP<br>    targetPort: http<br>  selector:<br>    app: ingress-nginx<br>  type: LoadBalancer</pre><p>Note that I’ve set a ssl cert (emphasized above). This is the cert for terminating SSL and should belong to the domain that you own. We shall see how to route traffic from this domain to specific services later in this article. For the sake of this example, let’s consider this domain to be:</p><p><em>kube-example.com</em></p><p>AWS certificate manager can be used to create a cert for this domain. Once created, note the arn for the cert and use that here to terminate SSL in the ELB. Now, Let’s deploy this Loadbalancer service</p><pre>$ short -k -f nginx-service.short.yaml &gt; nginx-service.yaml<br>$ kubectl create -f nginx-service.yaml</pre><p>The cluster will have a fully functional nginx load balancer fronted by ELB.</p><h4>Setting up Route 53 and Redirecting traffic</h4><p><em>Note that this part is not required for simple testing. You can use the ELB public URLs to test the routing capabilities.</em></p><p>Now that the ELB is up and running, you can configure a domain/sub-domain in route53 to point to this load balancer. Instructions for this can be found <a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer.html">here</a>.</p><p>Once setup, you can create services and redirect traffic to specific paths using ingress resources.</p><p>Let’s understand this with an example. As a quick recap, the domain we’re working with is <em>kube-example.com</em></p><p>Consider the example of exposing the Grafana dashboard for your Kubenretes cluster at the path /monitor. This can be achieved by created an ingress resource:</p><pre>ingress:<br>  annotations:<br>    <strong>ingress.kubernetes.io/rewrite-target: /</strong><br>    nginx.ingress.kubernetes.io/cors-allow-credentials: &quot;true&quot;<br>    nginx.ingress.kubernetes.io/cors-allow-headers: Authorization, origin, accept<br>    nginx.ingress.kubernetes.io/cors-allow-methods: GET, OPTIONS<br>    nginx.ingress.kubernetes.io/cors-allow-origin: kube-example.com/monitor<br>    nginx.ingress.kubernetes.io/enable-cors: &quot;true&quot;<br>  name: kubernetes-grafana-ingress<br>  namespace: kube-system<br>  rules:<br>  - host: kube-example.com<br>    paths:<br>    - path: /monitor<br>      port: 80<br>      service: monitoring-grafana<br>  tls:<br>  - hosts:<br>    - kube-example.com<br>  version: extensions/v1beta1</pre><p>Note the highlighted annotation. This annotation tells ingress to rewrite the path that is sent to grafana. The incoming request will be redirect as shown below.</p><pre><em>What User Requests: kube-example.com/monitor</em></pre><pre><em>What Grafana gets: kube-example.com/</em></pre><p>Also grafana requires CORS support. The configuration to add this is shown in the Ingress annotations above.</p><h4>Conclusion</h4><p>This setup of nginx ingress can be used to front your Kubernetes services and expose them to the world.</p><p>There is a lot more configuration to nginx and it mandates many blog posts to explain. If you’re interested in all the interesting ways to configure nginx, please comment below. I’ll create a blog series on it.</p><p>Stay tuned for more Kubernetes deep dives!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bd244278509" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kokster/how-to-setup-nginx-ingress-controller-on-aws-clusters-7bd244278509">How to setup NGINX ingress controller on AWS clusters</a> was originally published in <a href="https://medium.com/kokster">Koki</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>