<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Antoine Toulme on Medium]]></title>
        <description><![CDATA[Stories by Antoine Toulme on Medium]]></description>
        <link>https://medium.com/@jointhemachine?source=rss-9e3f4ade4009------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*74Uus815Uos-b1M68_M4wQ.png</url>
            <title>Stories by Antoine Toulme on Medium</title>
            <link>https://medium.com/@jointhemachine?source=rss-9e3f4ade4009------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:45:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jointhemachine/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using Chainlink Oracle Data with Splunk Ledger Analytics]]></title>
            <link>https://medium.com/splunk-dlt/using-chainlink-oracle-data-with-splunk-ledger-analytics-fd68546474c8?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd68546474c8</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[splunk]]></category>
            <category><![CDATA[ledger-analytics]]></category>
            <category><![CDATA[chain-link]]></category>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Wed, 01 Jun 2022 21:57:18 GMT</pubDate>
            <atom:updated>2022-06-01T21:57:18.193Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://chain.link/data-feeds">Chainlink Data Feeds</a> give smart contracts secure access to external data. The graphic below, <a href="https://blog.chain.link/analyze-decentralized-oracles-in-real-time/">taken from a Chainlink blog post</a>, explains the process:</p><ol><li>Chainlink nodes source data for a specific data point from multiple data aggregators.</li><li>Through a process involving multiple layers of aggregation, the Chainlink nodes come to an agreement on the most accurate value for that data point.</li><li>One of the Chainlink nodes sends a transaction to store this new data point on the ledger.</li></ol><p>Chainlink nodes provide this service across multiple chains for a wide range of data points. Developers can use this data in smart contracts as tamper-proof inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RiPiYY7Z2OrCRMh2" /></figure><p>With Splunk, we have built a view of this process as experienced by the Chainlink node operator. Note how each Chainlink node reports slightly different values over time. In the case of the feed, the average of the reported values is used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mtZ2IfvYrCovNZQa" /></figure><p>On <a href="https://data.chain.link/">https://data.chain.link</a>, you can view the list of Data Feeds for each chain. For more detailed information about Data Feeds, you can also consult the <a href="https://docs.chain.link/docs/reference-contracts/">developer documentation</a>. A popular feed is the <a href="https://data.chain.link/ethereum/mainnet/crypto-usd/eth-usd">ETH/USD pair</a>, which represents the value of Ethereum in US Dollars, created by aggregating price data from multiple exchanges and data aggregators.</p><p>The page on the ETH/USD pair links to a specific <a href="https://etherscan.io/address/0x5f4ec3df9cbd43714fe2740f5e3616155c5b8419">contract</a> on <a href="https://etherscan.io/">Etherscan</a>. With Etherscan, we can interrogate the view functions exposed by the contract. The most interesting view function is latestAnswer, giving us the latest data point:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CHxWOflx45oFyTRZ" /></figure><p>This is the value reported by the ETH/USD Chainlink Data Feed from the latest known block, as experienced by Etherscan. We can use our own infrastructure to interrogate the ledger and receive the answer for each block of the blockchain.</p><p>We ingest all those values and can show them as data points on a chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pd0hNd0g3LXqQhN_" /></figure><p>Better yet, those values come handy when we try to express the monetary cost of transactions — the dollar value of the gas cost associated with the transaction. That’s because we have this data available for every block. <a href="https://medium.com/splunk-dlt/visualizing-gitcoin-votes-with-splunks-web3-indexer-4190692c23d8">See our blog post on Gitcoin votes for a practical example.</a></p><p>Chainlink services have experienced rapid growth in the last few years. By providing access to a rapidly expanding library of critical market data on-chain, we expect Chainlink services will empower us with many more possibilities in the future to gain unique insights from data.</p><p>Do you have a suggestion for analytics for a Chainlink Data Feed, or would you like to see another visualization of Chainlink Data Feeds? Please <a href="https://discord.gg/HW8DhmBcsV">join us on Discord</a> to discuss.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd68546474c8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/splunk-dlt/using-chainlink-oracle-data-with-splunk-ledger-analytics-fd68546474c8">Using Chainlink Oracle Data with Splunk Ledger Analytics</a> was originally published in <a href="https://medium.com/splunk-dlt">Splunk DLT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enterprise Ethereum Alliance — Web3 Ledger Analytics with Splunk]]></title>
            <link>https://medium.com/splunk-dlt/enterprise-ethereum-alliance-web3-ledger-analytics-with-splunk-375f8ef08c2c?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/375f8ef08c2c</guid>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Thu, 12 May 2022 21:32:31 GMT</pubDate>
            <atom:updated>2022-05-12T21:32:31.075Z</atom:updated>
            <content:encoded><![CDATA[<h3>Enterprise Ethereum Alliance — Web3 Ledger Analytics with Splunk</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IaFwdStZhLjn7uGO" /><figcaption>EEA Monthly Education Series, Web3 Ledger Analytics</figcaption></figure><p>We presented with the Enterprise Ethereum Alliance as part of its education series on web3 ledger analytics.</p><p>You can watch the recording on Youtube:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FCrWmCLBM3RE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DCrWmCLBM3RE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FCrWmCLBM3RE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfc83f46de43a92d712d0738984460f1/href">https://medium.com/media/dfc83f46de43a92d712d0738984460f1/href</a></iframe><p>The webinar goes over ledger analytics in depth by first analyzing block and transactions data, then dives into transaction receipts and events, and makes a greater point about ledger data associated with business use cases. We also mention that we think we found the best web3 data platform… but really you should watch it to avoid spoiling the surprise :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=375f8ef08c2c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/splunk-dlt/enterprise-ethereum-alliance-web3-ledger-analytics-with-splunk-375f8ef08c2c">Enterprise Ethereum Alliance — Web3 Ledger Analytics with Splunk</a> was originally published in <a href="https://medium.com/splunk-dlt">Splunk DLT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Geth hosting with Splunk]]></title>
            <link>https://medium.com/splunk-dlt/geth-hosting-with-splunk-62045c615ab4?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/62045c615ab4</guid>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[splunk]]></category>
            <category><![CDATA[geth]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Wed, 19 May 2021 21:26:23 GMT</pubDate>
            <atom:updated>2021-05-19T21:26:23.282Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*jiBOP9VGVX9OsYj-yTVd8A.png" /></figure><p>Blockchain offers an ideal of independence by inviting participants to enter a public, permissionless network of peers. When you participate, you can create and maintain your view of the data that transit on the chain, generally constituted of blocks and transactions.</p><p>This article will show you how to run the Ethereum client <a href="https://geth.ethereum.org/">Geth</a>, developed by the <a href="http://ethereum.org/">Ethereum Foundation</a>. It will demonstrate how to expose and analyze its operation metrics, its logs, and most importantly its ledger data.</p><h3>Running Geth</h3><p>You can run Geth in a multitude of <a href="https://geth.ethereum.org/docs/install-and-build/installing-geth">well-documented ways</a>. You should choose the setup that works best for you. After <a href="https://www.youtube.com/watch?v=2I_Cfr-OUp4">listening again</a> to Péter Szilágyi’s advice on best operations practices, it seems best to run Geth directly as a service. This allows you to monitor the host metrics, knowing no other service is competing for attention on the box. The install doesn’t use Docker to be as close to the metal as possible.</p><p>The service is rolled out using Ansible and then installed Geth with this service file:</p><pre>[Unit]<br>Description=Go-Ethereum<br>Requires=network.target<br>After=syslog.target network.target<br>[Service]<br>Type=simple<br>ExecStart=sh -c &quot;geth \<br>          --{{ network }} \<br>          --datadir /gethdata/ \<br>          --nousb \<br>          --ipcdisable \<br>          --metrics \<br>          --metrics.addr=127.0.0.1 \<br>          --metrics.port=6060 \<br>          --http \<br>          --http.api=admin,db,eth,debug,miner,net,txpool,personal,web3 \<br>          --http.port=18545 \<br>          --http.vhosts=localhost \<br>          2&gt;&gt; /gethdata/geth.log&quot;</pre><pre>[Install]<br>WantedBy=multi-user.target</pre><p>Note the deployment uses a <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_templating.html">Jinja2 template</a> so it can interpolate the network based on how Ansible is configured.</p><p>Geth also uses a /gethdata folder to store data and the log file.</p><p>The setup exposes as many JSON-RPC APIs as possible to collect more data (See the section below Ingesting the ledger data). <strong>You should not expose all those methods to external callers</strong>, or anonymous callers can abuse your chain data, reset your chain head or get access to accounts. We recommend the use of a JSON-RPC proxy application whitelisting callers and filtering methods.</p><h3>Collecting metrics and logs</h3><p>Alongside Geth, the Splunk OpenTelemetry Collector runs to gather both logs from the Geth log file and metrics by connecting to the Geth node’s Prometheus port, sending them to the Splunk backend.</p><p><a href="http://opentelemetry.io/">OpenTelemetry</a> is an open source project that aims to help standardize how to set up to ingest metrics, traces, and logs. <a href="https://github.com/signalfx/splunk-otel-collector">Splunk’s distribution</a> comes with great documentation and offers Debian repositories to perform a safe installation.</p><p>In this particular case, the collector is installed through a <a href="https://github.com/signalfx/splunk-otel-collector/blob/main/docs/getting-started/linux-manual.md">manual installation using the Debian package</a>. You can choose to use a simple <a href="https://github.com/signalfx/splunk-otel-collector/blob/main/docs/getting-started/linux-installer.md">installer as well</a>, amongst other possibilities.</p><p>The OpenTelemetry Collector takes a yaml file as its configuration:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a90ca823fad7451e262a914068d9352e/href">https://medium.com/media/a90ca823fad7451e262a914068d9352e/href</a></iframe><p>A few things to note in this configuration:</p><ul><li>All host metrics are monitored with the hostmetrics receiver. It gives detailed memory, CPU and disk information.</li><li>The Geth client is scraped every 10s on localhost:6060.</li><li>The filelog receiver listens to the geth log file, parsing each line looking for the timestamp and sending all the content to Splunk.</li></ul><p>All the environment variables are separated and filled out in a separate file (/etc/otel/collector/splunk-otel-collector.conf), per the installation instructions.</p><h3>Ingesting the ledger data</h3><p>Now that basic metrics are in place, Ethlogger comes into the picture to ingest the ledger data and Geth metadata.</p><p>Ethlogger runs as a Docker container to avoid installing nodeJS on the VM. It is configured to run with the host network mode, so it can access the local Geth port. It’s also running with the option to restart always.</p><p>The ansible task looks like this:</p><pre>- name: Run ethlogger container<br>  community.docker.docker_container:<br>    name: ethlogger<br>    image: ghcr.io/splunkdlt/ethlogger:3.0.1<br>    volumes:<br>      - /gethdata/ethlogger:/app<br>    state: started<br>    restart_policy: always<br>    network_mode: host<br>    env:<br>      COLLECT_PEER_INFO: &quot;true&quot;<br>      COLLECT_PENDING_TX: &quot;false&quot;<br>      ETH_RPC_URL: &quot;http://127.0.0.1:18545&quot;<br>      NETWORK_NAME: &quot;{{ network }}&quot;<br>      START_AT_BLOCK: &quot;latest&quot;<br>      SPLUNK_HEC_URL: &quot;{{ splunkHecUrl }}&quot;<br>      SPLUNK_HEC_TOKEN: &quot;{{ splunkHecToken }}&quot;<br>      SPLUNK_EVENTS_INDEX: &quot;{{ splunkIndex }}&quot;<br>      SPLUNK_METRICS_INDEX: &quot;{{ splunkMetricsIndex }}&quot;<br>      SPLUNK_HEC_REJECT_INVALID_CERTS: &quot;{{ splunkRejectInvalidCerts }}&quot;</pre><h3>Building the battle station</h3><p>All the crucial operational data is streaming to the Splunk instance.</p><p>Splunk users can consult logs, metrics, and ledger data together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVC0rOZkGmHawTsABRL8vA.png" /><figcaption>Metrics and logs showing the overall health of the Geth node</figcaption></figure><p>Prometheus metrics from Geth are combined with resource usage metrics to understand the patterns of use and see what is impacting the node.</p><p>Splunk userssearch logs with infinite flexibility. For example, let’s see how many times the node went through a <a href="https://en.bitcoin.it/wiki/Chain_Reorganization">reorg</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qCmoL_9VLeRJrv5K1xQnzw.png" /></figure><p>From there, one can set alerts and watch the nodes. If something goes astray, the team will be notified and can look at dashboards to find out what is going on.</p><p>Data analysts can see how users send transactions to the chain, examining the parameters of a smart contract call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZU3mMQIoPy1ABVBx6LYjRw.png" /><figcaption>Calling an approve function on an ERC20 contract</figcaption></figure><p>This is just the tip of the iceberg of possibilities offered by Splunk. Here is sneak peek showing a graph of transactions at EthDenver:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dz3bUOe7FDPcsvq9GztHPg.png" /><figcaption>EthDenver analytics!</figcaption></figure><p>My colleague <a href="https://www.splunk.com/en_us/blog/author/sluedtke.html">Stephen</a> talks more about this in <a href="https://www.splunk.com/en_us/blog/it/connecting-ethereum.html">this blog post</a>.</p><p>That’s it folks! For more on how we leverage Splunk’s unique data platform capabilities, please visit our <a href="http://landing.splunkdlt.com/">website</a> or email us at <a href="/service/mailto:blockchain@splunk.com">blockchain@splunk.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62045c615ab4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/splunk-dlt/geth-hosting-with-splunk-62045c615ab4">Geth hosting with Splunk</a> was originally published in <a href="https://medium.com/splunk-dlt">Splunk DLT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Byte Serialization in Blockchains]]></title>
            <link>https://medium.com/whiteblock/byte-serialization-in-blockchain-e147347ab578?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/e147347ab578</guid>
            <category><![CDATA[serialisation]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[bitcoin]]></category>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Fri, 20 Dec 2019 16:13:06 GMT</pubDate>
            <atom:updated>2019-12-20T18:10:28.958Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UPAmoAUI9mY4CCPQ.jpg" /><figcaption>Cereal photo by <a href="https://unsplash.com/@nyanastoica?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Nyana Stoica</a> on <a href="https://unsplash.com/s/photos/cereal?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><em>This article was a collaboration between Antoine and Kevin.</em></p><p>In this post, we’ll discuss byte serialization in blockchain, why it matters, as well as how different chains approach it, including:</p><ul><li>Ethereum 1</li><li>Ethereum 2.0</li><li>Bitcoin</li><li>Secure Scuttlebutt (Bonus!)</li></ul><p>Serialization is the process of converting an object into a stream of bytes to store the object or transmit it to memory, a database, or a file using a common set of rules a system can understand. Its main purpose is to save the state of an object in order to be able to recreate it when needed. The reverse process is called deserialization. In this process, you’d use the same set of rules to decode the translated data back into its original form. Serializing data is also known as marshaling and deserialization is known as unmarshaling.</p><h3>Why is byte serialization important for blockchains?</h3><p>Serialized data does not have to be in bytes as long as it’s in a format that is efficient for the particular system the data will live in. For example, in WWII data was serialized into morse code in order to transmit data via voice radio systems. Bytes happen to be the most convenient format for computers to operate in. Inherently, blockchains maintain a state of the world to reach consensus. Blockchain based protocols like Bitcoin and Ethereum leverage this technology but in order for nodes to communicate across a network in order to maintain the state, they need to send packets of byte encoded data via peer to peer networking using some sort of standard serialization algorithm.</p><p>This is why serialization plays a very important role in distributed systems. When you represent a domain object in a blockchain, you will want to generate a proof of it. The object may be inserted into a data structure, which itself needs to generate a proof. A good way to prove your message is authentic is to sign it with a key or to include the hash of the message into a block. It’s the essence of a blockchain’s function, sending around hashes of transactions and blocks and making sure they match. Let’s take a look at some of the different implementations.</p><h3>Ethereum 1x: Recursive Length Prefix</h3><p>Ethereum uses a Patricia Merkle tree based on this byte serialization down to the point a single hash can identify the particular state of the tree. The purpose of <a href="https://github.com/ethereum/wiki/wiki/RLP">RLP</a> (Recursive Length Prefix) is to encode arbitrarily nested arrays of binary data, and RLP is the main encoding method used to serialize objects in Ethereum.</p><p>“… Data in Ethereum is <a href="https://stackoverflow.com/questions/3784143/what-is-the-difference-between-serializing-and-encoding">serialized</a> and deserialized as a <a href="https://stackoverflow.com/questions/4019837/what-do-we-mean-by-byte-array">byte-array</a>. Put, an array of bytes is a <a href="https://stackoverflow.com/questions/4019837/what-do-we-mean-by-byte-array">byte-array</a>.”</p><p><a href="https://github.com/ethereum/wiki/wiki/RLP">RLP</a> is like a binary encoding of JSON, if JSON were restricted only to strings and arrays. The RLP encoding function takes in an item. An item is defined as follows：</p><ul><li>A string (ie. byte array) is an item</li><li>A list of items is an item</li></ul><p><strong>Examples of item and items</strong></p><ul><li>“crypto”</li><li>[ [] ]</li><li>[ “Lion” ]</li><li>[ [ “cat”, 123, ‘d’, ‘o’, ‘g’] ]</li></ul><h3>Ethereum 2.0: Simple SerialiZe</h3><p>As part of the upgrade to Ethereum 2.0, researchers and developers have been working on significant improvements to the Ethereum protocol in addition to its network architecture. Based on what was learned with RLP, Simple SerialiZe (SSZ) was developed as the culmination of this hard work. This is the serialization algorithm standard for all data structures common across Ethereum 2.0 client implementations. It is outlined in the official<a href="https://github.com/ethereum/eth2.0-specs/blob/dev/specs/simple-serialize.md"> Ethereum 2.0 specification</a>. SSZ is poised to be the new standard for marshaling consensus data into bytes.</p><p>Specifically, the SimpleSerialize (SSZ) <a href="https://github.com/ethereum/eth2.0-specs/blob/dev/specs/simple-serialize.md">specification</a> defines how different types should be serialized and represented as a merkle tree. Building on RLP, SSZ offers a few more abilities:</p><ul><li>Deterministic lookup of an element in bytes–this means you can look at the bytes of the SSZ output, and if you know what the data structure is, you can look up the bytes of one of the fields of the data structure directly.</li><li>Merkle hash partial computation aka SSZ partials allow us to update the SSZ representation and recompute the hash without recomputing from scratch, optimizing the hashing process.</li></ul><h3>Bitcoin Serialization</h3><p>Let’s take a look at Bitcoin’s approach. Under the current <a href="https://btcinformation.org/en/glossary/consensus-rules">consensus rules</a>, a block is not valid unless its serialized size is less than or equal to 1 MB. The following fields also determine the serialized size:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*XQ_lkhPIrQr0US-Xy5iPKQ.png" /></figure><p>All data structures in Bitcoin use a custom Bitcoin specific serialization format. The standard that is followed is the Bitcoin defined standard, not any other standard. Two formats are currently supported by the Bitcoin community, the non-segwit format (click <a href="https://btcinformation.org/en/developer-reference#raw-transaction-format">here</a>). The segwit format is described in <a href="https://github.com/bitcoin/bips/blob/master/bip-0144.mediawiki">BIP 144</a>. Segwit implements a new serialization format for tx messages to the peer-to-peer protocol. Arguments have been made about supporting this new serialization method but some point out that it is an improvement over the current bitcoin blockchain which reduces the size needed to store transactions in a block. This is done by removing certain signatures with counting serialized witness data as one unit and core block data as four units.</p><h3>SSB: Secure Scuttlebutt (bonus!)</h3><p>Different forms of serialization are being used in <a href="https://scuttlebutt.nz/about/">Secure Scuttlebutt</a>, a global cryptographic social network. Here, they’re using JSON for its messages which configures messages to a specific format to allow signing.</p><p>Here’s an example from their docs:</p><p><strong>Message format</strong></p><p><strong>To create a message to post in a feed, start by filling out these fields:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/0*Op6lMhm-6yHZTkJQ.png" /></figure><p>All messages in a feed are signed by that feed’s long-term secret key, enabling recipients to verify that the message was truly posted by a particular identity and not tampered with as it gets gossiped and replicated throughout the Scuttlebutt network. Before signing a message, it must be serialized according to a specific canonical JSON format. Making it so that for any given message there is exactly one way to serialize it as a sequence of bytes, which is necessary for signature verification to work. The reference implementation verifies that all messages it receives are in the canonical format and rejects messages that aren’t. To read more about the specific rules for this serialization, check out their <a href="https://ssbc.github.io/scuttlebutt-protocol-guide/#message-format">docs</a>.</p><p>Although different networks approach serialization differently, it’s an important feature of a distributed system. At Whiteblock many of us are excited to continue assisting in the development of Ethereum 2.0 and hope to share more details about these improvements soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oakDRbZQMsL6Ffb6.jpg" /></figure><p>At Whiteblock, we’re actively working on providing the best tools to test and optimize distributed systems. Join us for our official launch on January 15th and stay tuned on all of our upcoming updates via <a href="https://t.me/whiteblockio">telegram</a>.</p><p><em>Originally published at </em><a href="https://whiteblock.io/byte-serialization-in-blockchain/"><em>https://whiteblock.io</em></a><em> on Dec 20, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e147347ab578" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whiteblock/byte-serialization-in-blockchain-e147347ab578">Byte Serialization in Blockchains</a> was originally published in <a href="https://medium.com/whiteblock">Whiteblock</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[No-Tag-Back Gossiping]]></title>
            <link>https://medium.com/whiteblock/no-tag-back-gossiping-c8b99906e891?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8b99906e891</guid>
            <category><![CDATA[gossiping]]></category>
            <category><![CDATA[p2p]]></category>
            <category><![CDATA[protocol]]></category>
            <category><![CDATA[network]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Wed, 11 Dec 2019 21:42:07 GMT</pubDate>
            <atom:updated>2019-12-11T21:42:07.906Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*BAvUBNU9C_1D0iZF2RQAOg.gif" /><figcaption>Tag!</figcaption></figure><p>In this blog post, I’d like to introduce you to an evening ramble with <a href="https://twitter.com/deaneigenmann">Dean Eigenmann</a> called No-Tag-Back Gossiping, or NTB-G. This post came about as we worked through ideas to make gossiping more reliable.</p><p>Make sure to <a href="https://hackmd.io/owviUNfeRkmSlgtJQcaQFg?view"><strong>read the spec</strong></a> first, as it clearly lays out the important components. Below, I’ll talk about the whys and hows of some of the requirements we establish.</p><p>When you gossip a message, the idea is that the message will traverse the whole network and eventually be sent to everyone. But not everyone cares about your message and the network might be bigger than you think. Before you know it, you clogged the network and pissed off everyone, and worst of all, they don’t know how to let you know about it or to shut you out. All they get is your message, one after the other, like an acid drip that just won’t stop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*khbAsDkrSve993lDxdmHoA.gif" /></figure><p>At the same time, what if you’d like to trigger reorgs of your network to make it easier to communicate with distant peers (in terms of number of hops)? And get them some messages in time? We’re going to cover all this.</p><p>Before we get started, please consider reading the <a href="https://www.gsd.inesc-id.pt/~ler/reports/srds07.pdf">Plumtree paper</a>. Otherwise not much of what follows will make sense to you. If we’re both going to pretend you read it, here are some coping strategies:</p><ul><li>Print it on tissues so you can read it, one sentence at a time, when you blow your nose.</li><li>Print it on cotton. Make a pillow of it. Many pillows even. Then splurge on Plumtree whenever you have TV night.</li><li>Note that paper is from 2007 and people must have come up with something better. Become outraged that the author of this article came up with such an old paper for what is a seminal topic, well-researched by academics. Get an acm.org membership. Research the subject. Map out all papers about it. Eventually take to Twitter and in 2 tweets put an end to this author’s career, bringing an end to his indigence in fact checking.</li><li>Live tweet the whole paper, one paragraph at a time. Add comments in the voice of Bob Ross or Neil de Grasse Tyson.</li><li>Or read below the cliff notes, which won’t replace reading the paper:</li></ul><p>Plumtree: A bunch of peers get connected. They send each other messages. You send to everyone. You get a new message, you broadcast to everyone (except the sender, doh). Eventually some of them reply to you saying, “hey, I got it already, thanks” and let you know to just send a signature or hash of the message going forward. But sometimes their paper boy is late, they don’t get the right news with Fox, so they come back and say “hey, thanks for that hash, can you give me the full message again?” in which case you start sending them full messages again.</p><p>Plumtree has been equated to epidemic broadcast trees. It allows the network over time to get better at circulating messages. And we’re going to riff on it through this whole post.</p><p>A few definitions for you, dear reader:</p><ul><li>Full message: all the bytes of the original message, as the sender intended.</li><li>Headers: the hash of the message, possibly signed by you (spoiler), so they can check they got it. Others call those attestations or metadata.</li><li>Eager peers: the unfortunate peers that get your whole messages.</li><li>Lazy peers: the peers who get just the headline, ie the signature or hash.</li></ul><p>OK, that was the hard part. Now let’s engage on the standard components of NTB-G.</p><h3>Identity through handshake</h3><p>First and foremost, you have to have some notion of identity. Take any peer-to-peer system; it’s usually a public key. And to show that you have the private key somewhere, you use that public key as part of the handshake process, meaning when you first connect to a peer. This notion of identity is going to be helpful down the road.</p><p>Note what we didn’t say: You can implement a system where your software has more than one identity. Actually that sounds fun. You can change your identity as often as you like. More power to you.</p><h3>Rate limiting</h3><p>You’re in this beautiful new world of peer-to-peer communications, and looking forward to blasting your entire collection of genuine gifs of german metal bands. But here’s the thing, not everybody likes your content or is even looking for it.</p><p>So they are going to apply rate limiting, which means they will slow down pushing or outright stop passing around your message if you send more than your quota. Here’s how it works:</p><p>We’re going to define a window of time relative to the two peers. Note we don’t use clocks, because clocks never work across computers very well. We’re going to apply a sliding window of time, defined usually in seconds, for how many messages we’re going to deal with from you. We’re going to allow a certain number of full messages, and an even greater number of headers. This way, if you all of the sudden have a million messages to send, you can first send all the full messages you are allowed, and then start sending headers.</p><p>We set those parameter limits at handshake time. This is important, because it gives you an idea of the throughput your peer can experience on the network — and therefore you know how many peers you’ll need.</p><h3>Blacklist</h3><p>If your node goes over the limits, the peer may decide to blacklist you. That’s pretty harsh. It means you get disconnected from the peer, and cannot reconnect, for a period up to the peer (and not advertised, so you can’t really guess or play it).</p><h3>New rules in NTB-G</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H7ecOyKJNTn2p891sROHSg.jpeg" /></figure><p>o far, we haven’t defined anything groundbreaking, so let’s go and play with our parameters a bit.</p><p>New rule:</p><ul><li>We are going to add metadata to each message, full or headers.</li><li>The metadata is the trail of the peers the message went through (or just a counter if you’re constrained for size). Each peer is signing the message with their pub key (including the previous metadata), so you can verify it’s legit.</li><li>We’re going to create a notion of max hops for the messages. We’ll go into that some more.</li><li>And we’re going to add, optionally, that any message can have a target or a trail of peers as targets.</li></ul><p>Let’s unpack.</p><p>First, during handshake, you add two new limits: the number of hops for full messages, and the number of hops for headers. Hopefully you’re more likely to keep repeating headers more than full messages, so your max hops for headers should be much higher than the max hops for full messages. Then, when your node receives a message, it verifies the validity of the message by checking the last peer signed it properly and signs it. It checks the list of peers — anybody blacklisted in there? If yes, your peer may decide to stop right there and then and stop propagating the message.</p><p>Your node changes the message to sign it with its own public key — you know the one we talked about earlier? Your node then sends the new message to all its eager peers and sends the headers to all lazy peers — just like Plumtree intended.</p><p>With a twist..</p><p>By counting the trail of peers or looking at the counter, your node may decide to stop propagating a message if it reached the maximum number of hops. Your node can stop propagation. In the case of a full message, it can switch over to headers and send them instead.</p><p>If the node gets headers and wants the full message, it can request the full message from the sender of the message, which may pass it back along. Note in that case you embed the path into the request, and any peer in that path with the full message can stop the request and send back the message.</p><h3>Routing optimization #1</h3><p>Now you’ve got a system where peers talk to each other, and have a remote idea of how they’re tied together. So as part of that request for a full message, your node may also give its personal information (“here’s my number”). That allows any remote peer in the trail to opportunistically connect directly, if that makes sense (and it does, here’s a distant peer who wants your message, you should optimize routing for them).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*WrBCbG9eJrtKo7iTyuxaCA.gif" /></figure><h3>Routing optimization #2</h3><p>A full message comes in, and you send to all your peers. Well, not the sender, right? You can also look at all the peers the message went through. If any of them are your peers, you can avoid sending them the full message. Send them the header’s message instead.</p><h3>Routing optimization #3</h3><p>You get a message, and it’s intended for a specific peer. It’s one of your peers. You send it to that peer and you stop propagating the message once you get an ACK back (or none, I never said we had to be transactional about this). You get a message and it’s for <em>you</em>. You don’t send that message to everyone you know.</p><h3>Gossiping as a gateway drug</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLfPaxfj91SF0iuIi_i6wA.jpeg" /></figure><p>Gossiping is noisy and messy. So sooner or later, you should specialize your comms. While it’s ok to blast everyone with consensus related info, it’s likely other peers will want to communicate with you directly. This type of reallocation of connections allows just that.</p><p>Step 1: send all your peers a free sample</p><p>Step 2: watch them send back a response asking for the real thing, alongside their connection info</p><p>Step 3: connect to them, either directly and/or gossiping over other peers (think TCP holepunching). You can encrypt messages using their public key if you feel particularly shy.</p><p>Step 4: Profit?</p><h3>In conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*RllJ1pwI3JEIOQ7HdtuaKg.gif" /></figure><p>In this blog post, I outlined some ramblings I had with Dean over Telegram and HackMD. We want to allow gossip communications to be resilient to spam, so we introduced rate limiting, and realism to the way they propagate to the network (gotta stop at some point). By introducing those limits, we discover new fun ways that allow us to optimize routing messages, which may make the system more viable under load with many, many participants.</p><p><em>Originally published at </em><a href="https://whiteblock.io/no-tag-back-gossiping/"><em>https://whiteblock.io</em></a><em> on Dec 11, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8b99906e891" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whiteblock/no-tag-back-gossiping-c8b99906e891">No-Tag-Back Gossiping</a> was originally published in <a href="https://medium.com/whiteblock">Whiteblock</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Maintaining Apache Tuweni — the 6 months mark]]></title>
            <link>https://medium.com/whiteblock/maintaining-apache-tuweni-the-6-months-mark-a4853813ffe7?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4853813ffe7</guid>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Sun, 01 Dec 2019 07:10:06 GMT</pubDate>
            <atom:updated>2019-12-02T17:36:57.928Z</atom:updated>
            <content:encoded><![CDATA[<h3>Maintaining Apache Tuweni — the 6 month mark</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*bQ192hC6IFURf6QZ2x8waw.png" /></figure><p>I have maintained in one form or another open source for close to 10 years.</p><p>I recently helped form the <a href="https://tuweni.apache.org/">Apache Tuweni project</a>, and the project is slowly making its way through the baptism of fire of issuing <a href="https://github.com/apache/incubator-tuweni/releases">its first releases</a> and reaching complete legal compliance.</p><p>Along the way, I get asked by other developers what this project will become and how long I will maintain it.</p><p>I find both questions incongruous. While I happen to have been around to push the project to the ASF, I don’t have a leadership position and I certainly don’t dictate a roadmap. All I can do is put my best code forward and review contributions from others to make the project a success.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*mOrFg8baBiUlu4Uq-TIX7A.jpeg" /><figcaption>Your patch is important. We will be right with you.</figcaption></figure><p>As for my role as a maintainer, I certainly plan to maintain the project forever. That however doesn’t mean what folks in traditional software development projects imagine. Everything I do is under my own time on a best effort basis, with no guarantee of quick fixes, or a rapid response on the lists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-I5arIyDF-IMe-DMQCtqng.gif" /><figcaption>Let a thousand flowers bloom</figcaption></figure><p>It’s not what you think. It’s never about code. My measure of success is to see the code of Apache Tuweni used elsewhere and allowing a thousand flowers, or p2p apps, to bloom. What is absolutely critical to me is to foster adoption and grow a community of developers and projects using this code — already spotted in the wild several times!</p><p>One of the greatest barriers of Tuweni to succeed is to encourage users, contributors and the community at large to communicate and start building things together. It looks like email in 2019 is just not cutting it anymore. Not one contributor or user used email to reach out, preferring Keybase, Telegram or Twitter.</p><p>It also looks like reassessing how the projects runs and <a href="https://www.apache.org/theapacheway/">the ideals of the Apache Way</a> need to be explained to new entrants. Amazingly, the blockchain crowd seems to lack understanding of open source old beards like me have been accustomed to. I have been enjoying a regimen of legal advice, patent troll encounters, copyright vs license discussions and even a taste of various open source foundations models to situate myself. I feel like this is lost on the new generation.</p><p>I’m excited for the future of this project. We have at hand the ability to unleash ground breaking, peer to peer applications that will reshape Internet. <a href="http://tuweni.apache.org/community/">Join us!</a></p><p><em>Originally published at </em><a href="https://whiteblock.io/maintaining-apache-tuweni-the-6-month-mark/"><em>https://whiteblock.io</em></a><em> on Dec 2, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4853813ffe7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whiteblock/maintaining-apache-tuweni-the-6-months-mark-a4853813ffe7">Maintaining Apache Tuweni — the 6 months mark</a> was originally published in <a href="https://medium.com/whiteblock">Whiteblock</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High Availability in Distributed Systems]]></title>
            <link>https://medium.com/whiteblock/high-availability-in-distributed-systems-2d3325e4ee33?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d3325e4ee33</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[high-availability]]></category>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Tue, 26 Nov 2019 14:01:04 GMT</pubDate>
            <atom:updated>2019-11-26T14:01:04.152Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqoLCgRmHhJ2N66efFN7Zw.png" /></figure><p><em>This article was a collaboration between </em><a href="https://medium.com/u/9e3f4ade4009"><em>Antoine</em></a> <em>and Kevin (</em><a href="https://medium.com/u/39d1dce2815c"><em>Bapecoin</em></a><em>).</em></p><h3>Definition of high availability</h3><p>High availability (HA) can be achieved when systems are equipped to operate continuously without failure for a long duration of time. The phrase suggests that parts of a system have been thoroughly tested and accommodated with redundant components to remain functional when a failure occurs.</p><p>Numerous modern applications are often distributed and cloud-based. Highly-available architectures facilitate scaling in order to meet demand and make them more resilient to failure. These applications usually have frequent deployments. The increased amount of moving parts during this process also increases the risk of something going wrong. To meet these challenges, application and infrastructure teams may adopt a deployment strategy suitable for their use case such as Blue-Green or Canary deployment. However, we will not be discussing high-availability as part of the development process in this article.</p><h3>History of HA and when it became important</h3><p>HA is a relatively new term that matches the expansion and professionalization of Web 2.0 companies. Back in 2004, it was common to take down websites for maintenance. The story of how HA became important is synonymous with the story of distributed databases. Historically, a database was either up or down, there was no such thing as a partial failure. As the internet was adopted globally, downtime began to directly impact users and business. Concurrently, there was a disconnect between development and operation teams. They worked as two independent squads, each with its own objective. Operators had little understanding of the code base and developers had little understanding of the operational practices. Developers were concerned with shipping code and operators were concerned with reliability.</p><p>This lack of communication between teams impacted the product and ultimately, end users. Since reliability is the single most important feature of any system today, a set of practices and culture coined as DevOps was created to improve communication and build better products. It subsequently became one of the most critical positions in every company, along with Site Reliability Engineers. The SRE role is a very opinionated concrete implementation of DevOps principles. Without reliability, users will not trust the system and thus not use it, leaving organizations with an expensive system that’s useless.</p><h3>HA in traditional client-server architecture</h3><p><strong>Middleware HA — load balancer<br></strong>A number of high-availability solutions exist such as load balancing and basic clustering. Load balancing also known as horizontal scaling can be achieved by adding new nodes with identical functionality to existing ones, redistributing the load amongst them all. By exclusively redirecting requests to available servers, reliability and availability can be maintained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*NjGWtbBRtxXUVe_O.png" /></figure><p><strong>Database master/slave set up<br></strong>Other solutions such as PostgreSQL Master-Slave architectures enable us to maintain a master database with one or more standby servers ready to take over operations if the primary server fails. Replication is established between the master and slave via SQL statements or via internal data structure modifications. When a main server fails, a standby server that is now equipped with almost all of the data of the main server can swiftly act as the new master database server.</p><p>Learn more at the official PostgreSQL site, <a href="https://www.postgresql.org/">https://www.postgresql.org/</a>.</p><p><strong>Database clustering<br></strong>A data cluster is when multiple nodes or instances connect to a single database. Database clustering enables redundancy, availability, scalability and monitoring. Several tools from IBM and Oracle exist to manage a variety of configurations that can be used to provide high availability. For example, if a node or Oracle instance process fails in an Oracle RAC configuration, other Oracle instances can provide service as they have access to a common database file system across <em>interconnect, </em>a private high-speed network for messaging between nodes.</p><h3>NoSQL revolution towards modern web architecture</h3><p><strong>MongoDB</strong><br>Different database technologies have been developed in response to a rise in volume of data stored about users, objects and products. To handle the scale and agility challenges that face modern applications and capitalize on the cheap storage and processing power available today, architectures like MongoDB’s sharded cluster were developed. This architecture consists of three main components: a shard, mongos and config servers. A shard is a replica set or a single Mongod instance that holds the data subset used in a sharded cluster. If you directly connect to a shard, only a fraction of the data contained in a cluster can be viewed. Consider deploying a sharded cluster when your system’s dataset outgrows the storage capacity of a single mongodB instance otherwise it will add unnecessary complexity.</p><p>Learn more about MongoDB’s sharded clusters here: <a href="https://docs.mongodb.com/manual/core/sharded-cluster-components/">https://docs.mongodb.com/manual/core/sharded-cluster-components/</a></p><p><strong>Cassandra<br></strong>Apache Cassandra is another notable example of a “NoSQL database.” By using the peer to peer gossip communication protocol, Cassandra enables every node in the cluster to communicate state information about itself and other nodes.</p><p>More details can be found in the official documentation here:<a href="http://cassandra.apache.org/">http://cassandra.apache.org/</a>. It’s also important to point out that Cassandra is a distributed open source database so we encourage you to contribute as well.</p><p><strong>ElasticSearch</strong><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html"><br></a>Equipped to handle large amounts of data, ElasticSearch is an open source analytics and full text search engine. It’s often used to enable search functionality for web applications as well. It does this by facilitating shard allocation and cluster-level routing. Indexes are split into shards and every shard is a self contained instance of Apache Lucene. Shards are spread across multiple nodes so when more capacity is needed, one can simply add more machines so that the load can be spread more efficiently. By communicating what document you’re interested in with a given server on your cluster for ElasticSearch, it can hash that into a particular shard ID. A mathematical function is performed that can very quickly figure out which shard owns the queried document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*0hn116qThOR9C2XV.png" /><figcaption>Sundog Education with Frank Kane</figcaption></figure><p>In the example above we can view how ElasticSearch maintains resilience to failure. If node 1 fails we’ll lose primary shard 1, and replica shard 0. If this took place, ElasticSearch would elect one of the replica nodes on 2 or 3 to be the new primary. We can then continue to service read requests and accept new data.</p><p>Learn more at <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html">Elastic.co</a>.</p><p><strong>Redis<br></strong>Redis cluster is a data sharding solution with automatic management, failover and replication. In cluster mode all nodes are communicating with each other. Note that every redis cluster node requires two TCP connections open, the standard Redis port used to serve a client plus the port obtained by adding 10,000 to the TCP port. “Redis Cluster uses a master-slave model where every hash slot has from 1 (the master itself) to N replicas (N-1 additional slaves nodes).” In the example below, there is a cluster of three masters, A, B, C, each with 1 slave used to replicate each master. The final cluster is then composed of A, B, C that are masters nodes, and A1, B1, C1 that are slaves nodes, the system is able to continue if node B fails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*wBMG68jEZFmQyk2s.png" /></figure><p>Learn more at Redis.io: <a href="https://redis.io/topics/cluster-tutorial">https://redis.io/topics/cluster-tutorial</a>.</p><h3>Big Data and high throughput systems</h3><p><strong>Apache Kafka<br></strong>Streaming technology processes new data as it’s generated into your cluster. Instead of dealing with it in batches, big data can be handled in real-time and published directly to your cluster. Kafka streams is an easy data processing and transformation library that offers data parallelism, distributed coordination, fault tolerance, and operational simplicity. Kafka servers can be set up so that they store messages from publishers and publish them to anyone that wants to consume them. Messages are associated with topics and they represent a specific stream. A consumer subscribes to one or more topics and receives data as it’s published. Kafka can then be spread out on a cluster of its own avoiding a single point of failure. Multiple servers can then run multiple processes and distribute the storage of the data on your topics and the processing of all the producers and consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*1FcQ-EqQDd8F0fwj.png" /></figure><p>- The anatomy of an application that uses the Kafka Streams library. (kafka.apache.org)</p><p>Learn more at <a href="https://kafka.apache.org/23/documentation/streams/architecture">https://kafka.apache.org/23/documentation/streams/architecture</a></p><p><strong>Apache Spark</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*SZj9v4ffUCn4SKeW.png" /></figure><p>The Spark framework is used for the distributed processing of large datasets. A driver program is developed and built around the SparkContext object. A cluster manager is ran on top of that which is responsible for distributing the work defined by the driver script among multiple nodes. The node might have an executor process with its own cache and list of tasks which is then given back to the cluster manager which is responsible for deciding what happens next.</p><p>Learn more at <a href="https://spark.apache.org/docs/latest/cluster-overview.html">https://spark.apache.org/docs/latest/cluster-overview.html</a>.</p><h3>Distributed systems</h3><p><strong>Bitcoin and Ethereum clients<br></strong>Blockchain technology has changed the way we think about data and privacy. Notably, blockchains allow all actors on the network to have access to the entire data set on their computer. Ethereum is a distributed public blockchain network like Bitcoin, however, they have different purposes and capabilities. In both cases, the nature of this distributed network enables high availability. If one node fails, all of the other nodes in the network are equipped with the full data set, therefore bringing down the network would require bringing down every node providing computation power.</p><p><strong>Light clients and quick response time<br></strong>Light clients expose a browser to the web3 interface that allows decentralized applications to access the Ethereum network or other distributed systems. These are useful if one is operating under volume constraints. In order to achieve quick response time light clients do not interact directly with the blockchain. Instead, they use full nodes as intermediaries. Another advantage is that it takes significantly less resources and storage in comparison to running a full node to run a light node without comprising that much security. Currently, Parity Ethereum and Geth are the most popular light clients. As mentioned by the Parity team, “They help users access and interact with a blockchain in a secure and decentralized manner without having to sync the full blockchain.” Learn more about light clients here: <a href="https://www.parity.io/what-is-a-light-client/">https://www.parity.io/what-is-a-light-client/</a></p><p><strong>The future of HA<br></strong>The emergence of blockchain technology and development of distributed systems has introduced several capabilities that we’re still creating use cases for. Today, mobile applications can run offline. As the demand for highly available applications increases, offline applications like Scuttlebutt are redefining the term itself. Unlike Twitter or Discord, there is no social separation created by those who run servers and those who use the application. Instead, every user runs the application directly on the device. There are some tradeoffs as messages can only be passed directly between friends via peer-to-peer. As we contemplate the future of high-availability, it’s a good idea to take a closer look at events such as Fortnite’s final showdown. The entire game was down and although no one could play, millions still remained connected waiting for the rumored new map to be uploaded. This poses the question — is high availability the end all goal?</p><p><em>Originally published at </em><a href="https://whiteblock.io/high-availability-in-distributed-systems/"><em>https://whiteblock.io </em></a><em>on November 25, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d3325e4ee33" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whiteblock/high-availability-in-distributed-systems-2d3325e4ee33">High Availability in Distributed Systems</a> was originally published in <a href="https://medium.com/whiteblock">Whiteblock</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Whiteblock at Devcon 5]]></title>
            <link>https://medium.com/whiteblock/draft-1-5c6b272556e9?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c6b272556e9</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[whiteblock]]></category>
            <category><![CDATA[devcon5]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Fri, 18 Oct 2019 14:47:46 GMT</pubDate>
            <atom:updated>2019-10-25T19:32:01.888Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*noWCD9sToH17ep5B.jpg" /></figure><p>Just last week, Whiteblock attended Ethereum’s premier community event, <a href="https://devcon.org/">Devcon 5</a>. This year saw over 3,000 crypto folks descend on Osaka, Japan for a week of technical talks and workshops largely aimed at developers. Myself, Rene and Trent represented the rest of the Whiteblock team, who have been heads down working on Genesis, our distributed systems testing platform.</p><p>Read on for my retelling of event takeaways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LmkdDvPe_nm6GgSG.jpeg" /></figure><p><strong>Monday — before it happened</strong></p><p>We decided to meet and work and visit Osaka together. First stop was the fish market, where we ate at small stands all sorts of local delicacies: unagi, shrimp, salmon sashimi, tempura, tokoyaki — just hot enough to burn your mouth on. We then walked to a Hawaiian-themed cafe in the Dontonburi district. There we worked on slides and discussed the agenda. Later on, Oskar and Dean from Status joined us and we worked there until we were kicked out by the Japanese staff — probably for being too loud.</p><p>We proceeded to ramble down the busy street lined up with restaurants, taking a second to visit an arcade where we witnessed how staking works in real life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9-TLMT9hiQ90iQPU.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*n-g-Mu-77keYmeVU.jpg" /></figure><p>Dean saw a Gucci hat, but could not bring himself to buy it. Later during the conference, he would convert to a pink Andrew Yang hat, and a fancy cap from the Solana cycling team. We eventually went to a Yakiniku place where we all ate delicious meats and filled up on curry and rice. In the afternoon, we walked among the streets up to a Starbucks, where we worked some more, before heading to our respective hotels. Trent pushed to the venue and scoped out a few interesting elements.</p><p><strong>Day 1</strong></p><p>We showed up bright and early, and just a little jetlagged to a large labyrinthian conference hall. After multiple false starts, we found our way to a large hall where we found no coffee. We had been contacted the week prior to participate to a P2P summit, so we headed to B10. We sat down on the 3rd row. Little did we know that we’d be seated there, our mouths agape, for the next 4 hours.</p><p>We’re network geeks, and we saw some of the <a href="https://twitter.com/ponarunkumar/status/1181371998707994624">best projects out there</a> presenting in lightning talk format. It kept on piling on. We gave our own version — <a href="https://drive.google.com/file/d/1JGvc2PlUcTSFNC-iExFvmgSCkjWjswAR/view?usp=sharing">available here</a> — that we modified to give a nod to the previous talk by Matt Slipper on <a href="https://www.scuttlebutt.nz/">SecureScuttlebutt</a>. The last half hour was interesting as it showed a lot of convergence with all the actors in the space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bpPJwHUyC4pYPYI2.jpg" /></figure><p>In the afternoon, we started collecting stickers in the sponsor hall. We have quotas to respect and insatiable coworkers who need their fix. We met a lot of friends along the way. Rene and Trent also sat in on the ETH Magician’s Roadmap session for a few of the topics.</p><p><strong>Day 2</strong></p><p>Wednesday was keynote day. A combination of fervor, crowd size and <a href="https://twitter.com/internetarchive/status/1181738069747105793?s=20">loud drum noises</a> drove me out to safety and gave me the satisfaction of enjoying plenty of coffee right outside the doors. From there, meetings piled on, but I managed to squeeze some time to get into Raul’s excellent talk and showcase of libp2p documentation on gossipsub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rEPaQGGoZeo7m_FF.jpg" /></figure><p>I left the conference with the distinct impression that I’d lost a bone in my foot, which doubled as a stone in my shoe.</p><p><strong>Day 3</strong></p><p>We set out to be present for the ETH 2.0 client summit. where each of the clients would go one by one and explain their particulars. We particularly enjoyed the talks from <a href="https://twitter.com/trent_vanepps/status/1182098232454742017">Joe Delong</a> and <a href="https://twitter.com/_tmio/status/1182108125920292864">Preston</a>.</p><p>The one thing I asked during Q&amp;A: are we ready for the multi-client testnet? Can Whiteblock get started? We’ve been pushing hard on our gossipsub testing framework, and it’s about time we offer a proper CI/CD pipeline. We’ll talk more about this soon.</p><p>The afternoon was spent in countless meetings and demos, along with a few intriguing discussions along the way.</p><p><strong>Day 4</strong></p><p>Started the day listening to Alistair talk about Polkadot, which was awesome. Unfortunately, I had to exit the room and message Rene, as she was reporting her tram was blocked due to the typhoon preparations. It turned out to be a dud in the end. I then spent some time with Dmitryi trying to drink a weird beverage closed by a marble — a <a href="https://www.google.com/search?q=ramune+drink&amp;oq=ramune+drink&amp;aqs=chrome..69i57.1918j0j1&amp;sourceid=chrome&amp;ie=UTF-8">Ramune</a>. 3/10 I need drinks to not pull pranks on me.</p><p>We then went to a nice room with 40 wonderful folks here to learn about Ethereum 2.0 networking. Part of this is related to our grant work supported by the Ethereum Foundation and Consensys. We delivered a 20-minute presentation <a href="https://drive.google.com/file/d/1UhDfNp4KYoYGGpA6SLgSvndEKB6Oo4RW/view">available here,</a> followed by insightful questions from the audience. How has networking in ETH 2.0 came to be — maybe worth a story of its own. We also learned Trinity has a discv5 solution we can start testing!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*78LNTiEHi4wymN-g.jpg" /></figure><p>I ran for the fifth time to the sponsor hall, grabbed a libp2p T-shirt (thanks Raul!) and an Infura hoodie (yay Dani!). At the same time, I received a notification that my flight had been canceled. I therefore bought a set of tickets to SFO:</p><ul><li>KIX&gt;HND&gt;SFO (changing my return flight)</li><li>KIX&gt;TAO&gt;SFO (through Eastern China Airlines)</li><li>KIX&gt;TPE&gt;CDG (wrong direction, but I have to get to Paris soon anyway)</li></ul><p>While cooling my credit card on a pack of soba, I listened to the panel discussing the state of P2P.</p><p>Our last event during the venue’s closing hours was a workshop for the DevCon scholars (or like Coogan couldn’t stop himself saying, the kids).</p><p>My hat’s off to the organizers, the “kids” were energized and eager to learn. I bashed their brains with 40 minutes of content on our testing methods. Given the poor Wifi, we’ll need to follow up and give them access to our environment so they can run tests from there. We were excited to see them get it right away and ask insightful questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*63UYuRGC2Zm9b2hw.jpg" /></figure><p>On the steps of the venue, under a menacing sky, we bid adieu to each other.</p><p>Rene joined <a href="https://twitter.com/helsfoftroy/status/1182560491601686528">a techno parade</a> of Louis Vuitton models, Trent disappeared in the bowels of the crypto community, and I ran off to the station. From there, I took a Shinkanzen to Fukuoka, mostly standing for 4 hours. Fukuoka was just right outside the mess. The next day, I flew to Seoul, slept in the airport almost comfortably, and flew home after using my baggage allowance for duty free.</p><p>Thanks for reading along to the end. Our team had an amazing experience making new connections within the community, we’re already looking forward to Devcon 2020! For more recaps on the event, read <a href="https://twitter.com/LefterisJP/status/1182865924728086529">this summary</a> from Lefteris of Raiden. Or perhaps, a more <a href="https://coinjazeera.news/massive-hurricane-hits-devcon-5-as-god-attempts-to-rid-the-world-of-ethereum-community/">light-hearted take</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c6b272556e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whiteblock/draft-1-5c6b272556e9">Whiteblock at Devcon 5</a> was originally published in <a href="https://medium.com/whiteblock">Whiteblock</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Peer to peer applications]]></title>
            <link>https://medium.com/@jointhemachine/peer-to-peer-applications-5ce9c1231a79?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ce9c1231a79</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[autonomy]]></category>
            <category><![CDATA[peer-to-peer]]></category>
            <category><![CDATA[humanism]]></category>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Sun, 14 Apr 2019 22:31:00 GMT</pubDate>
            <atom:updated>2019-04-14T22:31:00.653Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*dv_nNeXqKsHw7wByGop1TA.png" /></figure><blockquote>Humanism is a philosophical and ethical stance that emphasizes the value and agency of human beings, individually and collectively, and generally prefers critical thinking and evidence (rationalism and empiricism) over acceptance of dogma or superstition.<br>Wikipedia, humanism</blockquote><p>When the Internet initially emerged, it formed as a peer to peer network. When the Internet settled on using html and pastiche style, complete with blink tags, visitor counters and custom cursors, the custom was to link from one site to another. When the blogging movement took hold, those were called blog rolls.<br>Google initially built PageRank by crawling and finding the number of links between sites to rank them by popularity.</p><p>All this to say — when the Internet emerged and drew its first breaths, it was driven by humans connecting each other organically.</p><p>Fast forward twenty years and we are seeing a new generation of digital natives capture the Internet for themselves. They use centralized services such as Google, Facebook, WhatsApp or Telegram all day. They started chatting over Fortnite (or Counterstrike for the old geezers), took it to Discord, or Slack.<br>This population considers those services as a given and longs to use computers to help them in more constructive ways than being note takers or messaging services. You can see a stammering of that future through connected gadgets, smart watches or Alexa speakers.<br>None of them extend to the goal that will drive the 21st century: extending the aim of human agency to our ideals and actions forward.</p><h3>What is a peer to peer application?</h3><p>Peer to peer applications have a few defining traits. They generally involve multiple actors, have no central binding mechanism and offer a degree of ownership and control over data and decisions.<br>Napster, eMule, Bittorrent are well known peer to peer applications. They rely on central resources to help join the network and beacon resources, but data is exchanged directly between peers.<br>Massive multiplayer games sometimes rely on peer to peer networks for faster throughput.<br>IoT networks are established as meshes. There is a lot of subtlety there, between the use of a gateway, passive sensors or connected gadgets.<br>And of course, up and to the right on the magic quadrant, blockchain applications are paramount peer to peer applications, with complete control and ownership of data and transactions (at the expense of forking), and a complete lack of trust between peers.</p><h3>Peer agency, the reset and future of the web</h3><p>Centralized platforms have shown it is easier to monetize user interactions by leveraging their network effect ; they even go as far as making the product free for use by individual users, transacting and earning through the metadata exposed by the user interactions.<br>Peer to peer applications are a complete reboot of the value chain ; in the first place users have to find value in using the application for themselves, and find value in transacting with others on the network. The value of the individual interaction should be magnified by the network, but originates from the user.<br>It is my expectation that those applications will provide functionalities that increase the user agency, by extending their reach into the physical world, opening new ways to interact with physical or reality augmented objects, gather data through pictures or instrument readings, garnering intelligence and insights to help feed into decision models.<br>We will look into distributed applications through the lens of peer agency and develop use cases for those.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ce9c1231a79" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Honoured to be a member of the ASF]]></title>
            <link>https://medium.com/@jointhemachine/honoured-to-be-a-member-of-the-asf-d9bf5c74e43b?source=rss-9e3f4ade4009------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9bf5c74e43b</guid>
            <category><![CDATA[afs]]></category>
            <category><![CDATA[tuweni]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Antoine Toulme]]></dc:creator>
            <pubDate>Sun, 14 Apr 2019 22:06:08 GMT</pubDate>
            <atom:updated>2019-04-14T22:11:40.707Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E_XuOMhntNXkcl7_TUGAiw.png" /></figure><p>I have worked in open source projects right from the start of my career.</p><p>Over time when I look at code on the Internet, I quickly dismiss biases, technical debt or other shortcomings to start wondering: will this code still work in a year? Will it be possible to build with it?</p><p>Sometimes, the answer is no, and I make a willful decision to engage, contribute, smooth the rough angles of the project.</p><p>Most of the time, when the answer is yes, it’s because the project is run by the Apache Software Foundation (with the notable exception of Jackson through the immense work of Tatu).</p><p>I have had chances to engage with the Apache Software Foundation. By engaging on my own time, and helping to push releases through, I proved my worth and became a committer on <a href="https://buildr.apache.org/">Apache Buildr</a>. A few years down the road, I helped the project in an administrative capacity as chair.</p><p>I learned a good deal about open source practices, but also first and foremost communications, netiquette and collaboration on public mailing lists. I enjoy working and being challenged by others.</p><p>Fast forward this year, and I’m pushing to donate code I worked on at ConsenSys under the name Cava to the incubator. <a href="https://tuweni.apache.org/">Apache Tuweni</a> was born from a thread that last about 2 months and churned quite a few proposals (I actually lost count by now).</p><p>I was invited this year to continue to serve the foundation as a member. For me this is a huge deal. A watershed moment where I look back at ten years of work and open source. I am now in a place where I can help others and make software and open source awesome, and I am grateful for the opportunity.</p><p>My thanks to all the folks who showed me the way: <a href="https://twitter.com/boia01">Alex Boisvert</a>, <a href="https://twitter.com/lookfirst">Jon Stevens</a> and <a href="https://twitter.com/jimjag">Jim Jagielski</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9bf5c74e43b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>