<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rodolphe Marques on Medium]]></title>
        <description><![CDATA[Stories by Rodolphe Marques on Medium]]></description>
        <link>https://medium.com/@rodmar?source=rss-4dbbb277a0a3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CkxV8W5qoc6tX2mkw7xBAw.jpeg</url>
            <title>Stories by Rodolphe Marques on Medium</title>
            <link>https://medium.com/@rodmar?source=rss-4dbbb277a0a3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:36:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rodmar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Filecoin meets Nevermined]]></title>
            <link>https://medium.com/nevermined-io/filecoin-meets-nevermined-43179a610ae2?source=rss-4dbbb277a0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/43179a610ae2</guid>
            <category><![CDATA[filecoin]]></category>
            <category><![CDATA[nft-marketplace]]></category>
            <category><![CDATA[decentralized-storage]]></category>
            <category><![CDATA[ipfs]]></category>
            <category><![CDATA[access-control]]></category>
            <dc:creator><![CDATA[Rodolphe Marques]]></dc:creator>
            <pubDate>Fri, 11 Jun 2021 10:49:23 GMT</pubDate>
            <atom:updated>2023-03-08T10:24:48.991Z</atom:updated>
            <content:encoded><![CDATA[<h4>Leverage the benefits of Filecoin’s decentralized storage with Nevermined’s access control thanks to our latest integration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*30bM3C6CL0xFkOJ7b4cung.png" /><figcaption>Nevermined integrates Filecoin into its tech stack thanks to a recent grant</figcaption></figure><p>Since its inception, the core feature of Nevermined has always been data sharing. Over time we have enhanced the capabilities and services that you can provide on your data through Nevermined (<a href="https://docs.nevermined.io/docs/architecture/specs/Spec-COMPUTE/">Data In-Situ Compute</a>, <a href="https://docs.nevermined.io/docs/architecture/specs/Spec-FL/">Federated Learning</a>, <a href="https://docs.nevermined.io/docs/architecture/specs/Spec-PROVENANCE">Provenance</a>, <a href="https://docs.nevermined.io/docs/architecture/specs/Spec-NFT">NFTs</a>) but <a href="https://docs.nevermined.io/docs/architecture/specs/Spec-ACCESS">Decentralized Access Control</a> has always been a core part of our architecture.</p><p>One of the design principles of the Nevermined Access Control is that the data owner should never lose control over its data. We achieve this by never storing any data on Nevermined itself. Instead, we integrate with existing data storage solutions in order to make it as easy as possible for data providers to use Nevermined. Current integrations include centralized technologies like Amazon S3, Azure, On-premise, as well as decentralized solutions like IPFS. In this blog post, we will give an overview of how we integrated Nevermined with Filecoin (for a full technical specification see <a href="https://docs.nevermined.io/architecture/integrations/filecoin/filecoin-storage/">Filecoin Storage</a>).</p><h3>Nevermined Storage Providers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*xaNkxaCb1XiibTAZ" /><figcaption>Filecoin Storage Provider</figcaption></figure><p>At the center of every client interaction with Nevermined is the Gateway. In this context, the gateway works as an enforcer for the Access Control, by checking if both the client and the data provider fulfilled the access conditions previously defined on a service agreement. If everything is correct the Gateway fetches the data and forwards it to the client.</p><p>The client never gains any information of where the data is stored. This has two advantages: first the client is never able to share the location of the data; and second the location of the data or the storage solution where the data resides is transparent for Nevermined and the Gateway is always able to fetch the data for storage solutions with which it has integrations with.</p><p>The way that Nevermined provides integration with different storage solutions is through what we call Storage Providers. Storage Providers are code components that can be enabled in the Gateway. The Gateway is able to decide which service provider to use based on the information included in the URI for the data.</p><h3>Nevermined Filecoin Specification</h3><p>For Filecoin we developed a <a href="https://github.com/nevermined-io/metadata-driver-filecoin">Filecoin Storage Provider</a> that enables Gateway connectivity with the Filecoin network via <a href="https://docs.textile.io/powergate/">Powergate</a>.</p><p>We chose to use Powergate because of the API-driven solution they provide for storage across Filecoin and IPFS, and since we already supported IPFS this gives data providers the flexibility to take advantage of the cold and hot storage solution provided by Powergate by leveraging both IPFS and Filecoin. It also makes it easier for data providers to manage data under multiple addresses.</p><p>From a usability point of view, the only thing that changes for a data provider is how they define the URI that points to the data in the definition of the asset metadata.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b40159d19952ed292931afbeb158af50/href">https://medium.com/media/b40159d19952ed292931afbeb158af50/href</a></iframe><p>In case of files stored in Filecoin the URI starts with the `cid://` schema and can include additional information in order to take advantage of different features of Powergate:</p><ul><li><strong>CID Hash</strong> — Identifier of the content in the Filecoin network</li><li><strong>Powergate Host</strong> — Hostname of the powergate node that can be used to fetch the file</li><li><strong>Powergate Port</strong> — Port where is running the powergate service that can be used to fetch the file</li><li><strong>Powergate Token</strong> — Token to use to fetch the file</li><li><strong>Deal Id</strong> — Identifier of the Deal that allow to pin the file to IPFS</li></ul><p>Some example of Filecoin URIs include:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b27e1fe16c1c47d9c0197f2b6903e5b/href">https://medium.com/media/8b27e1fe16c1c47d9c0197f2b6903e5b/href</a></iframe><h3>Filecoin Decentralized Access Control with Nevermined</h3><p>With the Nevermined Filecoin integration users can now have granular access control over who can access the data. The entire Nevermined stack is now capable of using Filecoin and this does not include only the Gateway and the fetching of the data but also the possibility of uploading data directly to Filecoin through the SDKs.</p><p>To demonstrate the Filecoin integration we used it in our <a href="https://github.com/nevermined-io/cryptoarts">experimental NFT Marketplace</a>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F02aXym4DFZ4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D02aXym4DFZ4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F02aXym4DFZ4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e2504e11810ffc4ef43f8540d7f0de7/href">https://medium.com/media/5e2504e11810ffc4ef43f8540d7f0de7/href</a></iframe><p>This video shows how a user could upload a file directly to Filecoin through Nevermined (in this case using the <a href="https://github.com/nevermined-io/sdk-js">Nevermined Javascript SDK</a>).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_Vvc0dBB-4M&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_Vvc0dBB-4M&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2F_Vvc0dBB-4M%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a339ef37a28b3439079c6f770044216/href">https://medium.com/media/7a339ef37a28b3439079c6f770044216/href</a></iframe><p>This video shows how a user could specify a file that already exists in Filecoin.</p><h3>Conclusion</h3><p>By integrating Filecoin into Nevermined we are able to enhance the features of Filecoin by providing granular access control on top of your data stored on Filecoin. Not only that but data stored on Filecoin immediately benefits from all the current and future features of Nevermined, including compute to the data, full data provenance through W3C provenance standard, a decentralized storage solution for NFTs as well as enhancing the economics of the ecosystem by combining it with the data monetization features of Nevermined.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43179a610ae2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nevermined-io/filecoin-meets-nevermined-43179a610ae2">Filecoin meets Nevermined</a> was originally published in <a href="https://medium.com/nevermined-io">Nevermined</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nevermined & Credit Card Fraud Detection]]></title>
            <link>https://medium.com/nevermined-io/nevermined-credit-card-fraud-detection-91aef362d98?source=rss-4dbbb277a0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/91aef362d98</guid>
            <category><![CDATA[data-monetization]]></category>
            <category><![CDATA[data-sharing]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[data-in-situ-computation]]></category>
            <category><![CDATA[federated-learning]]></category>
            <dc:creator><![CDATA[Rodolphe Marques]]></dc:creator>
            <pubDate>Fri, 08 Jan 2021 12:10:58 GMT</pubDate>
            <atom:updated>2023-03-08T11:02:39.721Z</atom:updated>
            <content:encoded><![CDATA[<h3>Nevermined Use Case: Credit Card Fraud Detection</h3><h4>Leverage the power of Federated Learning with Nevermined</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b9PlXGgaCZ5jFqDCh8CW9w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@rupixen?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">rupixen.com</a> on <a href="https://unsplash.com/s/photos/credit-card?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><a href="https://nevermined.io/">Nevermined</a> offers its users the ability to build data-sharing ecosystems where untrusted parties can share and monetize their data in a way that’s efficient, secure and privacy-preserving.</p><p>In this blogpost, we will be showing how we can use <a href="https://www.nevermined.io/">Nevermined</a> to solve the <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Kaggle</a> Credit Card Fraud Detection problem. In doing so we will be showcasing several of Nevermined capabilities:</p><ul><li><em>Data Sharing</em> — Enables the sharing and access of digital assets between untrusted parties in the data ecosystem;</li><li><em>Data In Situ Computation (DISC)</em> — Allows the execution of models and algorithms without moving the data (i.e. “in situ”);</li><li><em>Federated Learning</em> — With its agnostic capabilities, Nevermined lets you pick the appropriate Federated Learning frameworks to work across multiple datasets at the same time. Extract value and insights where you couldn’t before.</li></ul><h3>Use Case</h3><p>As a baseline, we are going to use the Kaggle Credit Card Fraud Detection <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">dataset</a> and enhance it into a Federated Learning scenario.</p><p>One of the biggest problems when trying to train a model is finding large amounts of data. Now imagine we have two banks: Bank A and Bank B. Both of these banks have large amounts of credit card transactions that they aren’t really leveraging for machine learning purposes. They either lack the data science expertise or they are afraid of the penalties that may arise from mishandling the data under data protection laws.</p><p>Now we introduce a third party: a Data Scientist. The Data Scientist wants to build a service that can automatically flag fraudulent credit card transactions with a high degree of accuracy. Knowing the typical features that should be present in every credit card transaction, he already built a model using <a href="https://scikit-learn.org/stable/index.html">scikit-learn</a>.</p><p>Now the biggest challenge becomes getting access to real data so that the data scientist can use it to train the model to be able to make predictions with a high degree of accuracy. Banks won’t and can’t share this type of data with third parties, so the data scientist hits a roadblock.</p><p>Currently, there exists the capability to perform privacy-preserving machine learning through <em>Federated Learning</em>. Federated Learning can be seen as a form of privacy-preserving distributed machine learning. A machine learning model is trained in parallel over multiple data sets. The individual weights resulting from the training over each dataset are then aggregated in a privacy-preserving manner and this produces the final trained model.</p><p>However, orchestrating this type of analytical application is difficult. Orchestrating this type of analytical application across disparate datasets controlled by independent counterparties is virtually impossible.</p><p>But not anymore!</p><h3>Nevermined to the Rescue!</h3><p>To summarize our use-case:</p><ul><li>We have a data scientist that requires a large amount of private data to train a model</li><li>We have two distinct banks, Bank A and Bank B, with their own distinct infrastructure and data assets, and they would like to monetize their data</li><li>We need a system that allows for: interoperable data sharing between multiple parties; data monetization; an interface to run code spanning multiple organizations</li></ul><p>With Nevermined’s <em>Data Sharing </em>capabilities, both Bank A and B can both advertise their assets and monetize their data without having to move or change their existing infrastructure, this turning them into data providers. And Nevermined’s <em>Data In Situ Computation</em> (DISC) capability allows for consumers of the data, like the Data Scientist, to run generic computations (inside the scope allowed by the data providers) without ever having to access the data directly.</p><p>This means that private data remains private as it does not need to move. Now the Data Scientist has a single interface that he can use to train his model over data from multiple, wholly distinct and independent organizations. All the heavy lifting of orchestrating a Federated Learning session spanning multiple organizations is handled by Nevermined. This means the Data Scientist only needs to reuse his existing machine learning code and tools with little to no modification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*g_LtR1cAoiXVQnnxfvdyLQ.jpeg" /></figure><p>In the next sections we will describe how, through Nevermined, the following can be achieved:</p><ul><li>Data Providers can publish and advertise their data;</li><li>Data Consumers can pay for access to that data;</li><li>Data Consumers, like data scientists, can reuse their existing machine learning models to run Federated Learning over multiple, disparate datasets.</li></ul><h3>Publishing the data</h3><p>In order to make the data available and advertisable through Nevermined, both Bank A and Bank B first need to publish the data on Nevermined.</p><p>Below is a snippet of code from <a href="https://github.com/nevermined-io/fl-demo/blob/master/fraud-detection-xain/notebooks/nevermined_provider.ipynb">nevermined_provider.ipynb</a>. In order to publish a dataset we need to define some metadata describing what kind of asset this is.</p><p>In this example we define all of the required attributes, but in reality the metadata can have any number of additional attributes. One thing to note is that we are publishing a file, setting a price, and marking it as a dataset. We then publish this metadata to Nevermined using ‘create_compute’. This will create a compute type dataset meaning that it can only be used to run computations against, and cannot be accessed directly.</p><p>This is all each bank needs to do to publish their datasets on Nevermined.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53ac592a4ee87d0ac1f77b43721753b7/href">https://medium.com/media/53ac592a4ee87d0ac1f77b43721753b7/href</a></iframe><h3>Consuming the data</h3><p>Now that the data is published, the Data Scientist discovers both bank’s datasets through a marketplace (more on this in a later article) and decides that he would like to train his machine learning model over these two datasets. But before doing this, however, he needs to pay the required price in order to get permissions to run computations over the data.</p><p>Below is a snippet of code from <a href="https://github.com/nevermined-io/fl-demo/blob/master/fraud-detection-xain/notebooks/nevermined_consumer.ipynb">nevermined_consumer.ipynb</a>. The Data Scientist orders the data asset represented by the DID (Decentralized Identifier), which was generated earlier by Bank A during their asset publishing process.</p><p>Since a single asset in Nevermined can provide multiple services, we also need to specify which service we are ordering; in this case, it’s the <em>computing service</em>. If the order is successful, payment is facilitated via Nevermined and a unique service agreement is returned. The Data Scientist will later use this service agreement to request the execution of an algorithm over the data (the service he just ordered).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d8acb5ec8915082f96db4c54a5df33f/href">https://medium.com/media/8d8acb5ec8915082f96db4c54a5df33f/href</a></iframe><h3>Running computations over the data</h3><p>Up to this point, the Data Scientist has bought compute access to two datasets and now wants to use Federated Learning to train his model over both datasets.</p><p>Nevermined is agnostic to the computations or frameworks used on its execution environment. For this example, we are going to be using <a href="https://github.com/xaynetwork/xaynet/tree/v0.8.0">Xaynet Federated Learning Framework v0.8.0</a> together with <a href="https://scikit-learn.org/">scikit-learn</a>.</p><p>Below is a snippet of code from <a href="https://github.com/nevermined-io/fl-demo/blob/master/fraud-detection-xain/notebooks/federated_fraud_demo.ipynb">federated_fraud_demo.ipynb</a>. The Xaynet framework requires us to define the participant code. We start by preparing the data for training and creating the model we want to use. Next, we define the ‘train_round’ method that receives as input the weights of the global model sent by the coordinator.</p><p>We initialize the model with these weights and then we perform normal training using the scikit-learn model. In the end, we return the new weights as a result of the training round. The ‘serialize’ and ‘deserialize’ methods are just helper functions to serialize and deserialize the weights from and into <a href="https://numpy.org/doc/stable/reference/generated/numpy.array.html">NumPy</a> arrays. Finally, we start the participant which connects to the coordinator and waits for instructions.</p><p>This code will be run over both banks’ datasets.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efa95af220a259f5dfb89289ab7e7ffe/href">https://medium.com/media/efa95af220a259f5dfb89289ab7e7ffe/href</a></iframe><p>Once we have our code we need to host it (ex: on github) and then we need to publish it on Nevermined in the same way we published the banks’ datasets.</p><p>Below is a snippet of code from <a href="https://github.com/nevermined-io/fl-demo/blob/master/fraud-detection-xain/notebooks/nevermined_consumer.ipynb">nevermined_consumer.ipynb</a>. In the metadata we provide the information about the file to execute, we mark the asset as ‘algorithm’ and we specify both the ‘requirements’ for execution and the ‘entrypoint’. In the requirements we can set the Docker image that we want to use and in the entrypoint we define how the algorithm should be executed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7b17ca51b2ae06e57c1793df97add43/href">https://medium.com/media/c7b17ca51b2ae06e57c1793df97add43/href</a></iframe><p>Now that we have the computation that we want to run over the datasets we need to define the workflow. A workflow is a set of instructions to Nevermined describing how the computation should be carried out. Nevermined uses these instructions to configure the execution environment.</p><p>Below is a snippet of code from <a href="https://github.com/nevermined-io/fl-demo/blob/master/fraud-detection-xain/notebooks/nevermined_consumer.ipynb">nevermined_consumer.ipynb</a>. Defining a workflow is similar to defining a job in a CI like Github actions or CircleCI. In our case we have only one stage with one input which is the dataset from one the banks, and one transformation, which is the code we want to run.</p><p>We create one workflow for each bank.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3743e3299503ae59412da124deef9c9e/href">https://medium.com/media/3743e3299503ae59412da124deef9c9e/href</a></iframe><p>And finally we order Nevermined to execute the computations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a60ebc9ea7354640a66b8fdb8b947f20/href">https://medium.com/media/a60ebc9ea7354640a66b8fdb8b947f20/href</a></iframe><p>With this last step our algorithm will run on the execution environment of each bank, and the results will be published and made accessible to the Data Scientist.</p><p>Note that in the entry point of the algorithm we are using <a href="https://papermill.readthedocs.io/en/latest/">papermill</a> to execute the <a href="https://jupyter.org/">Jupyter Notebooks</a>. This shows how flexible Nevermined is in allowing data scientists to reuse tools they are already familiar with.</p><p>The result outputs will be two notebooks (one for each bank) and the global model. The notebooks are returned with each cell filled with the output as a result of the execution. The final aggregated model is returned as a serialized numpy array.</p><blockquote>Update 11/02/2021: We have recorded a video showing the demo in action.</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F857rj_bfkKw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D857rj_bfkKw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F857rj_bfkKw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/171fec4f30b8b1dbabe1474b18c626d1/href">https://medium.com/media/171fec4f30b8b1dbabe1474b18c626d1/href</a></iframe><h3>Conclusion</h3><p>In this article, we showed how a Nevermined data ecosystem can allow for easy interoperability between data providers and data scientists. They can run complex distributed machine learning tasks spanning multiple different organizations and do it in a way that is privacy-preserving and within the allowed scope of data protection laws.</p><p>You can find the full demo explained here and others in our <a href="https://github.com/nevermined-io/fl-demo">Github repo</a>, get a deeper understanding of Nevermined through our <a href="http://docs.nevermined.io/">documentation</a> and reach out to us for any questions on our <a href="https://discord.gg/mNd4dFVv">discord server</a>.</p><blockquote>If you would like to know more about the commercial opportunities Nevermined makes available for your organization, or are interested in having a demo, please reach out to us on info@nevermined.io or visit our website.</blockquote><blockquote>If you enjoyed this article let us know by clapping or leaving your comment below. Thanks to <a href="https://medium.com/u/8ab281366479">Aitor Argomániz</a>, <a href="https://medium.com/u/9104a5aaeb71">Don Gossen</a> and <a href="https://medium.com/u/69a316bf9043">Jesse Steele</a> for the editorial support</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91aef362d98" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nevermined-io/nevermined-credit-card-fraud-detection-91aef362d98">Nevermined &amp; Credit Card Fraud Detection</a> was originally published in <a href="https://medium.com/nevermined-io">Nevermined</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sensei guiding the Way]]></title>
            <link>https://medium.com/keyko/sensei-guiding-the-way-864e5cf6fa0c?source=rss-4dbbb277a0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/864e5cf6fa0c</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[data-ecosystems]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[virtual-events]]></category>
            <dc:creator><![CDATA[Rodolphe Marques]]></dc:creator>
            <pubDate>Thu, 19 Nov 2020 13:57:41 GMT</pubDate>
            <atom:updated>2020-11-19T13:57:41.405Z</atom:updated>
            <content:encoded><![CDATA[<h4>into the Odyssey Momentum: a virtual hackathon experience</h4><p>Odyssey is a non-profit online collaboration environment that connects governmental, corporate, and nonprofit partners with innovative ideas to collaboratively address complex 21st-century challenges. On November 13, they opened the doors of the Odyssey hackathon, one of the most important global events in the blockchain space.</p><p>This year, the Odyssey hackathon was rebranded and redesigned as Momentum, a unique, purpose-built Online Mass Collaboration Arena, which combines the virtual elements of a video game with meeting design in order to provide a fully immersive 3D experience.</p><h3>Our social experience — platform, interaction with other participants, jury, etc.</h3><p>The event happened in an extremely creative and complex virtual space, built for entertainment and functionality. Each team had an individualized space that could be customized and “meme-ified”.This space was intended to be the rallying point for each team to present who they are, what they are working on, and what their solution is. It was also a gathering space for the team as well as others to meet and discuss objectives, solution ideas, collaborations, etc…</p><figure><img alt="#beachtime" src="https://cdn-images-1.medium.com/max/1024/0*axzDQNOWrgtfqGje" /><figcaption>Don at the beach pretending to study a Waybill while the rest of us have the webcam off to hide our sleep depraved faces.</figcaption></figure><p>Integrated into the Mass Collaboration Arena was an interface providing the ability to enter specifics about the team and solution details. This then linked to a Miro board for a full representation of each team’s goals and accomplishments.</p><h3>Our Path to Enlightenment — the Odyssey Way</h3><p>In last week’s <a href="https://medium.com/keyko/working-with-odyssey-to-build-the-architecture-for-momentum-2020-aa0317506eb9">blog post</a> we described the challenge and provided a first cut of the solution we wanted to propose and implement for the <em>Compliant Air Cargo Flow </em>challenge as part of the <em>Internet of Logistics</em> track.</p><p>Turns out, however, that one of the best parts of Odyssey is not the intense coding and lack of sleep, but the opportunity to “sit down” with industry leaders and Challenge Owners to really try to understand what their pain points and priorities are when it comes to innovation.</p><p>Having a better understanding of the problems the Challenge Owners were trying to solve we realized that, even though our solution contained some of the building blocks we still needed to dive deeper into the matter.</p><p>The first thing that comes to mind when providing a solution for supply chains is provenance. By provenance we mean keeping track of all the interactions between all the authenticated parties involved and keeping a tamper proof and immutable record of those interactions. Doing this on-chain is not a new problem and solutions have existed since the inception of Bitcoin. In fact, we were doing it for digital art all the way back in 2014 at <a href="https://www.ascribe.io/">ascribe.io</a>.</p><p>Still, this is one of the fundamental building blocks of any supply chain solution and since it’s part of our work philosophy not to reinvent the wheel, we decided to implement the <a href="https://www.w3.org/TR/2013/NOTE-prov-overview-20130430/">W3C Provenance</a> specification on the Ethereum blockchain. Bam! Solved the supply chain problem!</p><h3>Interoperability FTW!</h3><p>Well…not quite. With that out of the way we set out to solve what we believe was the biggest problem that the Challenge Owners were facing: Interoperability.</p><p>A global supply chain includes a large number of entities and participants, from the manufacturer to freighter forwarders and subcontractors, to customs agencies. All counter parties need to work together to make it possible to deliver cargo to and from anywhere in the world in a timely manner. All of these entities along the supply chain produce and require access to certain data for different reasons: for compliance reasons, or to know what type of cargo is being shipped and how it should be handled.</p><p>Keeping track of who handled the cargo from point of origin to destination is something that can be solved by a provenance registry, but how do you ensure that the cargo is correctly handled and is not tampered with? And in case something goes wrong, how do we know who was responsible and how we detect the problems in a timely manner so that they can be resolved as soon as possible?</p><figure><img alt="memes need no alt text" src="https://cdn-images-1.medium.com/max/600/1*3JOn5Osz8Cv0cXbQfMDuag.png" /></figure><h3>Nevermined to the Rescue</h3><p>Enter <a href="https://nevermined.io/">Nevermined</a>, fresh <a href="https://medium.com/nevermined-io/introducing-nevermined-92ce2bf31d27">out of the oven</a>. With Nevermined we aim to provide a secure and privacy preserving, enterprise grade data sharing solution.</p><p>One of the core beliefs of Nevermined is that the data should not move. Nevermined itself does not store any data. Instead it integrates with existing data repositories and makes the data discoverable between the participants of a Nevermined data ecosystem. When a participant wants to access some data, it first needs to request access from the data provider, giving the data provider a fine grained control on who can access certain data.</p><figure><img alt="uhhh… nevermined" src="https://cdn-images-1.medium.com/max/750/0*1BJMRy3pIs_Lx5g1" /></figure><p>With both provenance and data sharing building blocks we started thinking about how our application should look like. For ease of use we decided to build a mobile application in which each entity in the supply chain would scan a QR code on the cargo and discover the data associated with that cargo. A typical supply chain flow would work like the following:</p><ol><li>A manufacturer would create some metadata about the cargo and register on-chain with Nevermined.</li><li>This metadata would contain information on how to access the <em>House Waybill</em> with information about the cargo.</li><li>With this metadata registered on-chain, the manufacturer can generate a QR code with a unique identifier that points to the metadata.</li><li>A ground freight forwarder, at pickup would scan the QR code and this action would authenticate both parties during the handover as well as present the freight forwarder with additional information about the cargo.</li><li>After analyzing the data, the ground freighter can accept responsibility for the shipment.</li><li>This would trigger a new event on-chain on the provenance registry that is digitally signed by all parties involved.</li><li>Moreover, the freight forwarder can append more data associated with the cargo through Nevermined, like for instance, a <em>Freight Waybill</em>.</li><li>This process repeats itself at each step of the supply chain until the cargo reaches the intended recipient or something wrong is flagged at one of the steps.</li></ol><figure><img alt="Is it okay to use the word “fucking”?" src="https://cdn-images-1.medium.com/max/640/0*UXGg44VxXPu7kC_e" /><figcaption>No rest for the weary</figcaption></figure><p>Fast forward 49 hours of intense coding, sore eyes, coffee, memes and a fair amount of other ✌️non-✌️alcoholic drinks, we were ready to demo our solution.</p><blockquote>The <a href="https://github.com/nevermined-io/dashboard-demo">final solution</a> was built entirely in TypeScript and integrated into Nevermined using the Nevermined <a href="https://github.com/nevermined-io/sdk-js">JavaScript sdk</a>.</blockquote><figure><img alt="we also post professional images with serious captions" src="https://cdn-images-1.medium.com/max/1024/0*91kkMkDMRuv4fus6" /><figcaption>Figma mockup of one of the many flows demoed. You can find the full mockups created during the hackaton at <a href="https://www.figma.com/file/zUqPfgyYUkMyqe0rZ1XTq1/Material-Design-Theme-Kit-(Copy)?node-id=34483%3A29337">figma</a>.</figcaption></figure><p>This flow demonstrates how a freighter would interact with the app in order to accept shipment responsibility for some cargo: a) he would start choosing the respective cargo from a list; b) it would get access to some information about the cargo; c) it would scan the QR code on the cargo to make sure it’s picking up the right shipment and optionally append some more information; d) if accepted the status of the shipment is updated and registered on-chain.</p><h3>Bonus Round — Decentralized Data Analysis</h3><p>It’s typical for parties involved in the supply chain to add data catalogers to the cargo while it is in their care. This is extremely important when the cargo contains medical supplies, like for instance, Covid-19 vaccines that need to be maintained in a certain temperature range.</p><p>The process of how the temperatures are checked at the time of handover between two freighters is mostly manual. The freight forwarder taking over the cargo will check the temperature at the time of arrival and this gives only a very narrow view of the way the cargo was handled.</p><p>In order to make this process more reliable and efficient, we decided to extend our solution and use another feature of Nevermined that we call Data In-Situ Computation (DISC). DISC was created for data providers that don’t want to allow access to the data per se, but want to allow third parties to run computations over the data. Nevermined enables this by providing all the tools necessary to bring the computations to the data.</p><p>Our use case assumed that a Covid-19 vaccine manufacturer would include in their cargo containers temperature sensors, thus becoming a data provider. A freight forwarder, before accepting the cargo, could request (or pay for) permission from the manufacturer to run some computation over that data, that would check if up to this point the cargo had been maintained at a correct temperature. If the result comes back negative, the freight forwarder would flag the shipment for further analysis to determine if there is an issue with the shipment. Ultimately, the might result in the freight forwarder deciding to not take responsibility for the shipment.</p><figure><img alt="can I use my AI on your data?" src="https://cdn-images-1.medium.com/max/1024/0*hcIorH4OEc87U9Z9" /><figcaption>Figma mockup of one of the many flows demoed. You can find the full mockups created during the hackaton at <a href="https://www.figma.com/file/zUqPfgyYUkMyqe0rZ1XTq1/Material-Design-Theme-Kit-(Copy)?node-id=34483%3A29337">figma</a>.</figcaption></figure><p>This flow is similar to the previous one except that after scanning the QR code we learn that this is a temperature sensitive shipment, and have the option to run a temperature analysis over the data provided by one or multiple data providers.</p><p>This can be done in a privacy preserving manner such that the freighter requesting the analysis does not learn anything other than if the temperature went above or below a certain threshold.</p><h3><strong>Key Takeaways</strong></h3><p>Odyssey is an important event for two main reasons: it enabled us to talk directly with industry leaders and understand in detail the problems that they are trying to solve; and it enabled us to network with like minded people and organizations.</p><p>It also gave us a chance to battle test Nevermined. This was great given that it’s such a new product, the learnings we took home will allow us to further our product market fit.</p><p>Moving forward we will continue to improve Nevermined. And having learned about all the unsolved challenges of the supply chain industry, we will continue to improve on the work we did during the Odyssey hackathon and make the supply chain example a showcase for Nevermined. So, keep an eye on our <a href="https://github.com/nevermined-io/dashboard-demo">repo</a> on Github, where all the action is!</p><p>We would like to give a big shout-out to <a href="https://medium.com/u/5e21944ff904">G Boccadifuoco</a> for keeping track of everything Odyssey related, organizing, and herding the team to make sure everybody was where they were supposed to be at all times (seriously how do you do it?)</p><p>Also to our demo presenters <a href="https://medium.com/u/9104a5aaeb71">Don Gossen</a> with his shady internet connection and <a href="https://medium.com/u/248539c9783">Jernej Pregelj</a> for taking one for the team and get on camera to present our solution (he wasn’t given a chance to refuse).</p><p><a href="https://www.linkedin.com/in/odionescu/">Oana</a> for forcing me to write this.</p><p>And all of the <a href="https://medium.com/u/13f610de4e62">The Keyko Dōjō</a> sensei.</p><p><em>Keyko is a Web 3.0 solutions provider offering integrated decentralized solutions and advisory for enterprise and startups. We thrive on providing organizations and individuals insights into this emerging digital world where capitalism and ownership are being profoundly rethought.</em></p><p><em>If you would like to know more about our services or are interested in having a chat please reach out to us on info@keyko.io or visit our website </em><a href="https://www.keyko.io/"><em>here.</em></a></p><p><em>If you enjoyed this article let us know by clapping or leaving your comment below.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=864e5cf6fa0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keyko/sensei-guiding-the-way-864e5cf6fa0c">Sensei guiding the Way</a> was originally published in <a href="https://medium.com/keyko">Keyko</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Queryable Assets in BigchainDB V 1.0]]></title>
            <link>https://medium.com/the-bigchaindb-blog/introducing-queryable-assets-in-bigchaindb-v-1-0-adbe1b86e622?source=rss-4dbbb277a0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/adbe1b86e622</guid>
            <category><![CDATA[digital-asset]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[json]]></category>
            <category><![CDATA[search]]></category>
            <dc:creator><![CDATA[Rodolphe Marques]]></dc:creator>
            <pubDate>Mon, 10 Jul 2017 11:59:27 GMT</pubDate>
            <atom:updated>2017-07-10T12:00:02.422Z</atom:updated>
            <content:encoded><![CDATA[<p>With <a href="https://github.com/bigchaindb/bigchaindb/releases/tag/v1.0.0">BigchainDB 1.0</a> we introduced a new feature which is queryable assets. This new feature allows users to perform a case-insensitive text search on the data stored in the assets. All the assets that match a text search and were created by valid transactions are returned in a list ordered by text score. Each returned element has the following structure:</p><pre>{<br>    &quot;data&quot;: {&quot;&lt;The asset data&gt;&quot;},<br>    &quot;id&quot;: &quot;The ID of the asset&quot;<br>}</pre><p>The datakey contains the asset JSON document that was added by the user in a <em>CREATE</em> transaction. The id key is the asset ID which is the same as the ID of the transaction that created the asset. Users can then use this ID to perform other transaction related queries like retrieving the transaction that created the asset or retrieving all the transactions related to an asset (the asset history).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqlsOlQpiyA4HrIgAEx7mQ.jpeg" /></figure><h4>Examples: 9984</h4><p>This new functionality is exposed by the <a href="https://docs.bigchaindb.com/projects/server/en/latest/http-client-server-api.html#assets">/api/v1/assets</a> HTTP endpoint. Let’s look at some examples of what we can do with these new feature. For this example I populated a BigchainDB 1.0 instance running locally with the following assets:</p><pre>assets = [<br>    {&#39;name&#39;: &#39;Luke Skywalker&#39;, &#39;race&#39;: &#39;Human&#39;},<br>    {&#39;name&#39;: &#39;Darth Vader&#39;, &#39;race&#39;: &#39;Human&#39;},<br>    {&#39;name&#39;: &#39;Leia Organa&#39;, &#39;race&#39;: &#39;Human&#39;},<br>    {&#39;name&#39;: &#39;Leia Organa Solo&#39;, &#39;race&#39;: &#39;Human&#39;},<br>    {&#39;name&#39;: &#39;Han Solo&#39;, &#39;race&#39;: &#39;Human&#39;},<br>    {&#39;name&#39;: &#39;C-3PO&#39;, &#39;race&#39;: &#39;Droid&#39;},<br>    {&#39;name&#39;: &#39;R2-D2&#39;, &#39;race&#39;: &#39;Droid&#39;},<br>    {&#39;name&#39;: &#39;Chewbacca&#39;, &#39;race&#39;: &#39;Wookiee&#39;}<br>]</pre><p>I will be using <em>curl</em> in these examples but this functionality can be used with the latest versions of both the Python and JavaScript BigchainDB drivers.</p><p><strong>Searching for a single word</strong></p><p>We can perform a simple word search by searching for all assets with a string containing the word <strong>human</strong>:</p><pre>curl -Gv <a href="http://localhost:9984/api/v1/assets">http://localhost:9984/api/v1/assets</a> --data-urlencode search=&quot;human&quot;</pre><pre>&gt; GET /api/v1/assets?search=human HTTP/1.1<br>&gt; Host: localhost:9984</pre><pre>[<br>  {<br>    &quot;id&quot;: &quot;c335be071847c5e318d81c2e8b1604d7102cf114f38665ee771419675ef45848&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Luke Skywalker&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;217eef71fb24c777c5c0204c9676bd53b0363730360a0d4a74013cc02bd62402&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Leia Organa&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;59f836dcdbb422c21d2f46f62a56581d262c3996440d2bf6af0dca3437b6e25b&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Darth Vader&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;3e7105b510a532817f90d5480200287dd2ea71fbcfa4433240250f2b7769ec01&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Han Solo&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;1c02983f47e5b824588483b4c69fd3285262a7ac3ee73405f620d0897d846756&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Leia Organa Solo&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  }<br>]</pre><p><strong>Match multiple search terms</strong></p><p>We can perform a search to match any of the search terms by using a string with space-separated search terms. This performs a logical <em>O</em>R<em> </em>on all the search terms in the string returns assets that match any of the search terms. For instance we may want to search for all assets that contain either the word <strong>leia </strong>or <strong>organa</strong> or <strong>solo</strong>.</p><pre>curl -Gv <a href="http://localhost:9984/api/v1/assets">http://localhost:9984/api/v1/assets</a> --data-urlencode search=&quot;leia organa solo&quot;</pre><pre>&gt; GET /api/v1/assets?search=leia%20organa%20solo HTTP/1.1<br>&gt; Host: localhost:9984</pre><pre>[<br>  {<br>    &quot;id&quot;: &quot;1c02983f47e5b824588483b4c69fd3285262a7ac3ee73405f620d0897d846756&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Leia Organa Solo&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;217eef71fb24c777c5c0204c9676bd53b0363730360a0d4a74013cc02bd62402&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Leia Organa&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;3e7105b510a532817f90d5480200287dd2ea71fbcfa4433240250f2b7769ec01&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Han Solo&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  }<br>]</pre><p><strong>Exact phrase search</strong></p><p>We can also perform searches that match an exact phrase by escaping the quotes in the search term. For example we can search for the exact phrase \<strong>”leia organa solo\”.</strong></p><pre>curl -X GET -Gv <a href="http://localhost:9984/api/v1/assets">http://localhost:9984/api/v1/assets</a> --data-urlencode search=&quot;\&quot;leia organa solo\&quot;&quot;</pre><pre>&gt; GET /api/v1/assets?search=%22leia%20organa%20solo%22 HTTP/1.1<br>&gt; Host: localhost:9984</pre><pre>[<br>  {<br>    &quot;id&quot;: &quot;1c02983f47e5b824588483b4c69fd3285262a7ac3ee73405f620d0897d846756&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Leia Organa Solo&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  }<br>]</pre><p><strong>Excluding terms</strong></p><p>It is also possible to exclude assets that contain a term by using a negated term. A negated term is defined by prepending a dash before the search term. For instance if we want to search for all assets that contain the word<strong> human</strong> but do not contain the word <strong>solo</strong> we can use a search string like “<strong>human -solo”.</strong></p><pre>curl -X GET -Gv <a href="http://localhost:9984/api/v1/assets">http://localhost:9984/api/v1/assets</a> --data-urlencode search=&quot;solo -han&quot;</pre><pre>&gt; GET /api/v1/assets?search=human%20-solo HTTP/1.1<br>&gt; Host: localhost:9984</pre><pre>[<br>  {<br>    &quot;id&quot;: &quot;c335be071847c5e318d81c2e8b1604d7102cf114f38665ee771419675ef45848&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Luke Skywalker&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;217eef71fb24c777c5c0204c9676bd53b0363730360a0d4a74013cc02bd62402&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Leia Organa&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;59f836dcdbb422c21d2f46f62a56581d262c3996440d2bf6af0dca3437b6e25b&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Darth Vader&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  }<br>]</pre><p><strong>Limit the amount of returned assets</strong></p><p>Since there is a potential of matching a big number of assets with a text search we can also limit the amount of results returned by using the <em>limit </em>argument.</p><pre>curl -X GET -Gv <a href="http://localhost:9984/api/v1/assets">http://localhost:9984/api/v1/assets</a> --data-urlencode search=&quot;human&quot; --data-urlencode limit=2</pre><pre>&gt; GET /api/v1/assets?search=human&amp;limit=2 HTTP/1.1<br>&gt; Host: localhost:9984</pre><pre>[<br>  {<br>    &quot;id&quot;: &quot;c335be071847c5e318d81c2e8b1604d7102cf114f38665ee771419675ef45848&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Luke Skywalker&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;217eef71fb24c777c5c0204c9676bd53b0363730360a0d4a74013cc02bd62402&quot;,<br>    &quot;data&quot;: {<br>      &quot;name&quot;: &quot;Leia Organa&quot;,<br>      &quot;race&quot;: &quot;Human&quot;<br>    }<br>  }<br>]</pre><h4>Implementation details</h4><p>This feature is only available when running with <a href="https://www.mongodb.com/">MongoDB</a> as the backend.</p><p>One of the first things that we had to do was to decouple the assets from the transactions and store them separately. The fact that the asset was nested inside a transaction that is nested inside a block made it difficult to query assets efficiently. The proposal and discussion about benefits about of decoupling the assets from the transaction can be found on GitHub issue <a href="https://github.com/bigchaindb/bigchaindb/issues/1275">#1275</a>.</p><p>The decoupling of the assets is handled purely at the database layer and it’s transparent to both the users and the the BigchainDB layer. When a block is being written to the database the assets are decoupled from the transactions in the block and stored in a separate collection. When a block is read from the database the assets are retrieved and the original block is re-assembled before passing it to the BigchainDB layer.</p><p>Another problem that we had to solve is that we don’t know the structure or schema of the assets that the users are going to write into BigchainDB. We need to look at the assets as unstructured data. The assets not only do not have a predefined schema but different assets will have different schemas. This makes it quite difficult for us the target specific fields in the document for indexing.</p><p>We were able to solve this problem by leveraging the the <a href="https://docs.mongodb.com/manual/text-search/">text search</a> capabilities of MongoDB. Besides allowing us to create indexes on particular fields of a document, MongoDB also allows us to create a wildcard text index that will index all the fields that contain string content without knowing what those fields are in advance. You can find more details about this implementation by checking the GitHub pull request <a href="https://github.com/bigchaindb/bigchaindb/pull/1460">#1460</a>.</p><p>Going forward we will also be adding text search capabilities to the data stored under the metadata<strong><em> </em></strong>field of the transactions. You can check the progress of this feature by following GitHub issue <a href="https://github.com/bigchaindb/bigchaindb/issues/1592">#1592</a>.</p><p>Be sure to follow this blog to stay up to date on future releases and to get a first-hand look at additions to the system, what has changed and why. Finally, if you’re a developer using BigchainDB, we want to hear from you. Send us an email at <a href="/service/mailto:contact@bigchaindb.com">contact@bigchaindb.com</a> and tell us your story.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=adbe1b86e622" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-bigchaindb-blog/introducing-queryable-assets-in-bigchaindb-v-1-0-adbe1b86e622">Introducing Queryable Assets in BigchainDB V 1.0</a> was originally published in <a href="https://medium.com/the-bigchaindb-blog">The BigchainDB Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rebooting the Web of Trust Workshop: A Recap]]></title>
            <link>https://medium.com/the-bigchaindb-blog/rebooting-the-web-of-trust-workshop-a-recap-c3ecbd755790?source=rss-4dbbb277a0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3ecbd755790</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[workshop]]></category>
            <category><![CDATA[web-of-trust]]></category>
            <category><![CDATA[decentralization]]></category>
            <dc:creator><![CDATA[Rodolphe Marques]]></dc:creator>
            <pubDate>Fri, 05 May 2017 15:50:07 GMT</pubDate>
            <atom:updated>2017-05-05T15:54:22.464Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fnl51AEw53m9e0jWPwh6GA.jpeg" /></figure><p>Rebooting the Web of Trust is an event where people from all over the world get together for a three day workshop to exchange knowledge and help create the next generation of decentralized web of trust based identity systems.</p><p>The fourth edition, which took place in Paris in late April, welcomed around 30 people with various backgrounds from academics to technologists, cryptographers, developers and even anthropologists.</p><p>Hosted by the principal architect at Blockstream and internet cryptography pioneer, <a href="https://twitter.com/ChristopherA">Christopher Allen</a>, the workshop is focused on designing and building decentralized identity systems. The goal is to generate technical whitepapers on topics decided by the group, followed by a hackathon to actually implement the ideas. There are now seven papers <a href="https://github.com/WebOfTrustInfo/rebooting-the-web-of-trust-spring2017/blob/master/draft-documents/README.md">in progress</a> as a result, with topics ranging from decentralized reputation systems to decentralized key management systems and functional identity.</p><p>In addition to developing new technology, another aim of the event is to align the produced specs with the work of standardization bodies like <a href="https://www.ietf.org/">IETF</a> or <a href="https://www.w3.org/">W3C</a>. The goal of one such topic selected by the group, the <a href="https://github.com/WebOfTrustInfo/rebooting-the-web-of-trust-spring2017/blob/master/event-documents/group-abstracts/SignatureAlignmentAbstract.md">LD Signature Format Alignment proposal</a>, for example, was to reuse the IETF JSON Web Signing specification to produce Linked Data Signatures. During the workshop three implementations were produced in <a href="https://github.com/WebOfTrustInfo/ld-signatures-js">javascript</a>, <a href="https://github.com/WebOfTrustInfo/ld-signatures-java">java</a> and <a href="https://github.com/WebOfTrustInfo/jsonld-signatures-python">python</a>.</p><p>Blockchain technology is an enabler for truly decentralized identity solutions. What drives BigchainDB to attend Rebooting the Web of Trust events is not only to better understand the space and what’s needed but also how to build and shape the technology in a way that ensures interoperability with other decentralized identity solutions while also appealing to the decentralized community overall.</p><p>Producing papers and implementing experimental initiatives are the two main objectives but how do we take this to the next level? Through clean and concise messaging, more papers, better engagement and easy on-boarding for newcomers, we hope to see the group grow to five million people worldwide, at least.</p><p>The <a href="https://github.com/WebOfTrustInfo/rebooting-the-web-of-trust-fall2017">next event</a> will be taking place in Boston, Massachusetts in October 2017. For more information on the previous design workshops or for ways to get involved, you can see all shared repositories for the Rebooting the Web of Trust <a href="https://github.com/WebOfTrustInfo">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3ecbd755790" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-bigchaindb-blog/rebooting-the-web-of-trust-workshop-a-recap-c3ecbd755790">Rebooting the Web of Trust Workshop: A Recap</a> was originally published in <a href="https://medium.com/the-bigchaindb-blog">The BigchainDB Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Integrated With MongoDB]]></title>
            <link>https://medium.com/the-bigchaindb-blog/how-we-integrated-with-mongodb-d6a45e776d6b?source=rss-4dbbb277a0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6a45e776d6b</guid>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Rodolphe Marques]]></dc:creator>
            <pubDate>Fri, 07 Apr 2017 13:02:29 GMT</pubDate>
            <atom:updated>2017-04-07T13:42:43.761Z</atom:updated>
            <content:encoded><![CDATA[<p><em>In </em><a href="https://blog.bigchaindb.com/bigchaindb-version-0-9-released-d7569e4509f3"><em>version 0.9</em></a><em>, we released support for running BigchainDB with MongoDB. It marked an important milestone in our strategy of supporting a variety of different database backends. This blog post looks at how we achieved it.</em></p><p>When we started working on BigchainDB, we considered and tested several popular distributed databases. Ultimately, we chose RethinkDB mostly for its simplicity and real time capabilities, namely the changefeed. We spent a lot of time developing our product on top of RethinkDB. Correctly understanding a complex system like a distributed database takes time. Understanding its guarantees and how to correctly use them, as well as finding solutions for the functionalities that the database does not provide took us several months.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*lrzGRzINQISGHm_R2xjHug.jpeg" /></figure><p>By mid 2016 we were confident that we had accumulated enough knowledge about the intricacies of distributed databases that we decided to make good on our promise of supporting multiple databases. In August 2016, we did our first internal proof of concept of BigchainDB running on top of MongoDB. This allowed us to estimate the amount of work required to support other databases as well as the changes that we would need to make to the code. We opted to build on MongoDB next because it had many useful features that we could directly leverage (see below), its consistency has improved compared to older versions, and it has emerged as a winner in the NoSQL distributed database category. Furthermore, we had developed great relations with MongoDB on both the technical and business fronts, which has given a boost to our efforts overall.</p><p>In the end, we were able to abstract the BigchainDB core code from the underlying database allowing us to plug in multiple database backends without the need to change the BigchainDB core code.</p><p>BigchainDB v0.9 marks the first BigchainDB release with official support for MongoDB. When <a href="https://docs.bigchaindb.com/projects/server/en/latest/quickstart.html">configuring</a> BigchainDB with v0.9, you will need to specify what database backend you want to use: `rethinkdb` or `mongodb.` MongoDB integration is still in progress but at this stage we have the same functionality that we provide with RethinkDB, except for sharding. With MongoDB, we decided to take a different approach and roll out different features in stages. As of this moment, you can use BigchainDB with a MongoDB replica set (fully replicated cluster).</p><p>Going forward we will be spending more time <a href="https://blog.bigchaindb.com/bigchaindb-2017-roadmap-d2e7123f9874">improving</a> the performance, scalability, security and stability of a BigchainDB cluster. We will be adding audit and monitoring capabilities to earlier identify problems in the cluster as well as continuous backups and recovery in case of critical failures. Improved scalability and performance through sharding. Improved security by adding node authentication for intra cluster communications as well as authentication for driver connections to the database. All of this by leveraging features provided by MongoDB.</p><h3>Abstracting the DB</h3><p>Abstracting the database layer was a quite interesting task from an engineering point of view. It helps making the overall architecture of our software better and, since both implementations must expose the same functionalities, it allows us to do a 1:1 comparison between the two databases.</p><p>Until BigchainDB v0.8, all the queries to RethinkDB were hardcoded in the core module of the software. The first step we took was to move those queries in a separate module, and have the core module calling just functions to insert, update, or retrieve data. The main assumption for this new (backend) module was: all the inputs are simple dictionaries, all the returned values are basic types (integers, strings, dictionaries) or iterables.</p><p>The second step in the integration with MongoDB was to improve abstractions. We extended the backend module to define abstractions to: connect to the database, define the schemas (creating collections and indexes), and process a changefeed. We decided to put the interfaces and abstract classes in the <a href="https://github.com/bigchaindb/bigchaindb/tree/0.9/bigchaindb/backend">bigchaindb.backend</a> module.</p><p>The architecture of the backend module is based heavily upon Python’s newly-introduced <a href="https://www.python.org/dev/peps/pep-0443/">single-dispatch generic functions</a>. Single-dispatch is convenient, because it allows Python, rather than something we design ourselves, to manage the dispatching of generic functions based on certain conditions being met (e.g. the database backend to use). If you want to know more about that module, in the same directory you can find a <a href="https://github.com/bigchaindb/bigchaindb/blob/v0.9.0/bigchaindb/backend/README.md">README</a> that digs into the details of the abstraction. We used a GitHub milestone to group all the issues related to this task. As you can see, <a href="https://github.com/bigchaindb/bigchaindb/milestone/6?closed=1">there was some work involved</a>.</p><p>Once the structure was defined, and all tests for RethinkDB were passing again (remember, we didn’t add any functionality, we just added a new layer to access the backend), we started working on implementing the queries for MongoDB. We grouped everything in a <a href="https://github.com/bigchaindb/bigchaindb/milestone/7?closed=1">milestone</a> to track the progress of this big task.</p><p>There are two parts where RethinkDB and MongoDB implementations diverge: the global notification system and how replication and sharding are handled. BigchainDB relies heavily on a global notification system to be able to react every time there is a change in the database. RethinkDB provides this functionality out-of-the-box (regardless of what replication or sharding settings are being used), the <a href="https://www.rethinkdb.com/docs/changefeeds/python/">changefeed</a>. With MongoDB we needed to have the same functionality and we achieved that by <a href="https://docs.mongodb.com/manual/core/tailable-cursors/">tailing</a> the <a href="https://docs.mongodb.com/v3.4/core/replica-set-oplog/">operations log</a>. The operations log is a special type of collection that keeps a record of all operations that modify the data stored in the database. You can see the different changefeed implementations for <a href="https://github.com/bigchaindb/bigchaindb/blob/0.9/bigchaindb/backend/rethinkdb/changefeed.py">RethinkDB</a> and <a href="https://github.com/bigchaindb/bigchaindb/blob/0.9/bigchaindb/backend/mongodb/changefeed.py">MongoDB</a> in our code base. Another difference is the way both databases handle replication and sharding. For this reason we decided to release support for MongoDB in fully replicated mode and handle sharding in a later stage.</p><p>The last change was related to the configuration and initialization of the system, to allow the user to choose the backend to use. Just type `bigchaindb configure mongodb` and `bigchaindb init` to start using BigchainDB with MongoDB!</p><p>Be sure to follow this blog to stay up to date on future releases and to get a first hand look at additions to the system, what has changed, and why. Finally, if you’re a developer using BigchainDB, we want to hear from you. Send us an email at <a href="/service/mailto:contact@bigchaindb.com">contact@bigchaindb.com</a> and tell us your story.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6a45e776d6b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-bigchaindb-blog/how-we-integrated-with-mongodb-d6a45e776d6b">How We Integrated With MongoDB</a> was originally published in <a href="https://medium.com/the-bigchaindb-blog">The BigchainDB Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>