Holonym is a zero knowledge protocol for Sybil resistance, anonymous KYC, and wallet recovery. Verify users without sacrificing privacy. The project started as a hack-a-thon to link academic web accounts to public on-chain identities. We’ve pivoted since then to focus on privacy as first principle for identity on the internet. Learn more at https://holonym.id ✨ Continue reading below to learn how Holonym got its start with zero knowledge.

Vision of a Future Web

We launched the Holonym alpha in April 2022 to unlock the next wave of decentralized web protocols that are open to use by anyone, yet also resistant against bots, fraud, and abuse. Our team built toward a vision of the internet where any decentralized application (dApp) can verify users. In this vision, every claim, , is treated as a fact about an account attested by the owner of that identity. Together, many nyms may combine to describe a holistic user-configured identity, a Holo, that abstracts account ownership away from centralized services and back to individual users.

Decentralized Societies (DeSocs) can bootstrap credibility from the reputation of their verified members that have linked off-chain credentials.

Alpha Release: Public Account Linking

The Holonym DID Registry alpha was launched at ethAmsterdam as a first step towards this vision. The goal of the alpha version was to demonstrate the linking of a signed credential to an Ethereum account with a cryptographic proof that the owner of an ORCID, Discord, or really any web account, corresponds to a specific decentralized identifier (DID). We encountered many engineering challenges around requirements for privacy, security, and seamless UX. Despite these challenges, we built a novel architecture that solves a difficult problem. The Holonym alpha system verifies any web credentials on smart-contract compatible blockchains. Now, smart contracts can interact with user identities from websites like Google and Twitter. Verified sellers can exist in decentralized Web3 marketplaces and check verifications on-chain. Decentralized Societies (DeSocs) can bootstrap credibility from the reputation of their verified members that have linked off-chain credentials.

For example, OpSci is using Holonym’s alpha architecture to power Verse, a decentralized science society registry of verified public academic accounts. This is just the tip of the iceberg. Many other use-cases that require public claims on identity are waiting to be unlocked. However, being public on the internet should always be an option and not a requirement for using a service. Privacy should be the default setting for internet users.

In this next iteration, we’ve redesigned the Holonym beta to offer users privacy and pseudonymity by default, embedding user consent into any actions that may result in public identification.

Charting a New Path: Privacy First for Users

How do you verify someone’s identity without compromising user data to any single person or system? Shortly after releasing our alpha version, our team hunkered down to redesign Holonym from scratch addressing this seemingly impossible question. This time we started with zero knowledge proofs for privacy (ZKPs) built-in as first principles. In this next iteration, we’ve redesigned the Holonym beta to offer users privacy and pseudonymity by default, embedding user consent into any actions that may result in public identification. The impetus for this was a request by Lobby3 to prove US residency of their DAO members engaged in US political and civic advocacy. ZKPs make this possible while also offering several advantages beyond privacy, and will likely become a key part of next generation internet infrastructure. Let’s dive in and explore some of these future-forward perks and the interesting use-cases that become possible!

Privacy as a Perk

One of the more obvious advantages of ZKPs is privacy. Zero knowledge means users do not need to reveal their credentials to prove facts about them. This breakthrough feature makes it possible to maintain an open decentralized database of private claims on identities. The private claims are converted into a proof that the user has a valid identity signed by an identity issuer. We call this database of private claims a privacy pool, where any application can verify user proofs by checking if they are a member of the pool. The privacy pool is a global Merkle tree that only accepts new proofs on identity if they can be proven valid by an issuer.

Users have full control over when to reveal facts about themselves and how much information to include in those claims.

We have designed Holonym so that the issuer knows nothing about the user except that they received a cryptographically signed valid credential such as a driver’s license, or passport from an authority (i.e., the DMV or State Department). The Holonym architecture is a novel application of ZKPs that minimizes leakage of user information outside of the parties that are already privy to that knowledge. For example, users that prove their identity with a legal ID need only to trust the authority that issues the credential and that their own computer is secure. All other services, agents, and requestors are completely blind to the government ID and can only make requests via simple proofs about the identity, such as age, residency, sex, etc. In summary, Holonym users are offered privacy by default. Users have full control over when to reveal facts about themselves and how much information to include in those claims. The protocol is decentralized and anyone can run an issuer to maintain the network while retaining zero knowledge about users or their identities.

Security as a Perk

A less obvious benefit of zero knowledge for Holonym users is the security that credentials are accessible and recoverable only by the owner. Holonym employs a nullifier scheme that makes duplication or impersonation of an identity incredibly impractical. In this scheme, all credentials are appended with a salt and pepper, not for flavor, but to protect against dictionary attacks that attempt to map all possible encryptions to common finite sets of strings such as names, states, or dates. The pepper is a random string that’s appended to the credential and only known to the user, whereas the salt is a random string that is newly generated each time the user creates a new account. In practice, this means that a user can prove that they are a real person using a verified credential hidden within the privacy pool via a pepper but may act through different accounts using a different salt for each new anonymous account.

For the first time, online communities can establish basic safeguards against impersonation, fraud, and bot attacks without requiring their users to give up privacy, allowing them to remain anonymous.

The security offered by the privacy pool scales logarithmically with each new member, or leaf, added to the global Merkel tree. For example, there is a 50% chance of correctly guessing the identity with two members in the set, 33% for three, and so on. These security mechanisms combined with private proofs on identity allow for users to store and retrieve secrets, such as a password or seed phrase, making account recovery as simple as presenting proof of a set of valid credentials that uniquely identify the user. This means that a user can recover a lost asymmetric key set, such a crypto wallet, without relying on a centralized service or revealing who they are to any service, agent, or entity.

Interoperability and Scalability as a Perk

Holonym employs a modular architecture for computing ZKPs that is both highly interoperable with any web service and easily scalable. Once a user presents a valid credential, the issuer encrypts and appends it to a fixed-length array. The first two fields of this array contain 1) the address of the credential issuer and 2) the nullifier for the user. These fields are mandatory. The final four fields can be completely custom and are defined by the format favored by the issuer. Holonym was built to allow any identity authority to run their own issuer and produce custom proofs that fit their specific use-case requirements.

For example, the DMV might deploy their own issuer to verify detailed information about your driving history, address, eyesight, and age. While a DAO may sign attestations about a user’s on-chain activity, membership in specific societies, or attendance at local events. Another key property of ZKPs is their ability to compress computations into manageable, succinct expressions that serve as proof that a specific series of steps were taken to achieve a specific result. Custom fields may include entries that correspond to a ZKP of a complex execution environment leading to the computation of a specific result. This way we can prove a specific result was obtained by following a known algorithm without storing the entire algorithm along with the result. This degree of customizability allows for the protocol to accommodate a wide variety of use-cases and promotes scalable decentralization of identity issuers.

Holonym was built to allow any identity authority to run their own issuer and produce custom proofs that fit their specific use-case requirements.

User Experience as a Perk

Decentralized applications are still relatively nascent and rely on complex technology that are difficult to interact with and place high cognitive burden on new users. The user experience of these protocols resemble the webpages of the early 90s, clunky, poor design, and full of impenetrable jargon. Holonym employs an architecture that abstracts much of the complexity of interacting with decentralized web technology into the background. Although credentials are stored on blockchains, users do not have to purchase crypto tokens in order to pay the fees for interacting with blockchain protocols. Until recently, zero knowledge circuits have been notoriously difficult to compile, with long loading times that make client-side computation impractical. Holonym has taken multiple steps to minimize these proof times, such as utilizing the Poseidon hash function and a light weight standard for data structures used to store credentials. A new user can mint a Holo that contains a zero knowledge proof of identity and unique personhood in less than a minute using familiar UX patterns such as phone number verification and ID scanning with their phone. The proof takes only 3 seconds to generate on an average laptop. The ZK proofs allow for even more seamless decentralized UX than would be possible otherwise — the user can submit cross-chain transactions without having signing anything, because the proof is used in place of a signature.

Decentralized permissionless protocols can comply with government regulations by verifying valid credentials, proving residency, or satisfying age limits … Internet users can remain anonymous but take their reputation with them across communities …

Previously Impossible Use-Cases, Now Possible

The ZKP architecture powering Holonym opens a universe of novel use cases that are baked in with privacy as a default setting. For the first time, online communities can establish safeguards against impersonation, fraud, and bot attacks without requiring their users to give up privacy, allowing them to remain anonymous. In the same vein, decentralized permissionless protocols can comply with government regulations by verifying valid credentials, proving residency, or satisfying age limits. The modularity of the stack also allows users to maintain an audit trail that is private and only visible to relevant authorities. Internet users can remain anonymous but take their reputation with them across communities and web applications allowing for previously impossible use-cases such as anonymous credit scores that are verifiable, or identification of protections against money laundering. All of these use-cases have previously relied on a centralized authority to verify identities and claims, for the very first time we can achieve the previously impossible balance of preserving privacy on the internet while maintaining accountability.

Future Forward Looking

ZKPs are amazing and beyond the popular use cases, they have use cases that many would not imagine. The learning curve is steep, but well worth it. They allow for our protocol to have better privacy, security, interoperability, and UX. Instead of privacy, who would’ve thought their primary use case in the early 2020s would be blockchain scalability? One can only imagine what zero knowledge proofs will be used for in 100 years or even 20 years.

Jump into the Privacy Pool!

Does the fight for privacy on the internet resonate with you? Would you like to be an early adopter of cutting edge zero knowledge technology in web3? Consider signing up for our private beta, joining our discord, or perusing our white paper.

More information can be found on our documentation at https://docs.holonym.id. Also keep an eye out on https://verse.opsci.io for exciting new updates in DeSci

Originally published at https://medium.com on December 9, 2022.

From Public Web Account Linking to General Use Zero Knowledge Proofs: was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, Lobby3 will begin testing KYA (know-your-anon) zero knowledge technology to prove the residence of DAO members engaging in US political advocacy and supporting election campaigns. KYA is made possible by Holonym, a zero knowledge cloak that powers identity verification of internet users without revealing private sensitive information. Check it out at https://app.holonym.id/verify. Continue reading below to learn how Holonym is innovating in zero knowledge identity technology.

Digital Civic Engagement on the Decentralized Web

How does Web3 fit into a world of Web2 compliance and regulations? Where does the Web3 “anon” fit into open civil discourse and the need for accountability required by our traditional institutions? These questions are at the top of the mind for Lobby3 DAO, a political advocacy organization established by Andrew Yang’s Humanity Forward Foundation.

Lobby3 is a pioneer Decentralized Autonomous Organization (DAO) established to demonstrate blockchain technology for public and technology policy through civic-engagement with congressional lawmakers. The DAO has taken on Humanity Forward’s mandate to pursue policy reform and technological innovation that directly leads to greater financial inclusion through decentralization and direct action. Lobby3’s plans to accomplish this mission in the following way:

• Build Advocacy Infrastructure. Lobby3 will fund issue-based advocacy organizations in DC to build core advocacy infrastructure.
• Policy Maker Education. Lobby3’s partners will work to educate lawmakers in Congress and across the country on the positive impacts of Web3.
• Pilot Program. The advocacy team will establish partnerships with state and city leaders across the US to run Web3 pilot programs that address poverty, financial inclusion, and create greater financial opportunities for marginalized communities.
• Lobby3 DAO. The Lobby3 DAO will be established once key infrastructure, such as secure digital voting, is in place. The DAO will vote on policy ideas, community priorities, and where DAO funds should be used.

Web3 Coordination for Political Collective Action

As a DAO, Lobby3 runs differently from traditional congressional lobbying firms or political advocacy organizations. Membership is open to anyone in the world that purchases the Lobby3 Non-Fungible Token (NFT), granting the owner access to communications, voting platforms, and access to events. A key challenge for Lobby3 is achieving compliance, trust, and transparency as a politically-active digital organization within the US while remaining true to principles of decentralization, privacy, and internet freedom. Digital political engagement systems can be enhanced through Web3 tools for verifying claims and user credentials to prevent spam and facilitate discourse.

Holonym is an identity protocol that enables members to prove their affiliations and credentials, while remaining in ownership of their data, an important component of Lobby3 as a testbed for civic digital tooling.

Holonym provides a fundamental privacy layer that empowers citizens to participate in safe public spaces on the internet while preserving security and ownership of their data. In the future, Lobby3 may even explore how a privacy-preserving civic-engagement wallet may empower voting, political engagement, and digital advocacy. For example, Sybil resistance via Holonym can unlock one-person-one-vote or universal basic income for internet users.

Digital Advocacy Requires Regulatory Compliance

As a lobbying organization, special regulations and compliance requirements apply to Lobby3 that, at first-glance, may clash with Web3 decentralization and privacy principles. Anyone around the world can become a Lobby3 member and participate in activities that influence US policy or public sentiment. This is an issue for the DAO if it ever needs to prove that certain actions were the result of legal activities by US citizens, either for compliance or for public transparency. So a rigorous and privacy-preserving ID system is a Day 1 must-have.

• The Foreign Agents Registration Act requires “Foreign agents or lobbyists that act on behalf of a ‘foreign principal’ (governments, parties, entities, or individuals) to influence US policy, trade, tourism, or other public opinion are required to register with the DoJ under the Foreign Agents Registration Act (FARA).”
• The Federal Election Campaign Act (52 U.S.C. § 30121, 11 CFR 110.20) prohibits foreign nationals from 1) making any contribution, donation, expenditure, disbursement in connection with any federal, state, or local election, 2) making any contribution or donation to any committee or organization of any national, state, district, or local political party, 3) making any disbursement for an electioneering communication, 4) making any donation to a presidential inaugural committee.

To address these laws, Lobby3 is exploring how pseudonymous members can present a private proof of residence. This proof must preserve the privacy of the individual, meaning no leaking of identifying information, but must also pass rigor for residence verification.

Holonym has completed a pilot project to help Lobby3 overcome these challenges and demonstrate tangible real-world use-cases for cryptography in digital governance and coordination. Holonym is a zero-knowledge passport that allows individuals to verify their real world credentials, and produce proofs of claims about them without revealing more information than needed. This partnership aims to tackle reporting requirements while maintaining member privacy.

Holos are Zero Knowledge Visas for Web3

Holonym accomplishes verification of off-chain credentials using Vouched, a service that allows users to verify their identity or residence using government ID, phone number/email, or social security number. Information from these documents is verified against databases managed by identity authorities. Vouched will eventually be just one of many options because Holonym’s architecture is decentralized, allowing anyone to create and add a credential verifier to the protocol.

Holonym Identity Proofs in Action

The application is live now at https://app.holonym.id and just requires the installation of a browser extension to get started. Early users will be critical for contributing to the anonymity strength of the privacy pool, where each new user increases the difficulty for brute-force attacks on the anonymity set. Support privacy in Web3 by minting your Holo now! Walkthrough appears below:

Walkthrough Minting a Holo

Proving Stuff with your Holo

Looking Forward

The power of zero knowledge technology is just beginning to be understood. Holonym was built to enable privacy by default on the internet for all users. Our team is eager to collaborate with DAOs that require on-chain privacy but are seeking solutions for Sybil resistance, private non-coercive voting mechanisms, and compliance with regulators to expand operations outside of Web3.

Jump into the Privacy Pool!

Does the fight for privacy on the internet resonate with you? Would you like to be an early adopter of cutting edge zero knowledge technology in web3? Consider signing up for our private beta, joining our discord, or perusing our white paper.

More information can be found on our documentation at https://holonym.gitbook.io/holonym-alpha/

Originally published at https://medium.com on November 3, 2022.

Web 3 Privacy Tools for Civic and Social Advocacy was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.

At ETH Denver, we were challenged to solve the issue of verifying claims about DIDs for DeSci dApps. A major obstacle for any permissionless application is overcoming exploitation by unverified accounts impersonating somebody else (e.g., a posting a journal article pretending to be a scientist or NFT pretending to be its creator).

We built Holo to provide on-chain identity services, as decentralized as possible. The most common use case of Holo is the linking of Web2 accounts like Twitter and Discord to a blockchain account. We’ve gotten a lot of questions about how the protocol verifies Web2 accounts, so we wrote this piece.

Holo uses web token forwarding for linking Web2 accounts. Essentially, it links blockchain addresses to web2 accounts. It does this by forwarding ( JSON) web tokens to the blockchain.

What are JSON Web Tokens (JWTs)?

JSON Web Tokens (JWTs), also pronounced “jots,” are cryptographically signed pieces of data used for account authentication and authorization throughout the web. Ever clicked “sign-in with Google?” When you signed in, Google returned a JWT. This JWT contains info about your account in JSON format, much like this:

Figure 1. A fake simplified JWT. Note that we phased out google accounts since the time of writing as email is PII. Email verification will return to Holo with ZK proofs for anonymity.

It says

1. Who you are: holo@opsci.io
2. Who issued the JWT: accounts.google.com
3. Who the JWT is intended for: someapp

It really says quite a lot more, but this is just a smaller tutorial version that only says these three items.

Why should I care that Google gave me JSON which says all these claims? Because Google gives something else along with it: a cryptographic signature which proves this came from Google. Essentially, you’ve been handed a certificate which you can show someapp that you are holo@opsci.io. someapp can then verify the signature to know it’s from Google.

Cryptographic signatures are not only found in JWTs. They are used all over the web. Importantly for us, they’re essential to blockchains. Signatures are how you prove that you spend a currency and not anyone else — you sign the transaction to spend your money; nobody else can sign for you nor forge your signature. With a bit of code, some blockchains can verify these signatures on JWTs. Ethereum and many other EVM-compatible blockchains can do so (as long as they support the 0x05 modular exponentiation precompile, in case you were wondering).

The blockchain is entirely public. JWTs are meant to be private. Remember how JWTs are signed certificates for a certain app saying that you are who you say you are? If anyone finds your JWT, they can thus impersonate you. So it’s generally considered a bad idea to give away your JWT.

Luckily, our smart contracts invalidate every JWT once they’re used the first time. And these JWTs are not valid for anything but our smart contracts. This is thanks to the “aud” audience claim which restricts them to an app named someapp. JWTs are restricted to a specific app — in our case, specific smart contract. So nobody can impersonate you and present your identity to other apps.

Bribes for Identities. Remember how the blockchain is public and anybody can see your transactions before they’re added to the next block? It is a huge issue if people can see your JWT before the block is finalized. Adversarial parties can see the JWT while the block is assembled, copy it, and submit it themselves to the same block. If they add more gas (effectively a bribe), they can also have miners prioritize it over yours! Basically, they can impersonate you to the smart contract.

Commit + Reveal. Luckily though, cryptography is our friend. We can actually hide a JWT by submitting a cryptographic hash of it. Only somebody who knows the same JWT can reproduce that same hash — realistically speaking, that is. Technically, it’s possible to produce the hash without knowing the JWT, but for even a supercomputer to do so, it would take more time than the earth has existed for, and not the biblical literalist age 😋

Then, once the block where we’ve committed to this JWT by giving its hash is finalized, we can reveal the actual JWT. The smart contract can then reproduce the hash commitment since it now knows the JWT, and it can see we had committed the JWT in an earlier block. Therefore, we couldn’t have just stolen someone’s JWT from the same block. We had to have known the JWT beforehand to produce its hash.

Enforcing Ownership. You may have noticed a flaw in this logic — somebody can frontrun the commitment (the hashing) in addition to the reveal. I.e., they can copy the hash in the first block and then copy the reveal in the later block. So it seems as if they knew the JWT in the first place, because they knew its hashed value. To fix this, instead of just commiting hash(JWT), you can commit hash(JWT + your address). If an attacker copies this commitment, they don’t gain anything. After you or they reveal the JWT, the smart contract can see it was linked to your address, not theirs.

Figure 2. Commit/reveal to prevent frontrunning.

Tying it all together

Now, you can see an overview of the web token forwarding:

1. User has a blockchain address and JWT of a web2 account
2. User commits hash(JWT + blockchain address)
3. After block, user reveals JWT
4. Smart contract verifies that commitment of (JWT + blockchain address) exists for user’s address
5. If it exists, smart contract verifies the signature on the JWT
6. If it exists, smart contract reads other data from the JWT, such as which web2 account it belongs to
7. Smart contract links user’s address with user’s web2 account

What next?

We encourage you to build the following with Holo:

• Web3 marketplaces with verified sellers
• NFTs with embedded identities to prevent counterfeits or link to Twitter accounts
• DAO tools that can interact with Discord accounts on-chain
• Crypto payments to a Github or Twitter handle
• Noncustodial wallet recovery using Holo + Lit protocol

We’re a team of buidlers want to make it as easy to BUIDL on as possible. Hit us up on our discord channel and look at our docs to begin! Or visit our Twitter for updates.

Further Reading

Provable and Computable Identity for Future-Proof Scientific Workflows

Originally published at https://hack.opsci.io on July 28, 2022.

Linking Web Accounts on-Chain for Fun & Better DeSci was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.

💫 OpSci and Algovera have brought together key innovators in AI and distributed web technology to present a web-native vision for scientific research communities that prize collaboration, transparency, and interoperability. You can catch the session on May 24th between 10:00a and 11:30a CET as part of the DeSci.Berlin event.

Session Abstract

Access to the tools and resources required for studying human society, the natural world, and our collective impact on it, have been traditionally limited to those with privileged access. Currently, a fresh wave of data is being onboarded to a new open source infrastructure, opening up vast new opportunities for secondary use-cases. Digital communities all around the world can now access massive volumes of data and collaborate in real time, increasing the collective throughput of scientific research and development of real-world applications. Raw data has latent value that requires processing to extract insights and enable useful predictions, e.g. using machine learning and AI. However, this is a complex process involving many steps and moving parts. Decentralization offers potential benefits, but development of the underlying infrastructure is early-stage and fragmented. Key bottlenecks include access to compute and storage resources, tools for transparency and governance, and limitations of current intellectual property. In this session, we explore current developments in decentralized AI infrastructure, such as decentralized web storage, accessible compute markets, decentralized autonomous research communities, and knowledge artifacts as synthetic non-fungible assets — with a particular focus on open science communities.

Composable DeSci Flows Via Token-Wrapped Data & Algorithms

Trent McConaghy, Ocean Protocol

Talk Abstract: Science is about making models to predict unseen events, in a reproducible way. It’s data all the way down. Data can be raw measurements, cleaned data, features, scientific models, or predictions. Data can be algorithms to build the models. Data can be dynamically changing.

This talk describes how to use Ocean for composable, reproducible data flows by tokenizing data & algorithms: ERC721 data NFTs to wrap the base data/algorithms, and ERC20 datatokens as access control tokens. Compute-to-Data flows preserve privacy and control. Data & algorithms can use Web3 storage or compute.

Data NFTs and datatokens are the atomic building blocks. They naturally interoperate with Web3 wallets, DAOs, DEXes, and other Web3 tools. From this, we can construct many DeSci flows, from AI-training provenance to scientific model commons to algorithm marketplaces.

Trent McConaghy co-founded Ocean Protocol. He has worked on AI since 1997 (circuit synthesis, statistical verification), and blockchain since 2013 (incl. Token Engineering, agent-based simulation, and Ocean). He holds a PhD in EE. He’s written two textbooks and 40+ peer-reviewed papers and patents.

The Decentralized AI Stack

Richard Blythman, Algovera

Talk Abstract: The AI stack has many components that need to come together for building useful applications, including storage, compute, marketplaces, access control, funding and organizational structure. We will outline some of the work that Algovera has been doing to decentralize and integrate various layers of the stack, as well as some of the applications being developed in our ecosystem of independent AI teams.

Richard is a machine learning engineer with a research background. He co-founded Algovera to tackle issues he experienced in big tech, universities and startups around applications, funding, and the time taken to develop AI apps.

Decentralized infrastructure for science and Compute over Data

Christopher Hill, DeSci Labs & David Aronchik, Protocol Labs

Talk Abstract: Together, Protocol Labs and DeSci Labs are democratizing access to scientific infrastructure by creating technologies to promote reproducibility, replicability, and recalibrate the incentives to produce and verify new knowledge. The ability to run compute, whether for data processing or serving (e.g. microservices) is a major missing piece of the DeSci stack. By enabling spare compute power globally to be contributed to an open market of providers and scientists, the cost of compute can become accessible to all scientists, promoting global knowledge sharing.

Christopher Hill is a co-founder of DeSci Labs. Chris is an interdisciplinary scientist who worked at the crossroads of neuroscience, economics, and machine learning. He holds a PhD in Neuroeconomics.

David is Lead of Research Development at Protocol Labs, helping, deploying and organizing our community building the next generation of the Internet. Previously, David led Open Source Machine Learning Strategy at Azure, product management for Kubernetes on behalf of Google, launched Google Kubernetes Engine, and co-founded the Kubeflow project and the SAME project. He has also worked at Microsoft, Amazon and Chef and co-founded three startups.

Decentralized AI for planetary regeneration

Rafael Kaufmann, Digital Gaia

Talk Abstract: We are building prosthetic brains for ecosystems to support and automate decision-making in impact investment. We will give an overview of the technical architecture and network incentives, and highlight some interesting design considerations.

Raf is a technologist with experience encompassing physics, economics, AI, statistical and complex systems modeling, product management, entrepreneurship, and organizational development. After 6 years learning about collective intelligence and applying it to scale organizational effectiveness at Google, he co-founded Digital Gaia with the mission of building Earth’s neocortex.

Thank you to Richard and Algovera for working together to make this event happen! You can learn more about Algovera at the accompanying blog post on their site.

Join the OpSci Movement

Does the idea of a free, open, internet of science ring a resonant chord with you? Consider joining the OpSci community to learn, connect, and collaborate with others building a commons for co-discovery.

Originally published at https://hack.opsci.io on May 19, 2022.

DeSci Berlin: Decentralized AI Infrastructure for Scientific Research in Autonomous Communities was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.

Holonym is a decentralized protocol for bridging credentials across web applications, blockchains, and physical records. The protocol was originally developed to support the needs for scientists building research web apps in Web3. You can check out the DeSci DID registry (alpha), a Web3 scientist “yellowpages,” powered by Holo at https://verse.opsci.io.

🤖 Automation in Science Requires Strong Identity Claims

Decentralized Science (#DeSci) is a nascent web-native movement seeking to deploy scientific applications on decentralized internet protocols that are autonomous and community owned. The automation of scientific web services — such as permission-less publishing, laboratory protocol marketplaces, and grant making — requires a portable persistent identifier that links data across applications, verifies credentials, and maximizes the discoverability of research activity. Most importantly, this identifier must be controlled by who it describes and protect against impersonation. Current infrastructure for assigning identifiers to academic entities falls short of this requirement list, yet there is a chance to bridge existing tooling with new technology to solve these problems.

“Holonym, is an idea inspired by Nick Szabo, the first to introduce smart contracts and virtual personae in the late 90s. A Holo is defined as a whole greater than the sum of its parts, where each linked Nym provides a distinct bit of information about a person.”

💠 Holonym is a Decentralized Identity Protocol

OpSci has translated the requirements for automated scientific web services into Holonym, an autonomous protocol for linking web accounts, credentials, and identities under a unique persistent decentralized identifier (DID). Holos are a permanent reference to a unique entity and may act as a service endpoint for authorized applications to access and compute on associated metadata. The name Holonym, is inspired by Nick Szabo, the first to introduce smart contracts and virtual personae in the late 90s. Holo is defined as a whole greater than the sum of its parts, where each linked Nym provides a distinct bit of information about a persona.

The Holonym protocol is decentralized, meaning that the protocol is non-proprietary, and runs on a global network of nodes. The code is open source and available for anyone to implement, integrate, or fork into their own application. Holos are self-sovereign, meaning that they are controlled by the identity’s owner. Each Holo can only be created or modified by the entity that owns the cryptographic keys used to sign and authorize actions on the decentralized networks that the protocol is deployed on.

🔗 Holonym is an Identity Layer for all dApps

Holonym is a low-level persistent identity layer for the internet that enables a variety of use cases, specifically for decentralized Apps (dApps) that require access to verified user metadata. Any dApp can access a publicly listed Holo, filter by, or compute on, associated data and resolve the results on the front-end of any arbitrary website. In other words, developers have incredible flexibility to write versatile “view functions” on the underlying Holonym database.

What are Unstoppable Applications?

dApps are web applications that execute their instructions, or smart contracts, on a distributed network of nodes that ensure tamper-free consensus and no single point of failure. Consensus is typically implemented on a blockchain, where the previous history of all dApps is public and permanently stored by the nodes in the network. dApps have the advantage over centralized web services in that they are interoperable and autonomous; this means their activity is completely transparent, under no central control, permanently recorded, can’t crash, be turned off, or shut down.

What can I build with Holo?

Holonym smart contracts enable accounts to be aggregated into a single Holo and indexed by the public cryptographic key of the owner. Public key lookup is a simple but powerful feature that allows the owner to link their interactions on a distributed network to “off-chain” and “offline” activity. Some examples of “view functions” that link a user’s data could include:

• Discoverable Content by Verified Creators: A bibliographic database of verified academic authors, where anyone can filter by field, institution, altmetrics, or perform regular expression search to find and directly request a paper from an author through a peer-to-peer routing protocol like IPFS.
• Reputation Leaderboard: A leaderboard directory of discoverable users ranked by reputation that have contributed to a Decentralized Autonomous Organization (DAO) by holding or staking governance tokens, participating in token-based voting, or executing smart contracts for DAO operations.
• Token Gated Listings: A listing board where members are only allowed to post if they have met some minimal deterministic criteria that can be executed by a smart contract and receive a token that makes them discoverable. For example, a medical imaging consultant that has passed training for administering a new device and received a Non Fungible Token (NFT) as a certificate proof of completion.

Many more possibilities abound, where a developer’s imagination is the only limitation 💫

🌊 Streamlining DeSci dApp Workflows

OpSci’s mission is to build infrastructure in support of web-native scientific communities. In the short time we’ve been around, a rising wave of DeSci projects have emerged. We built Holonym to provide a primitive for identity verification and support the most promising use-cases in this emergent space: Permission-less Journals, Decentralized Science Markets, and Grant Making.

Journals of the Future: Automated Peer Review, Publishing, and Archival

A “permission-less” automated journal is a hypothetical dApp where any author can submit an article, have their content archived permanently, get matched with relevant peer reviewers, and publish their paper into mainstream indices completely automatically, without requiring pre-approval from an editor or administrator. There are multiple challenges to be solved for a permission-less journal, most stemming from verified identities.

1. Authors must not be able to impersonate other scientists to take credit for their work
2. Peer reviewers must be able to prove their expertise or credentials in order to be appropriately matched with papers
3. Integration with legacy academic publishing web services will require authors to present academic credentials or identifiers

Decentralized Marketplaces for Laboratory Services

A decentralized science marketplace is a platform where service providers can publish laboratory protocols, or analysis workflows that can be executed on demand and their results made securely available to the consumer. A key challenge for market operators is moderating and ranking listings by reputation and quality of services offered. Market consumers are more likely to purchase services when service providers can prove their reputation.

Science DAO Grant Making

A highly active field of the DeSci movement is exploring novel mechanisms for crowdsourcing funds to issue grants for scientists performing high risk research that doesn’t fall into government funding mandates. Some early pioneers have begun accepting crypto as donations, sold Non-Fungible Tokens, either artistic or legal claims to the underlying IP, or raised capital through token offerings. More sophisticated grant making will require the introduction of reputation and verified identity in order to scale as a decentralized service.

For example, consider a science DAO that offers services for industry and collects a transaction fee for each purchase. That transaction fee is deposited into a treasury that awards grants on a quarterly cycle. Who should grants be issued to? How is the mechanism protected against spam, or impersonation attacks? What is the overhead for due diligence?

What do DeSci dApp Developers Need?

We’ve spoken to builders in the DeSci space and key pain-points for deploying autonomous science dApps are:

1. Trusting users are who they say they are
2. Linking off-chain data to a provable identity
3. An open directory of DeSci users, contributors, members and their associated activity

The most common request to address these problems is easy integration of ORCID, a centralized persistent identifier that has been adopted by approximately 50% of academic researchers worldwide. However, ORCID falls short on multiple accounts before it can be adopted by DeSci DAOs:

• ORCID is owned and maintained by a centralized organization, not the stakeholders
• API access to underlying metadata is a paid feature only available to members
• Users do not truly own their identifiers, the keys are stored on private servers
• ORCID does not produce verified claims of academic credentials

📚 For the Record: A Short History and Future Roadmap

Holonym was born out of a call to action from the Open Science DAO in January 2022 to reimagine how web-native academic credentials could be improved using principles of decentralization, open source, community governance, and self-sovereignty. The idea was incubated in the months leading up to ethDenver, where the project was announced at the DeSci hackathon project pitches session, a part of the DeSci@ethDenver conference.

A team was formed around the project pitch and we quickly deployed a prototype smart contract that forwarded web tokens issued by identity providers, such as ORCiD or Google, to a database stored on the Polygon blockchain and indexed by 0x Ethereum formatted public addresses. The web token forwarding (WTF) hackathon project won three awards from ethDenver judges for our novel contribution to on-chain, smart-contract compatible identity verification.

Inspired by our early success within the Harmony ecosystem, we established a 6 month incubatory product roadmap with a call for open participation and are currently exploring funding avenues.

🚀 Alpha Launch at ethAmsterdam on DeSci Day

The DeSci Identity Registry hosted on https://verse.opsci.io will be the first demonstrated use-case of Holonym and will be revealed in Amsterdam on April 20th, 2022 during DeSci Day as part of the Ethereum Devconnect conference. We will support free account verifications on Gnosis Chain by airdropping new users that hold the Holonym DeSci Day Proof of Attendance Protocol (POAP) token that proves their attendance at the event on that day.

We would like to announce a second call to action for developers to join us during the ethAmsterdam hackathon to explore novel use-cases of Holonym, both in DeSci-dApps and beyond!

✨ How to Get Started with Holonym: Simple Developer Integrations

Projects can integrate Holonym into their project with a few lines of code. If you are developing a web dApp, use the wtf-lib package (the package gets its name from the protocol underlying Holonym, the web token forwarding protocol). First, install the package from the command line.

npm install wtf-lib

Then import the library into whatever files need to use Holonym.

import wtf from 'wtf-lib'

Configure wtf to use your JSON RPC provider. Whether you are using Moralis, QuickNode, or something else to access blockchain networks, wtf only needs the URL endpoint for the JSON RPC node. The following line tells wtf to use the public Mumbai testnet RPC node for all queries to Holonym contracts on Polygon.

wtf.setProviderURL({polygon : 'https://rpc-mumbai.maticvigil.com'})

We can now query Holonym. One of the most useful features of Holonym is the aggregation of various web profiles. We can retrieve a user’s entire Holonym profile with the getHolo() function.

const holo = await wtf.getHolo(userAddress) const { name: name, bio: bio, orcid: orcid, google: google, twitter: github, github: github } = holo

The holo object contains a key-value pairing for the user’s name, bio, and for every service supported by Holonym. The value of the orcid variable, for instance, will look like “0000–0000–0000–0000” if the user has linked their ORCID to their Holo. It will be null if the user has not linked their ORCID.

We can also get an object of all crypto addresses that have Holonym profiles.

const addresses = await wtf.getAllUserAddresses()

The addresses object contains lists of crypto addresses, sorted by network and by which service the address has associated with their Holo. For example, a user who has linked their ORCID to their Holo on Gnosis Chain will be in the following list:

const usersWithOrcidOnGnosis = addresses['gnosis']['orcid']

The wtf-lib package provides a number of other views on the Holonym user registry. You can learn about them in the docs here, and you can check out the GitHub repo here. Don’t hesitate to open an issue on GitHub or reach out for integration support in the channel on the OpSci discord!

🪜 Next Steps: Integrations, Requested Features, & Community Launch

The next few months, we expect to explore additional niche use-cases to discover the most pressing concerns for those building web applications in Web3 and that need discoverable identities. We are excited to work with DeSci DAOs and developers that want to integrate identity verification into their application or community tooling. If you are a developer and interested in learning more, we want to hear from you! Please consider completing our survey.

In the spirit of progressive decentralization, we expect to pass governance to the community by assigning the cryptographic keys that control the underlying smart contracts to a decentralized autonomous organization (DAO). Once the tooling is mature and the protocol is stable, we will leave it to the community to decide the best way to capture the value of an automated identity verification protocol.

🙏 Thanks to…

David from Lit Protocol, the team at Boost VC, DeSci Labs, Renee from talentDAO, Ted from Proof of Humanity, Niklas from LabDAO, and Patrick from ResearchHub for feedback, advice, and support for integrations. Thank you to the Harmony Grants DAO for choosing Holonym as one of the winners at ethDenver.

Resources & Links

DeSci Identity Registry

holonym.id

docs.holonym.id

Join the OpSci Movement

Does the idea of a free, open, internet of science ring a resonant chord with you? Consider joining the Opscientia community to learn, connect, and collaborate with others building a commons for co-discovery.

Originally published at https://hack.opsci.io on April 19, 2022.

Provable and Computable Identity for Future-Proof Scientific Workflows was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.

The way scientific data is stored and shared today does not adequately leverage the currently available digital infrastructure for persistent storage, intelligent queries, and standards for machine readability. In this next installment of the article series, Rich in Data, Poor in Wisdom: Science Needs a Decentralized Data Commons, we introduce a pilot experiment to permanently archive hundreds of terabytes of open-access neuroimaging datasets on peer-to-peer storage networks.

The Short Lifecycle of Scientific Data

Massive volumes of high quality data on human health, social activity, the environment, and our impact on the planet are routinely collected, analyzed for insights, and used to guide evidence-based decision making that fuels innovation and progress.

But what happens to high quality scientific data when all the original questions are answered?

For the millions of scientific articles likely published in 2020, a record breaking year for science publishing, less than half were published with their data. Open access repositories for research data such as Dataverse, Dryad, Figshare, and Digital Commons have been around for almost a decade. However, the likelihood of a dataset remaining extant is still estimated to decline 17% every year after publication.

Why is it that scientific data can be so easily be lost when web technology is at the height of cultural adoption, energy consumption, and economic impact?

Barriers for an Open Data Commons

Scientists have cited several systemic and technical blockers that prevent them from sharing their research data. The most common concern lies with the fear of losing intellectual ownership and suffering a lack of attribution for work. Why share the expensive data that I have painstakingly collected for others to publish before I can?

Richard D. Morey on Twitter: "Who wants a #researchparasites badge? (apologies to @OSFramework and @BrianNosek :) ) #openscience pic.twitter.com/S8USgzICbz / Twitter"

Who wants a #researchparasites badge? (apologies to @OSFramework and @BrianNosek :) ) #openscience pic.twitter.com/S8USgzICbz

Data Parasites are a virtual scientific boogeyman for some principal investigators collecting valuable and expensive scientific data. The “Data Parasite” is a pariah label describing researchers that publish high volumes of articles using open data collected by other groups.

Peer-reviewed publications are the reputation metric for academics, the so-called Currency of the Realm. It is not difficult to grasp how very little incentive exists for researchers to share any data that places them at a disadvantage from securing future funding for their own projects.

For the researchers that shrug the adversarial Nash Equilibria that dictate the availability of knowledge to the public, several technical barriers remain to be overcome. Scientific datasets sometimes contain sensitive information and must be painstakingly screened before publishing. Data must also be properly annotated and formatted into schemas that can be understood and applied by other researchers.

Data is not useful if the following is absent:

1. the context in which it was collected
2. the history of modifications, and
3. verification of integrity.

Even after overcoming several cultural, institutional, and technical challenges researchers must still deal with poor infrastructural support and high cloud service costs for scientific data repositories. In the best case scenario, a costly lease for data storage is obtained on university or third-party servers for some limited period of time. These centralized solutions pose a risk for valuable data in the event that the services suffer a critical failure, and this happens more often than we’d like.

DeSci: A Truly Public Records Archive

Automated pipelines for dataset archival, contribution tracking, and virtuous value feedback loops built on decentralized web technology may provide solutions to some of these challenges. The Filecoin network is a hallmark example of incentive engineering that subsidizes the cost of high quality data to provide free storage for verified providers such as library archivists, public service providers, and other public goods. Scientific data archived on Filecoin is content addressed and can be indexed with versioned databases stored in a distributed network to track contributions, history, and ensure integrity of requested content.

Here we describe an approach to leverage public scientific data aggregators to automate free archival pipelines for massive datasets distributed across several service providers, academic servers, and centralized repositories.

Wading into the Ocean of Neuroimaging Data

Large data collecting consortia have generated a deluge of neuroimaging data in their quest to map the brain at multiple scales. Several permissioned repositories have emerged to share data across institutions to authorized contributors. Standards for interoperable data analysis and machine learning workflows have also gathered adoption for those working in these semi-closed ecosystems.

Tools like DataLad, have emerged as an open source utility to manage datasets distributed across institutional infrastructure. DataLad also maintains a metadata index of published data, providing a starting point for a public records archive to take form for all of neuroscience.

The DataLad team has aggregated an index of over 280TB of neural data at datasets.datalad.org. This data also contains detailed versioning, notes, and machine readable data to support automated workflows.

However, despite the ability to define many remote sources for datasets, some data still becomes inaccessible due to faulty servers. DataLad is excellent for indexing datasets, but the hosting servers might themselves become unavailable over time in the absence of maintainers. To fix this, we need an automated process to fix broken links — repair workers that poll for missing files and fix the data proactively.

Decentralized web protocols offer a potential improvement in metadata portability, data availability, and content integrity through the use of:

• content addressable data — data is indexed by its content instead of its location, solving the issue of broken links if there are enough replications of that data
• programmable incentives for data storage — rewards for reliable storage providers, to improve reliability of the network
• provenance tracking — who created the dataset, who were the study participants, what changes were made and when
• censorship resistance — due to their distributed nature, and reliance on widely used protocols
• bandwidth & speed that scales with global adoption — more people storing the data, more data availability
• data sharing agreements embedded within the data- using encryption and role based access control for enforcing agreements automatically. Royalties can also be programmed in.

First step: Going from DataLad to Filecoin

Efforts to index data together at one place (like DataLad) inherit the same problems as their original data sources did. We need an easy to use pipeline to archive data verifiably, and redundantly, for the long term.

The key hypothesis:

decentralized, verifiable, long-term storage of important scientific data is key for lowering barriers for doing science

As a first step, we’ve been working on moving the widely used datasets indexed by metadata aggregator datasets.datalad.org onto Filecoin storage. In this use case, we demonstrate an automated pipeline that scrapes data from DataLad and archives it for long term storage on Filecoin, providing one piece of the puzzle towards sustainability and maintenance of public science goods. The goal of this experiment is to identify bottlenecks in the process and think of good metadata structures that are forward-looking.

Why Filecoin?

Filecoin is a decentralized storage protocol that uses incentive mechanisms for getting reliable and verifiable long-term storage. Storage providers get block rewards for being dependable, and bad actors get their stake slashed. Also, by creating essentially an open marketplace for providers, the storage costs for the user are driven down tremendously.

Filecoin is ideal for long-term archival for the following reasons:

1. Provably unique copies of the data, or Proof-of-Replication. We can make redundant copies of the same data, and get proof that they are actually replicated.
2. Verifiable storage throughout the time period agreed to in the deal, or Proof-of-Spacetime. We can verify that the storage providers continue to store the data, and don’t drop the files in between.
3. Usage of a general tree data structure (IPLD) for files. This combined with the upcoming Filecoin Virtual Machine (Smart Contracts layered on top of storage) will allow a variety of ways to access and process the data while lowering the cost. For example, we’ll be able to request for retrieval of only a subset of files — say, all files that match a query we write.

You can learn more about PoRep and PoST here and here

Outline of our Approach

On a high level, there are four types of entities involved in the process of on-ramping datasets from DataLad’s remotes onto Filecoin archival:

1. Servers — hold neuroimaging data. These might be hosted on-premises universities, or on cloud compute infrastructure.
2. Our Bridge — preprocesses the data, creates auction requests to the Textile Broker, and makes datasets available to the miners in a required form.
3. The Textile Broker — coordinates with miners to create storage deals.
4. Miners — store the packaged data for a specified length of time, in exchange for mining rewards.

Walkthrough of the entire process:

1. A dataset is selected from the DataLad index and its contents are periodically fetched into a storage droplet hosted on DigitalOcean.
2. The first pre-processing step is to catalog the data and clean up any unavailable file pointers. Files might be missing due to some of the DataLad remotes becoming unavailable. This unavailability due to servers going unmaintained is one of the major problems we’re solving. It is also worth noting that the original version history is preserved, and archived along with the dataset.
3. In the second pre-processing step, data needs to be packaged nicely before it gets hauled off to Filecoin miners. There are several points to consider at this step:
   i. Filecoin requires data to be packaged into Content Addressable aRchives (CAR) — these are serialized archives of the IPLD DAG constructed from the dataset. We use the Powergate CLI tools for this step.
   ii. The Dataset needs to be smaller than 32GB. Anything larger has to be broken up into smaller chunks, each stored separately. This requires addition of another step during retrieval where we maintain metadata of all the required pieces of a particular dataset, and join them together when we need to access the data.
4. The CAR file is made available for miners via an HTTP server. We set up a basic NGINX server for this.
5. An auction request is sent to the broker. The broker then orchestrates storage deals with miners in the network.
6. Miners who accept the deals proceed to pull the data from our server, perform various checks, and finally store the data in their node.

The auction is finished when either of the two conditions are satisfied:

1. The deal deadline expires.
2. The auction has been accepted and data stored by the specified number of miners in the request.

Some numbers and statistics

Each step in the preparation process takes some amount of time.

Step 1: Downloading the data. This is the fastest step in our pilot experiments, taking a few minutes at most. Time taken in this step will increase linearly with the dataset size (some of the datasets we’ll be dealing with are 10s of TB large).

Step 2: The first step is to convert the data into a serialized IPLD DAG, called the CAR. This process is linear in the size of the input:

If we extrapolate these numbers for our server running on a droplet, where a 2.05GB file took 191.47 seconds, so a 32 GB file will take 2988s or ~49 minutes.

Step 3: The next step is to host the file on a server to make it available to miners, for the duration of the auction. This step takes the longest of the lot, 5 days is the recommendation for the auction deadline, to give the storage providers enough time to pull the data off our server.

The duration for step 3 is capped at 5 days since each auction has a cap of 32Gb on the data size (miners will pull at most 32 GB per auction). For data larger than 32 GB, we’ll have to chunk the data into multiple pieces, and auction them off separately, each auction still taking 5 days at most.

The way forward

Outstanding technical challenges

We have a grand vision, but foresee several problems to be solved before we get there.

Balancing version control with quality archival — Filecoin archival is best thought of as taking “snapshots” of the current state of a dataset. If the dataset is updated in the future, how do we “patch” these snapshots? What is the best frequency of re-archival, if we were to batch these changes? What kind of logic would we need to re-assemble the version history from a set of snapshots?

Getting both low-latency access and redundant, long-term storage — Filecoin is great for storing large amounts of data for the long haul, but the layers of security are at odds with retrieval speed. Every retrieval requires making a new retrieval deal, and takes time.

Exciting research is ongoing on this front as well. Estuary provides combined IPFS and Filecoin storage, automatic re-pinning and deal making on expiry, as well as automatically breaking up larger files into smaller chunks for storage.

Another option is layering computation on top of, and co-located with, storage infrastructure. This will afford more fine-grained control over data retrieval, while also reducing costs. This is the idea behind the upcoming Filecoin Virtual Machine — smart contracts running alongside incentivized long term storage.

Adjusting retrieval methods to fit each use case — how do we structure the query methods so suit different data structures? In some cases we’d like to optimize for fast “join”s between two tables (like we do in SQL or labelled multi-dimensional arrays). In another case like data visualization, we would like low-latency retrieval for spatially adjacent data points — fetch the data in the chunk one unit to the left.

Other components of the DeSci stack

We’re working towards the goal of a self-sustaining, open science community; complete with democratized funding for new research, highly available and verifiable open datasets, and reputation tracking for participants in the ecosystem. Several pieces of the puzzle are yet to be built before we get the flywheel of science turning on its own.

• Funding mechanisms for important public goods: storage infrastructure, overhead requirements for deals with providers, and running content nodes and gateways.
• Rich and interoperable metadata standards that link across domains.
• Open data markets — keeping in mind regulatory compliance, data provenance, and keeping bad actors out.
• Standards for science-specific DID profiles with researcher info, affiliations, ORCID

We are currently working on closing the loop of archival with tools for retrieval. We already have a working DataLad remote for IPFS, and once we bind together Filecoin archival with IPFS retrieval, everyday DataLad users can begin to benefit from this work.

Join the Decentralized Open Science Movement

Does the idea of a free, open, internet of science ring a resonant chord with you? Consider joining the Opscientia community to learn, connect, and collaborate with others building a commons for co-discovery.

Articles in This Series

1. Decentralized Content Networks for a Permanent Science Data Commons: IPFS
2. Engineering Incentives for Data Storage as a Commodity: Filecoin
3. A Permanent Web of Linked Data: Arweave
4. Peer-to-Peer Storage without a Blockchain: Storj
5. One of the First Decentralized Cloud Storage Platforms: Sia
6. The World Computer’s Hard Drive: Swarm
7. A Permissionless Data Layer for Science Web Apps
8. OpSci Commons: A Decentralized and Autonomous Knowledge Commons

References

DataLad. (n.d.). DataLad. Retrieved January 20, 2022, from https://www.datalad.org/

DANDI. (n.d.) DANDI Archive. Retrieved January 20, 2022, from https://gui.dandiarchive.org/#/

Park, Y., & Greene, C. S. (2018). A parasite’s perspective on data sharing. GigaScience, 7(11), giy129. https://doi.org/10.1093/gigascience/giy129

Pronk, T. E., Wiersma, P. H., van Weerden, A., & Schieving, F. (2015). A game theoretic analysis of research data sharing. PeerJ, 3, e1242. https://doi.org/10.7717/peerj.1242

Toulas, B. (2021, December 30). University loses 77TB of research data due to backup error. Retrieved January 20, 2022, from https://www.bleepingcomputer.com/news/security/university-loses-77tb-of-research-data-due-to-backup-error/

Vines, T. H., Albert, A., Andrew, R. L., Débarre, F., Bock, D. G., Franklin, M. T., Gilbert, K. J., Moore, J. S., Renaut, S., & Rennison, D. J. (2014). The availability of research data declines rapidly with article age. Current biology : CB, 24(1), 94–97. https://doi.org/10.1016/j.cub.2013.11.014

Wiener-Bronner, D. (2013, December 23). Most Scientific Research Data From the 1990s Is Lost Forever. Retrieved November, 13, 2021, from Most Scientific Research Data From the 1990s Is Lost Forever — The Atlantic

xarray: N-D labeled arrays and datasets in Python. (2022, January 28). Xarray. Retrieved January 28, 2022, from https://xarray.pydata.org/en/stable/index.html

This post was written in collaboration with Shady El Damaty and Yaroslav O. Halchenko. Thanks to Matt McCormick for suggestions.

Originally published at https://hack.opsci.io on January 28, 2022.

A Permissionless Data Layer for Science Web Apps was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is it?

The ethDenver hack-a-thon and generic web3 palooza is taking place from February 11–20, with an estimated 20k attendees this year.

What’s happening?

The event includes a hack-a-thon with over 500k in prizes and bounties spread out across several tracks. #BUIDL week kicks off on the 11th at the Art Hotel and continues until the 17th. The in-person hack-a-thon may transition to a virtual hack-a-thon after the conference is concluded.

The main event begins on the 17th and includes significant keynote speakers before ending with an awards ceremony for winning hack-a-thon projects followed by an after-party.

After the conference, there is a retreat in the Rocky Mountains at the Breckenridge Mountain Resort. Conference events and socials will also take place amidst the fun in the snow.

How to get involved?

There are some stirrings to get a few official DeSci events on the calendar. If you would like to participate in any conference events or apply for a travel grant please complete this registration form.

If you plan at all on attending, REGISTER NOW, both on ethDenver’s site and complete the OpSci registration form linked above.

OpSci Events in the Works

• DeSci Social
• OpSci Metaverse Science Avatar Hack-a-thon Project: NFT profile project that displays interactive statistics, achievements, and open science contributions in-browser.
• DeSci Panel with 15minute lightning talks and Audience AMA: DeSci Data Commons, DeSci IDs & Reputation, Data Archival Workflows, A Short History of DeSci

Originally published at https://hack.opsci.io on January 15, 2022.

ethDenver #DeSci Events and Travel Grants was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article was originally published on the NeurotechX Content Labs Blog

It’s no secret. Tech companies have amassed vast fortunes by predicting the behavior of online users using rudimentary metrics such as clicks, swipes, and taps. Today, corporate coffers are full and ready to fund research on direct measures of how you think, feel, and react when plugged into the web.

70% of all new economic value created by tech companies in this next decade will be claimed by digital platforms and services that commodify all aspects of our life, predicts the World Economic Forum. The majority of this value will be harvested from our personal data, including biometric brain data gathered through neurotechnologies.

Neurotechnologies are steadily becoming more integrated into everyday wearables (e.g. headphones, earphones). More invasive neurotechnologies that have existed for decades in a clinical capacity (for example, to treat conditions such as depression and Parkinson’s disease), are also now undergoing commercial development. Most notable of these is Elon Musk’s Neuralink and the dark horse Synchron.

Advances in neurotech are hurtling forward without taking the time to ensure adequate space for ethical discussion, particularly around privacy protection. Nevermind that our very brains are becoming public commodities. Most of us can agree that predicting and influencing user behavior with overt metrics is one thing, but taking a peek inside unsuspecting users’ heads to make a buck may be a step too far. The fight for privacy protection today, like the pioneering Chilean Neurorights Bill, is a fight for Self-Sovereign Brains in the not-too-distant future.

If you ask us, our thoughts are an intimate expression of self, agency, and identity and should always remain sovereign. At the same time, neurotechnology is an inevitable eventuality with compelling advantages for the future of neuroscience, knowledge-sharing, and health.

Fortunately, recent advancements in peer-to-peer decentralized technology and cryptography may minimize potential privacy risks while unfettering the benefits of neurotechnology toward a prescient view of a majority digital economy in the next decade.

Consumer loss of privacy: How it started

The first two decades of the 21st century have seen the mass automation of economic engines of value creation on digital platforms engineered to engage and retain users. Personal data troves amassed by software service providers are transmuted into predictive models of human social activity, commerce, and patterns of decision-making to drive optimized revenue-generating products and services.

This first wave of personal data aggregation has so far been limited to external signals of user intention and behavior; an evolution of the infamous Tinder-swipe. Essentially, cognition and preference are being inferred by motor metrics alone.

Clicks, taps, swipes, speech, and text have served very well as motor inputs for training machine learning models to predict decision-making, with exceptional success in behaviour modification, ranging from purchasing decisions to voting.

Despite this abundant power, Big Tech has continued to test the boundaries of what degree of privacy consumers are willing to sacrifice in return for services. For example, in 2010 Google attempted to transition into more intimate metrics of the mind itself, with eye-tracking technology acting as a proxy for cognitive states to boost ad targeting. At the time, this provoked a strong guttural reaction from consumers and anti-privacy groups.

For Big Tech, the issue was timing, not the technology itself. Given enough time, the convenience offered by service platforms engineered to maximize feelings of pleasure would eventually outmatch user caution.

Consumer loss of privacy: How it’s going

The pace of innovation over the past decade has resulted in a range of non-invasive technologies that passively capture and use our biometric data. Today, smart watches read our heart rate and emojis respond to our facial expressions. With the rise of consumer neurotech wearables, our brain metrics are now up for grabs.

Low-cost and non-invasive neurotech “wearables” are widely available. These existing devices are capable of decoding both general brain activity (e.g. inferred levels of stress and concentration) and specific thoughts (i.e. mental commands) through electrodes touching the scalp. Current solutions range from relaxation apps that measure a user’s cognitive state to modify virtual worlds, to “pass-thoughts” (as an alternative to traditional passwords), and protocols around “risk reduction” (for example flagging when a person operating machinery is drowsy).

According to the Neurorights Initiative, current neurotechnologies have the potential to infringe on our rights to mental privacy, personal identity, free will, and protection from algorithmic bias. For example, invasive technologies have been documented as disrupting people’s sense of identity and challenging people’s sense of self. One patient undergoing stimulation for depression reported, “it blurs to the point where I’m not sure … frankly, who I am” (Yuste et al., 2017).

Manipulation of brain activity through neurotechnologies confers risks in terms of removing ultimate control over our own decision making. For example, it would already be possible with our current neurotechnologies to learn “brain states” associated with particular spending or browsing habits. In turn making it possible to take advantage of, and even induce with targeted content, favourable brain states.

Informed consent is also a key ethical issue. Tech demos pitched as company “recruitment drives” could in fact be considered veiled PR campaigns for human clinical trials of invasive consumer neurotech devices. In order to uphold citizens rights to informed consent, science will have to adapt quickly to navigate this new blurring of science and spectacle. One such concern is mass elective brain surgery to implant devices ultimately controlled by one central company, board of directors, or even perhaps subject to the whims of a single influential personality.

A Catalyst for Rapid Digitalization

The sudden digitalization of society accelerated by COVID-19 has signaled a cultural shift in attitudes towards consumer services powered by biometrics. A new swell of interest in vaccination and digital health records, biometrics for security, automation of civic services, and personalized health has emerged. With this, regulators are struggling to keep up with this rapid pace of innovation.

A Capitalist’s Carte Blanche

The United States of America, while retaining an overwhelming share (66%) of the digital economy, has not yet passed comprehensive personal data protections, let alone recognized the unprecedented danger of social engineering for consumerism.

The most comprehensive legal precedents have been established by the 2018 California Consumer Privacy Act. Experts have pointed out that the CCPA classifies biometric as generic personal data, providing regulatory enforcement gaps that may endanger unwitting consumers.

The lack of special protections for this data, such as verifiable informed consent, enables prospective companies to establish precedents for massive data harvesting and social engineering campaigns that may make the political capital required for any future regulatory response too immense for any meaningful government action.

History has shown how powerless our political institutions can be in the face of powerful tech lobbyist groups. But if our government can’t protect us from the data predators of silicon valley, who can? Maybe only we can.

A Global Rallying Cry for Decentralization

Personal data regulations in the United States affect a global consumer market of 4.27 billion internet users, whose personal data falls under the jurisdiction of the 2018 Cloud Act, superseding local and international law, such as the EU’s 2016 General Data Protection Regulation (GDPR).

This has led to an outcry and swift legal action, including the shoring up of privacy protection laws in the European Union and the diversion of massive institutional funding to catalyze homegrown startups, attract foreign investment, and catalyze bleeding-edge research in privacy protection.

A particular focus of internet privacy activism has honed on migrating user data, financial and civic services, and critical cloud infrastructure from centralized servers to alternative peer-to-peer networks. This has had the overall effect of rebuilding the web on a decentralized network of numerous small individual nodes that in time may rival the computational strength, storage capabilities, and stored data value of Big Tech.

Not My Keys, Not my Identity!

The European Commission has led the charge with a surprising announcement to grant every EU citizen a digital ID wallet that serves as a personalized firewall between their most sensitive data and cloud service providers. The wallet operates using end-to-end cryptography to store and transmit data, so that only those that are intended to receive transmitted data are able to read its contents.

The digital ID wallet uses asymmetric cryptography to achieve improved privacy. Each time a new wallet is created, a mathematical solution is computed to generate a public/private pair of keys, or a long string of letters, numbers, or symbols.

The public key can be freely shared with others on the internet and is used to encrypt messages that only the private key holder can read. In this way, data remains protected through obfuscation from harvesting operations. The only way for the data to be compromised is if the user gives up or leaks their private key, or if the encryption is broken through brute force; a nontrivial task for modern systems.

Cryptographic key pairs grant absolute ownership, or self-sovereignty over the data of the user wielding those keys. Individuals can store their data on their own devices and choose what they share with external parties without depending on a centralized service to process their sensitive information.

The proposed digital ID wallet regulations have the additional heavy-hitting consequences that all internet services providers operating in the EU will be required to interact with citizens through this digital wallet. This is only the tip of the iceberg in a slew of technological advancements in cryptography that are beginning to find a place on the digital battlefield for consumer privacy.

Strength in Numbers

Despite these important advancements, asymmetric encryption alone is not sufficient for verifying or enforcing data ownership when multiple parties are involved and exchanging information at very fast rates. Imagine that Bob shares their brain data with Dr. Alice, who promises to provide Bob with a diagnosis on their health. Dr. Alice opens the dataset and realizes they can’t perform the computation-heavy diagnosis on their own so they must send the re-encrypted data to a third party, Sam. The ownership of the data cannot be guaranteed in this case if the data is re-encrypted with a new private key pair unless there is an impartial third party that can guarantee record keeping of the original data provider, which individuals they’ve shared their data with, and the terms of sharing. Today, these legal constructs are built into unwieldy code running on centralized cloud providers that may not always meet definitions of impartiality due to regulatory or geopolitical backdrops.

Blockchain technology has emerged as a promising counterweight to centralization of web infrastructure. A blockchain is a distributed ledger that records transactions between parties and is as secure as the number of unique nodes participating in the network. The greater the participation, the more difficult it is to manipulate the global state of the blockchain or to extract data or information from individual nodes.

Autonomous Solutions to Data Ownership

A smart contract blockchain solves Bob’s data sharing solution by running smart contracts; specialized code optimized to execute agreements between agents on a redundant ledger of permissions and transactions that is exceptionally cost-prohibitive to corrupt or modify.

The results of a smart contract are considered valid when sufficient consensus is achieved on the computation by each node in the network. This decentralizes the burden of proof and reduces the possibility of a violation of the smart contract agreement.

The utility of smart contracts have been beautifully demonstrated by the fledgling decentralized finance ecosystem running on the Ethereum Virtual Machine. 55 billion US dollars are locked in smart contracts that execute automated agreements between businesses and individuals for lending, borrowing, payment, insurance, and other complex financial products.

Recently, Neuroethics groups have advocated for the use of blockchain technologies as a potential protection against infringements of Neurorights (Yuste et al., 2017). The Neurorights Initiative advises centralized storage of brain data should be restricted and instead advocate for decentralised storage and managed using blockchain technologies. The inherent transparency of these technologies enables data to be tracked and audited, and smart-contracts allow for the transparent control of data usage without the need for a centralised authority (Yuste et al., 2017). Competitors to Neuralink, such as inBrain have taken note and committed to researching this technology for their products.

The benefits of smart contracts do not stop at opening up complex financial tools to a global population. The Ocean Protocol is an open source smart contract software recognized by the World Economic Forum that allows users to tokenize their data on a distributed ledger, solidify claims of intellectual property, and control the monetization of their data through programmable royalties and embedded licenses.

Ocean has also introduced Compute-to-Data, a smart-contract infrastructure for executing algorithms on sensitive data without ever revealing the information to the requestor. This way Dr. Alice can use Sam’s sophisticated analytic services without revealing Bob’s identity or their data.

The Ocean Protocol technology stack has encouraged the growth of decentralized autonomous organizations (DAOs) of data providers that pool their data together into vaults controlled by smart contracts that enforce community consensus. A neurotech data union would have the ability to decide under what terms their data is used and enforce consent in real time. For example, Bob can join forces with others to generate highly coveted pristine quality open-access neuroimaging datasets reserved for scientific researchers but that perhaps require payment for access by marketing or for-profit researchers.

The Campaign for Self-Sovereignty: What Should You Do?

The fight for privacy online has entered a new chapter. Niche cult software, such as asymmetric encryption has begun to enter mainstream consumer applications intended to shield users from prying eyes. For the very first time, public-private key pairs coupled with decentralized smart contracts that enforce agreements between data providers and consumers now empower individuals to dictate the terms of how their digital footprint is used.

You can contribute to this fight by creating your own digital identity wallet, joining a data DAO such as DataUnion, or the OceanDAO, joining the Open Science DAO OpSci to learn about self-sovereign research participation, or running a node on a distributed peer-to-peer network.

Vote with your data by using end-to-end encryption whenever possible and cut off the flow of revenue to big tech companies that profit from your digital activity. Failure to coordinate may very well set the path to a dystopian future of mass monocultural thought, where stimulus-reward pairings have replaced the joys of art, creativity, and the pursuit of enlightenment through rational thought and scientific discovery.

There are many unknowns regarding how regulations, technology, and geopolitical interests will unfold — only time will reveal the result of consensus on the direction our digital communities will take in the imminent future.

References

Yuste, R., Goering, S., Bi, G., Carmena, J. M., Carter, A., Fins, J. J., … & Wolpaw, J. (2017). Four ethical priorities for neurotechnologies and AI. Nature News, 551(7679), 159.

Written by Shady El Damaty, Ph.D. & Sarah Hamburg, Ph.D., edited by Hazal and Garrett Flynn, with artwork by Firas Safieddine.

Originally published at https://medium.com on July 27, 2021.

Not My Keys, Not My Thoughts was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article is part of an on-going series on decentralized file storage for science introduced in Rich in Data, Poor in Wisdom: Science Needs a Decentralized Data Commons.

Swarm is a permissionless, censorship-resistant, peer-to-peer storage and communication system that scales automatically and is sustained by economic incentives. However, as of this writing, it has not been fully implemented. Having grown out of the Ethereum project, it is intended to complete the goal of creating an entirely decentralized world computer on which any program could be written or executed and any data could be stored by anyone with an internet connection. If Ethereum is the world computer’s CPU, Swarm is “the world computer’s storage and communication,” though its emphasis is storage (source). Here we review the basics of Swarm, its incentive system, and its access control system.

Swarm Network Basics

The Swarm network consists of nodes which store and transfer chunks, which are files or parts of files. Every node and every chunk has an address. Nodes and chunks with similar addresses are grouped into neighborhoods. Every node in a neighborhood connects to every other node in the neighborhood. Nodes also connect to some nodes outside their neighborhoods, but the further apart two nodes’ addresses are, the less likely they are to be connected.

All nodes in a neighborhood are incentivized (via postage lotteries, discussed below) to store the chunks whose addresses are in the neighborhood. Every chunk is 4 KB or less. Any data upload that’s more than 4 KB is uploaded as multiple chunks linked together in a hash tree, and a manifest entry is created which contains a reference to the hash tree’s root node. Every chunk is either a content-addressed chunk — whose address is a hash of the chunk’s data — or a single-owner chunk. A single-owner chunk is the same as a content-addressed chunk, except it also includes an ID and cryptographic signature attesting to the chunk’s integrity.

Incentive System

Swarm’s incentive system accounts for both storage and bandwidth sharing.

Storage Incentives

Swarm’s storage incentives encourage nodes to store some data for at least as long as payment for that data supports storage. The most important construct for these incentives is the postage stamp (source). A postage stamp is a proof of payment submitted alongside a data upload, and the proof is submitted through Ethereum smart contracts. This payment is used to cover recurring storage costs for the nodes who store the data and the cost of forwarding the data to its neighborhood. By requiring a node to commit before sending a request to the payments associated with the request, Swarm discourages nodes from flooding the storage network with useless data and requests.

Swarm incentivizes long-term storage with postage lotteries, which occur every few blocks. In a postage lottery, storage nodes in a neighborhood offer to store chunks for certain prices, and the nodes who offer the lowest prices win the lottery and are compensated. To win, however, a storage node must also submit successful proofs of storage (which is a simple binary merkle tree proof) for all chunks for which it is responsible. Once a lottery round is complete, all the nodes in the neighborhood are required to store chunks at the lowest price. Postage lotteries, which are funded by postage stamps, encourage storage nodes to store all data in their neighborhood and to offer the lowest prices.

While postage stamps and lotteries encourage good behavior, Swarm has a couple mechanisms to discourage bad behavior. To prevent nodes from simply collecting initial fees and then deleting data, Swarm requires storage nodes to submit a security deposit before selling storage. If the node fails a proof, it loses its security deposit. When the node stops selling storage, its security deposit is returned. Swarm also has a sort of litigation process, allowing for more interactive incentivization. If a storage node fails to serve a chunk to a requestor, the requestor can submit a security deposit and challenge the storage node. The node must prove that either it or another node possesses the chunk. If it fails, it loses its security deposit, but if it refutes the challenge, the challenger loses its security deposit.

Bandwidth Sharing Incentives

On the bandwidth sharing front, nodes must be rewarded for routing valuable data in a timely fashion but punished for malicious behavior, such as spamming. At a high level, Swarm’s bandwidth sharing incentive system consists of five mechanisms.

First, intermediate nodes on a route are rewarded only when the data reaches its destination. A piece of content must travel multiple hops (i.e., across multiple nodes) before it reaches its requestor. By ensuring such nodes only get paid if the network successfully serves the request, Swarm incentivizes intermediate nodes to forward data and requests.

Second, nodes closer to the requestor are paid more for a given chunk than nodes further away, creating a chunk marketplace. When I request a chunk, the request is forwarded by many nodes to a node storing the chunk. The storer node then sells the chunk to a node that is closer to me but still too far to transfer it directly to me. That node then sells the chunk to a node even closer to me. This buying and selling repeats until the chunk reaches me, at which point I pay the node who directly sent it to me. In this way, each intermediate node along a route profits from the difference between the buy-price and sell-price of a requested chunk, so nodes have an incentive to forward chunks.

Third, every request has a time to live (TTL). If the request is serviced within the TTL, the requesting node must pay for the servicing of the chunk. Requesting nodes are thus disincentivized from sending large numbers of requests in attempts to have the requests serviced quickly.

Fourth, nodes can sell any chunk they store. They are thus incentivized to cache popular chunks. This turns Swarm into not just a static storage network, but a distribution network that scales with demand.

Fifth, sanctions are placed on nodes who request non-existent chunks too frequently. This disincentivizes nodes from requesting non-existent chunks.

Swarm Accounting Protocol (SWAP)

The Swarm Accounting Protocol (SWAP) is also important for bandwidth sharing on Swarm, but it is more than a mere incentive. With SWAP, nodes create a bandwidth-based credit system by recording how much bandwidth their peers are consuming. Nodes can cash in their credit for BZZ tokens on Ethereum. If a node uses too much bandwidth without contributing, other nodes can stop transferring chunks to and from the node until it settles its bandwidth debt in BZZ. This credit system allows, for example, some nodes to upload for free by first routing chunks until earning enough credit to pay for the upload. Also, by adding another layer of reputation, SWAP guards the network from potential free-riding nodes.

Access Control and Encryption

Swarm intends to provide a truly peer-to-peer access control system for storage. Currently, with Web2 systems (such as Google and Amazon), access control is ultimately governed by the companies who control the products. For example, with Google Docs, a user cannot stop Google from accessing a document, no matter what privacy settings the user has; additionally, anyone with access to the servers on which the document is stored can access it. Instead of storing data in locations secured by trusted parties, Swarm uses layers of encryption to restrict access. All access-controlled data is encrypted before upload, so the sensitive data is never really stored on Swarm. This redesign of access control requires new definitions for read access and write access. Read access on Swarm is the ability to decrypt a file, and write access is the ability to change the reference in a file’s manifest entry. The next paragraph clarifies these definitions with a discussion of the manifest entry.

Recall that a manifest entry is created for a file if the file is greater than 4 KB. In the case of storing an encrypted file, the manifest entry stores three things: a reference to the encrypted data, a decryption key, and information about how to get an access key (which cannot be distributed through Swarm). Not only is the data itself encrypted, but the reference and decryption key are also encrypted. They are encrypted together and can be decrypted by the access key. Thus, to retrieve and decrypt an encrypted file on Swarm, a node must

1. retrieve the manifest entry,
2. get the access key,
3. use the access key to decrypt both the reference to the file’s root and its decryption key,
4. use the reference to retrieve the encrypted file, and
5. decrypt the file.

The below graph represents the manifest entry, where an ellipse indicates encryption.

Links and Resources to the Swarm Ecosystem

• Swarm website. Available at https://www.ethswarm.org.
• List of Swarm grant recipients. Available at https://www.ethswarm.org/ecosystem.html.
• “Swarm: Storage and Communication Infrastructure for a Self-sovereign Digital Society.” Swarm team. Available at https://www.ethswarm.org/swarm-whitepaper.pdf.

Conclusion

The Swarm team aspires to create “the world computer’s storage and communication,” a peer-to-peer storage and communication protocol that complements Ethereum smart contracts. Swarm’s base layer consists of nodes storing and transferring chunks of data. The incentive system encourages nodes to both store data and share bandwidth, discourages misuse of these resources, and creates a bandwidth-based credit system (SWARM) which allows bandwidth debt to be settled in BZZ tokens on Ethereum. Swarm uses layers of encryption to achieve decentralized access control. Even though Swarm’s feature set is not complete (as of this writing), Swarm has a buzzing ecosystem: developers are creating apps such as decentralized databases, email systems, and meme generators.

Join the Decentralized Open Science Movement

Does the idea of a free, open, internet of science ring a resonant chord with you? Consider joining the Opscientia community to learn, connect, and collaborate with others building a commons for co-discovery.

Articles in This Series

1. Decentralized Content Networks for a Permanent Science Data Commons: IPFS
2. Engineering Incentives for Data Storage as a Commodity: Filecoin
3. A Permanent Web of Linked Data: Arweave
4. Peer-to-Peer Storage without a Blockchain: Storj
5. One of the First Decentralized Cloud Storage Platforms: Sia
6. The World Computer’s Hard Drive: Swarm
7. A Permissionless Data Layer for Science Web Apps
8. OpSci Commons: A Decentralized and Autonomous Knowledge Commons

References

Trón, V. (2020, November 17). Book of Swarm: Storage and Communication Infrastructure for a Self-sovereign Digital Society. Retrieved December 9, 2021, from https://docs.ethswarm.org/the-book-of-swarm.pdf

Swarm. (2021, June 13). Swarm: Storage and Communication Infrastructure for a Self-sovereign Digital Society [White paper]. Retrieved December 9, 2021, from https://www.ethswarm.org/swarm-whitepaper.pdf

Prahalad, B. (2018, January 7). Merkle proofs Explained. Retrieved December 2, 2021, from Merkle proofs Explained.. This article explains how to obtain and… | by Belavadi Prahalad | Crypto-0-nite | Medium

Originally published at https://hack.opsci.io on December 22, 2021.

The World Computer’s Hard Drive: Swarm was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article is part of an on-going series on decentralized file storage for science introduced in Rich in Data, Poor in Wisdom: Science Needs a Decentralized Data Commons.

Sia is a blockchain-based decentralized data storage marketplace created by Skynet Labs, and with a whitepaper published in late 2014, it might be the first of its kind. Its blockchain has a cryptocurrency called Siacoin. The actors in the Sia network are renters (or clients), hosts, and miners. Renters pay hosts to store data, while miners create blocks and verify transactions, proofs of storage, and new blocks. New blocks are validated by Proof of Work. Here we cover transactions and file contracts on Sia, possible issues with Sia’s on-chain incentive system, ways to mitigate these issues, and Sia’s economics.

Transactions and File Contracts

Sia transactions can be used to transfer Siacoin, create file contracts, record storage proofs, and they even have a field for arbitrary data. “A file contract is an agreement between a storage provider and their client” (source). A contract specifies the:

1. agreement’s duration,
2. frequency at which the host must provide storage proofs (i.e., the challenge frequency),
3. address that receives funds if a proof succeeds,
4. address that receives funds if a proof fails,
5. amount of funds that get distributed when a proof succeeds or fails,
6. number of allowed failed proofs, and
7. edit conditions.

The parameters of a file contract can be updated if its edit conditions are met. The challenge frequency also includes challenge windows that specify the time windows during which proofs must be submitted, allowing for the possibility that a challenge or proof might not be routed immediately.

File contracts would be useless without storage proofs. In Sia, the storage proof for a file is simply a Merkle proof, “a [random] segment of the original file and a list of hashes from the file’s Merkle tree.” When a file contract is created, the file is split into same-size pieces and hashed to create a Merkle tree. The contract stores the Merkle root and the size of the file. To verify a storage proof, the file segment and hashes from the proof are hashed together and compared to the Merkle root in the file contract. If they match, the proof succeeds.

Issues with Incentive System-And Their Remedies

The incentive system that emerges from this design fails to fully address a few important cases. The authors of the Sia whitepaper name two such cases: closed window attacks and what we might call data withholding attacks. A closed window attack occurs when miners “exclude storage proofs from blocks.” By excluding proofs from certain hosts, miners can ensure those hosts are penalized for failing to provide proofs within a challenge window. Closed window attacks, however, are only feasible for a miner or group of miners who control a large percentage of the network. Hosts can reduce the likelihood of this attack by only accepting contracts that have large challenge windows.

A more feasible attack is a data withholding attack. On Sia, hosts have no incentive to stay online more than required for providing proofs and, in some cases, no incentive to return client’s data. We can imagine a host that remains offline as much as possible, only coming online to submit storage proofs. Such a host would be paid the full amount of the contract, even if they never returned the renter’s data. There is no mechanism inherent to Sia that incentivizes the return of renter data. If, however, the renter stores multiple copies of the same file with different hosts, the renter can create an incentive for the renters to return their data. The renter can offer a reward to the host who returns the data the quickest, creating a market in which hosts compete with each other to transfer the renter’s data. Prudent renters will store the same file with multiple hosts to avoid data withholding attacks.

Sia’s Economics

Sia’s economics include two noteworthy points: Siacoin is permanently inflationary, and the Sia blockchain includes a second cryptocurrency, called siafunds. 30,000 siacoins are created with every block. Sia developers chose an inflationary — rather than deflationary — design so that miners always have an incentive to mine. Siafunds are the primary mechanism by which Sia’s creator, Skynet Labs, earns revenue. “When a contract is created, 3.9% of the contract fund is removed and distributed to holders of siafunds.” Members of Skynet Labs own the majority of siafunds.

Skynet

Skynet Labs created a platform on top of Sia called Skynet, which simplifies interaction with the Sia network. Skynet simplifies user experience with what it calls portals. A Skynet portal allows users to upload data to the Sia network without thinking about the underlying mechanisms. The portal operator handles the details of establishing file contracts with storage nodes and maintaining enough funds to pay for storage. The most popular portals offer free storage.

Links and Resources to Sia Ecosystem

• Sia website. Available at https://sia.tech.
• Skynet website. Available at https://siasky.net.
• “Sia: Simple Decentralized Storage.” David Vorick and Luke Champine. Available at https://sia.tech/sia.pdf.
• Skynet app store. Available at https://skapp.hns.siasky.net/#/apps/all.

Conclusion

Sia is an intuitively designed decentralized storage marketplace constituted by clients, hosts, and miners who interact on the Sia blockchain. Transactions on the blockchain can include file contracts, in which hosts agree to store data for clients, and storage proofs, in which hosts prove they are storing client data. Sia’s incentive system doesn’t inherently address closed window attacks or data withholding attacks, so actors must take precautions to prevent these. In addition to its primary cryptocurrency, Sia has siafunds which extract revenue from the network for siafund holders. Skynet Labs also built Skynet, a platform which makes using the Sia marketplace easier. Sia and Skynet are worthy competitors in the decentralized storage sector for anyone wanting to build fully decentralized apps.

Join the Decentralized Open Science Movement

Does the idea of a free, open, internet of science ring a resonant chord with you? Consider joining the Opscientia community to learn, connect, and collaborate with others building a commons for co-discovery.

Articles in This Series

1. Decentralized Content Networks for a Permanent Science Data Commons: IPFS
2. Engineering Incentives for Data Storage as a Commodity: Filecoin
3. A Permanent Web of Linked Data: Arweave
4. Peer-to-Peer Storage without a Blockchain: Storj
5. One of the First Decentralized Cloud Storage Platforms: Sia
6. The World Computer’s Hard Drive: Swarm
7. A Permissionless Data Layer for Science Web Apps
8. OpSci Commons: A Decentralized and Autonomous Knowledge Commons

References

Mining on Sia — Sia Support Docs. (n.d.). Sia. Retrieved November 11, 2021, from About mining on Sia — Sia Support Docs

Prahalad, B. (2018, January 7). Merkle proofs Explained. Retrieved December 2, 2021, from Merkle proofs Explained.. This article explains how to obtain and… | by Belavadi Prahalad | Crypto-0-nite | Medium

Skynet Basics — Skynet Guide. (n.d.). Skynet. Retrieved December 11, 2021, from Skynet Basics — Skynet Guide

Vorick, D., & Champine, L. (2014, November 29). Sia: Simple Decentralized Storage. Retrieved December 11, 2021, from https://sia.tech/sia.pdf

Originally published at https://hack.opsci.io on December 19, 2021.

One of the First Decentralized Cloud Storage Platforms: Sia was originally published in OpSci on Medium, where people are continuing the conversation by highlighting and responding to this story.