<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rudra on Medium]]></title>
        <description><![CDATA[Stories by Rudra on Medium]]></description>
        <link>https://medium.com/@rudrapanda?source=rss-b56bada386b6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*BpLV09oct0bwNXiI.jpg</url>
            <title>Stories by Rudra on Medium</title>
            <link>https://medium.com/@rudrapanda?source=rss-b56bada386b6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 15:13:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rudrapanda/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Privacy-Preserving Verifiable Credentials]]></title>
            <link>https://rudrapanda.medium.com/privacy-preserving-verifiable-credentials-109788f12a36?source=rss-b56bada386b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/109788f12a36</guid>
            <category><![CDATA[identity-management]]></category>
            <category><![CDATA[decentralized-identity]]></category>
            <category><![CDATA[verifiable-credentials]]></category>
            <category><![CDATA[self-sovereign-identity]]></category>
            <category><![CDATA[online-security]]></category>
            <dc:creator><![CDATA[Rudra]]></dc:creator>
            <pubDate>Wed, 28 Feb 2024 02:41:03 GMT</pubDate>
            <atom:updated>2024-02-28T02:41:03.622Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I will explain how Verifiable Credentials (VCs) can be used to satisfy certain Cameron’s laws of Identity to provide privacy-preserving security. I explained these laws in my “Self Sovereign Digital Identity: The Idea” article.</p><p>Let me begin with some concepts.</p><h3><strong>Verifiable Credential</strong></h3><p>Examples of credentials are driver’s licenses, passports, movie tickets, educational certificates, etc. In the physical world, a physical form of these credentials is presented to receive the required service. For example, the driver’s license may be presented to an authority to assert the ability of the subject to drive a vehicle, the passport may be presented to immigration authority to assert the ability to travel, the movie ticket may be presented to the person at the gate to asserts admission to the movie theatre and a seat. In all these cases, the physical document is verified to provide the requested service. In the case of a driver’s license and passport, the picture on the document is matched with the presenter of the document to verify the authenticity of the presenter and the document. In the case of a movie ticket, whoever is the holder of the ticket is given admission to the theatre and a seat — the admission is not for a specific person but rather for whoever has the ticket.</p><p>In the digital world, these credentials are presented in digital form. The VCs use cryptography to provide the ability to verify these digital credentials.</p><p>The credentials carry the proofs along with them, and these proofs are used to verify the authenticity of the credentials. The issuer specifies the credentials as attributes and values — also known as claims. The claims are digitally signed by the issuer which is the proof. There are credential metadata that provides information such as the issuer, validity of the credential, etc. All these three together make up the verifiable credential (VC). Figure 1 is a reproduction of the diagram from <a href="https://www.w3.org/TR/vc-data-model-2.0/#credential-subject">Verifiable Credentials Data Model v2.0</a> that depicts the basic components of VC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1UIUNcUCi9lnYiW_tyf-SA.png" /><figcaption>Figure 1: Basic Components of Verifiable Credential</figcaption></figure><p>The VC contains information about the issuer either as a URI or a DID (Decentralised Identity) that can be dereferenced to obtain more information about the issuer, and the most relevant information is the public key to verify the authenticity of the VC. It will be more privacy-preserving if the issuer information is specified as a DID, and the DID and DID documents are available in a VDR (Verifiable Data Registry). The verifier then uses the VDR to retrieve the required information rather than connecting with the issuer directly. This reduces the chances of the verifier colluding with the issuer and aggregating information about the subjects. This also keeps the relationship between the subject and verifier confidential from the issuer.</p><h3>General Lifecycle of Credentials:</h3><p>Credential is for a subject. It’s issued by an issuing authority known as an issuer.</p><p>In the physical world, the issuer provides a physical document that stays in the possession of the holder. Holders are authorized subjects to keep the credentials for themselves or another subject. The parent may keep the passport for their children. In this article, I will use the most common scenario to keep it simple and more focus on features of VCs that provide privacy-preserving security. The most common scenario is subject and holder are the same — interchangeable in this article.</p><p>In the digital world, the issuer provides a digital document that the holder stores on a device. It’s preferred that the credential is stored on a private device that’s in the possession of the holder rather than on a centralized repository. The former provides more security and privacy than the latter. The software on the device that provides this capability for storing credentials in a secure and privacy-preserving way is known as a digital wallet. For this article, the holder means a software agent or wallet that’s running on the device. Holder, Software Agent, and Wallet are interchangeable in this article.</p><p>Whenever the subject needs a service the holder presents the credential to the service provider or an authorized representative, they are known as verifiers. The holder digitally signs the verifiable credential and attaches the signature as proof. There is also presentation metadata. The metadata, verifiable credentials, and proof make up the verifiable presentation (VP). The verifiable presentation can contain multiple VCs issued by different issuers. Figure 2 is a reproduction of a diagram from <a href="https://www.w3.org/TR/vc-data-model-2.0/#credential-subject">Verifiable Credentials Data Model v2.0</a> that depicts the basic components of VP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QP_elatvOC46czcDAf33CA.png" /><figcaption>Figure 2: Basic Components of Verifiable Presentation</figcaption></figure><p>Figure 3 below shows the lifecycle of a digital credential</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Qnfi-ZmJTH0V9l-2dePQw.png" /><figcaption>Figure 3: Digital Credential LIfecycle</figcaption></figure><h3>Trusts</h3><p>In the digital world, one of the ways to establish trust is to exchange cryptographic keys and use those keys in subsequent interactions. Each party in the relationship creates a private public key pair. They store the private key and exchange the public key with the other party in a secure way. Whenever the parties want to exchange information confidentially, they use the public key of the other party to encrypt the data — the one with the private key can only decipher this encrypted data. Whenever one of the parties wants to give confidence to the other that they are the sender of the information, they sign it using their private key. The receiver verifies the signature using the sender’s public key. No one else can produce the same signature except the one in possession of the private key.</p><p>Whenever the parties want to keep their relationship private, they create public private key pair exclusively for that relationship and no one else knows about the public keys outside of that relationship. In decentralized identity, this is accomplished using peer DID and DID Documents. Whenever a party wants to maintain a public identity, they publish their DID and DID Document in a verifiable data registry (VDR) — the DID Document contains the public key.</p><p>Since the focus of this article is the most common scenarios, let us assume the following conditions have been established to create a trust framework without going into the details of how. I may cover the trust framework in detail in one of my future articles.</p><ul><li>The issuer has a public DID in a VDR that can be accessed by the verifier</li><li>Issuer and Subject and their respective Wallets: Peer DID [I have described Peer and Public DIDs in my article “<a href="https://medium.com/@rudrapanda/decentralised-identifiers-dids-888b11875a50">Decentralised Identifiers (DIDs)</a>”]. A private relationship between issuer and subject is more privacy-preserving. This protects from aggregating information about the subject. The software agents manage and exchange digital assets on behalf of the issuer and subject. It’s very much possible that the software agents may change. It becomes easier to replace software agents by keeping them independent of their users. The software agents will have their own DID and DID Documents. For the same reason that the issuer and subject have a private relationship, their software will have a private relationship too.</li><li>Subject and Verifier and their respective Wallets: Peer DID</li><li>Issuer and Verifier: The verifier trusts the issuer. They rely on issuer DID and DID Documents stored in a VDR to verify the credentials issued by them.</li></ul><p>Figure 4 shows the established trust among Issuer, Holder, and Verifier</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eSxDCFeQTS4Q8c35zCvwIA.png" /><figcaption>Figure 4: Trust</figcaption></figure><h3>Privacy-Preserving Verifiable Credential</h3><p>Let us now examine how VC satisfies the following Cameron’s laws of Identity to provide privacy-preserving security.</p><ul><li>User Control and Consent</li><li>Minimal Disclosure for Constrained Use</li><li>Justifiable Parties</li><li>Directed Identity</li></ul><p><strong>User Control and Consent, and Justifiable Parties:</strong></p><p>In a typical digital interaction, the holder initiates the request for a service using the service provider App. The holder indicates that it will use its wallet to authenticate itself. The service provider displays a QR Code that the holder device scans the code — the service provider and holder devices establish a trusted relationship or connection between them.</p><p>The verifier App will request the holder to provide certain credentials to access the service. The wallet alerts the holder about the verifier and credentials being requested. The holder will have the option either to provide consent or decline the request. Only upon receiving the consent of the holder, the wallet will exchange the requested credentials with the verifier.</p><p>The holder has complete awareness of the verifier (Justifiable parties) and control of exchanging credentials. There may be a question — what if the verifier shares these credentials with others without the knowledge of the holder? There are various options to deal with such a situation such as not providing any identity information in the VP rather than providing a pseudonym — essentially it will come down to the content of the VP and signatures that will make it useless if it’s shared with anyone else. I will cover the details of dealing with such situations in one of my future articles.</p><p><strong>Minimal Disclosure for Constrained Use, and Directed Identity</strong>:</p><p>These are important features of VC/VP to provide privacy-preserving security. To understand these features, I will use the use case that I described in my article “<a href="https://medium.com/@rudrapanda/self-sovereign-digital-identity-how-it-works-96ea3f5aa014">Self-Sovereign Digital Identity: How it Works?</a>”.</p><p>Alice is the holder in this example who needs to obtain three different credentials from three different issuers. The typical workflow is</p><ol><li>Alice uses her Wallet to establish a secure connection with the issuer’s Wallet.</li><li>Alice uses her Wallet to request credentials from each of the issuers.</li><li>The issuer issues credentials after completing the required validations.</li><li>Alice’s Wallet stores the credentials.</li></ol><p>The table below shows the minimum required information each of the example credentials contains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HiljX1jharKM_KjixHbL5w.png" /><figcaption>Figure 5: Example Credentials</figcaption></figure><p>The metadata section provides information for the machine to be able to read and interpret the credentials. The claims section provides information about the subject. The proofs section provides information for the machine to be able to validate the authenticity and integrity of the credentials.</p><p>The algorithms used to provide proofs play an important role in satisfying minimal disclosure and directed identity. There are two ways to provide proofs:- a) the proof is embedded in the credential itself b) the proof that wraps the credential. One of the recommended approaches for embedded proof is <a href="https://w3c.github.io/vc-data-integrity/#proofs">Verifiable Credential Data Integrity</a>. I am using this approach for my example. The examples in the table above show the minimum required information to be provided for Data Integrity Proofs.</p><p>One thing to observe in the above table is that the identity of Alice (DID) has not been provided. How is the credential associated with Alice then? One of the options is to provide her DID as the “id” value in the Credential Subject section. However, that will not be privacy-preserving since that will reveal her relationship with the issuer. This can be also used to aggregate information about Alice across different credentials. Hence we need a mechanism (attribute) that’s privacy-preserving and</p><ul><li>Associates the credential to Alice the Holder</li><li>No one else knows about this attribute other than Alice the Holder — hence it is privacy-preserving</li><li>It can be used to link all credentials issued to her — we need this ability to take information from multiple credentials and create one presentation for the verifier. We need to prove that these multiple credentials are issued to the same holder.</li></ul><p>This attribute is known as “Link Secret” — it’s only known to Alice and hence is a secret, it links all credentials issued to her and hence is a link. How does it get associated with the credentials while maintaining its secrecy?</p><p>The idea of minimal disclosure is only to disclose the attributes of a credential that are truly required for that interaction. The “Link Secret” is considered an attribute of a credential that’s never revealed. The credential is created by the issuer. How will the issuer make the “Link Secret” attribute part of the credential without the knowledge of the secret itself? This problem is solved by using “<a href="https://asecuritysite.com/encryption/ped">Pederson Commitment</a>” which is a cryptographic technique. Pedersen commitments are cryptographic algorithms that allow a holder to commit to a certain value without revealing it or being able to change it. What makes it possible for the holder to commit to the value of “Link Secret” without revealing or being able to change is the Camenisch-Lysyanskaya Signature scheme — also commonly referred to as the CL Signature Scheme.</p><p>In the CL Signature Scheme, the issuer generates a public key which will have variables equal to the number of attributes in the credential plus two — the additional two variables are required for the underlying math. In our example passport credential, the issuer needs to create a public key with twelve variables — nine for the nine claims, one for “Link Secret”, and an additional two for the math. The algorithm works like this</p><ul><li>The holder creates an initial commitment for the nine attributes using the nine corresponding variables from the public key and shares it with the issuer. The issuer can validate the commitment since they are aware of the nine claims.</li><li>Then the Holder can create a commitment for the “Link Secret” value using the tenth corresponding public key variable. The Holder can add this commitment to the earlier commitment, and share the new overall commitment with the issuer. The math helps the issuer to validate the new commitment without having to know the value of the “Link Secret”.</li><li>The issuer signs the new commitment assuring the authenticity and integrity of the credential.</li></ul><p>With the addition of “Link Secret” to the credential, we can keep the DID relationship between the holder and the issuer private which is one of the requirements of the directed identity.</p><p>Like the Holder was able to keep the “Link Secret” secret by creating a commitment for it, they would be able to keep any of the claims secret by creating a commitment for it and not revealing the actual value. Then issuer validates the overall commitment and signs the credential that includes both the revealed and committed values. The Holder includes this credential in a verifiable presentation, signs it with their private key, and shares it with the verifier. The verifier verifies the VP signature to verify that it’s coming from the intended holder and that the integrity of the VP was not compromised. It verifies the credential signature to verify that it was issued by the trusted issuer and that the integrity of the credential was not compromised. The verifier can validate the credential even though some of the claims are hidden because of the underlying math used in the CL Signature scheme.</p><p>So far we have seen that by using the CL Signature scheme and commitments, we can satisfy minimum disclosure and directed identity requirements. One of the attributes will remain the same every time the credential is generated — that’s the signature. Suppose for different iterations of the credential different claims are being revealed. The signature, which remains the same, can be used to correlate all these revealed claims. One of the techniques used to address this problem is called <a href="https://en.wikipedia.org/wiki/Blind_signature">Blinded Signatures</a>. Using this algorithm, the Holder will be able to demonstrate to the verifier that they know the signature without revealing it. This itself will be sufficient for the verifier to verify the authenticity and integrity of the entire presentation while keeping the VP unique every time it’s generated, hence preventing any chance of correlation.</p><p>Now let us apply the above concepts to our use case where the DPS (Depart of Public Safety), the Verifier, wants to verify certain claims about Alice, the Holder. The DPS Wallet knows that the required claims can be obtained from the following credentials</p><ul><li>Social Security Card issued by Social Security Administrator whose DID is “did:gov:ssa”</li><li>A passport issued by the Department of State whose DID is “did:gov:dos”</li><li>A bank statement issued by BankA whose DID is “did:vdrA:bank”</li></ul><p>The DPS Wallet retrieves the credential definition for Social Security Card and Passport from VDR “gov”, and Bank Statement from VDR “vdrA”. Once again, these are my made-up VDRs and DIDs for my example use case. Using these credential definitions, the DPS Wallet creates a presentation request something like this</p><ul><li>Get “Full Name” from Social Security Card</li><li>Get “Full Name” from Passport</li><li>Get “Full Name”, “Address Line1”, “Address Line2”, “State”, and “ZIP” from Bank Statement</li><li>Check that the Holder, Alice, is older than 16 by using the rule: “DOB” from Passport + 16 &gt; current date</li><li>Check that the passport is valid by using the rule: “Date of Expiration” from passport &gt; current date</li><li>Check that the bank statement is not older than sixty days by using the rule: “Start Date” from Bank Statement &gt; current date — 60 days</li></ul><p>Note that DPS Wallet didn’t request values in DOB, Date of Expiration, or Start Date, but it requested results from the rules using those values — this is an example of minimal disclosure for constrained use.</p><p>The DPS Wallet sends this presentation request to Alice’s Wallet. Alice’s Wallet creates commitments (hides) all attributes except for “Full Name” on Social Security Card, Passport, and Bank Statement, and “Address Line1”, “Address Line2”, “State”, and “ZIP” on Bank Statement. It runs the rules from the presentation request. The results of the rules are verified using algorithms like <a href="https://crypto.stanford.edu/bulletproofs/">Bullet Proofs</a> or SNARKs. An aggregate proof is provided to prove that the same “Linked Secret” was used in all credentials that are part of the verifiable presentation. The Wallet uses the “Blinded Signature” scheme to prove that it knows the signatures without revealing them. The Wallet combines all these into a Verifiable Presentation and shares it with DPS, the verifier. DPS verifies the authenticity and integrity using the cryptographic signatures. It may then execute some business rules such as the same name across all credentials, the address fields matching with the address Alice provided on the original request, etc. After successful validation, it issues a RealID.</p><h3>Conclusion:</h3><p>The focus of this article was to show how Verifiable Credentials and Presentations support privacy-preserving security by meeting Cameron’s laws of Identity. We saw the use of Wallets by the Holder, Issuer, and Verifier to exchange requests, VCs, and VPs. The protocol that supports the secure exchange of these artifacts is known as DIDComm. I also touched on various concepts like CL Signatures, Pederson Commitment, Blinded Signatures, Bullet Proofs, SNARK, etc. to show their use in creating VCs and VPs to support Minimal Disclosure for Constrained Use and Directed Identity. I plan to cover the following topics in my future articles.</p><ol><li>DIDComm Protocol</li><li>Implementation examples of concepts: CL Signatures, Pederson Commitment, Blinded Signatures, Bullet Proofs, SNARK</li><li>Trust Framework</li></ol><h3>References:</h3><p>[1] <a href="https://www.w3.org/TR/vc-data-model-2.0/#credential-subject">Verifiable Credentials Data Model v2.0</a></p><p>[2] <a href="https://w3c.github.io/vc-data-integrity/#proofs">Verifiable Credential Data Integrity</a></p><p>[3] <a href="https://asecuritysite.com/encryption/ped">Pederson Commitment</a></p><p>[4] <a href="https://en.wikipedia.org/wiki/Blind_signature">Blinded Signatures</a></p><p>[5] <a href="https://crypto.stanford.edu/bulletproofs/">Bullet Proofs</a></p><p>[6] Verifiable Credentials Flavors Explained by Kaliya Young</p><p>[7] Toward a Post-Quantum Zero-Knowledge Verifiable Credential System for Self-Sovereign Identity by Dutto, Simone, et.al</p><p>[8] Blockchain-based Verifiable Credential Sharing with Selective Disclosure by Mukta, Rahma, et.al</p><p>[9] Blockchain-Based Decentralized Verifiable Credentials: Leveraging Smart Contracts for Privacy-Preserving Authentication Mechanisms to Enhance Data Security in Scientific Data Access by Faruk, Md Jobair Hossain, et.al</p><p>[10] Information privacy?! by Camenisch, Jan</p><p>[11] Designated Attribute Proofs with the Camenisch–Lysyanskaya Signature by Papagiannopoulos, Kostas, et.al</p><p>[12] A signature Scheme with Efficient Protocols by Camenisch, Jan Lysyanskaya, Anna</p><p>[13] <a href="https://techcommunity.microsoft.com/t5/security-compliance-and-identity/decentralized-identity-verifiable-credentials-deep-dive/ba-p/3690641">Decentralized Identity: Verifiable Credentials Deep Dive</a> by Dingle, Pamela</p><p>[14] An Efficient System for Non-transferable Anonymous Credentials with Optional Anonymity Revocation by Camenisch, Jan Lysyanskaya, Anna</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=109788f12a36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decentralised Identifiers (DIDs)]]></title>
            <link>https://rudrapanda.medium.com/decentralised-identifiers-dids-888b11875a50?source=rss-b56bada386b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/888b11875a50</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[decentralized-identity]]></category>
            <category><![CDATA[digital-identity]]></category>
            <category><![CDATA[identity-management]]></category>
            <dc:creator><![CDATA[Rudra]]></dc:creator>
            <pubDate>Sat, 09 Dec 2023 14:29:47 GMT</pubDate>
            <atom:updated>2023-12-09T14:29:47.169Z</atom:updated>
            <content:encoded><![CDATA[<p>In my last <a href="https://medium.com/@rudrapanda/self-sovereign-digital-identity-how-it-works-96ea3f5aa014">article</a> “Self Sovereign Digital Identity: How it works?”, I had shown the use of various concepts like DIDs, DID Documents, Verifiable Credentials (VCs), etc. to provide privacy preserving security. In this article, I am diving deeper into the DID and DID Documents.</p><p>The decentralised identifiers are <strong>globally unique identifiers</strong>. The subject(entity, user, holder) chooses <strong>a system (software) that they trust</strong> to generate this unique identifier. The software may be running on the devices that they possess or provided by a provider as service. Such providers are referred to as “method”. The software running on the device is referred to as agent. The format of these identifiers has three parts — it always starts with “did”, the first part, which indicates that it’s a decentralised identifier (DID). The remaining two parts depend on how they get generated.</p><ul><li>When generated using the agent — “<em>did</em><strong><em>:</em></strong><em>peer</em><strong><em>:</em></strong><em>unique-identifier</em>”: The second part is always “peer” indicating that it’s generated using the agent running on user device. This is typically used for private and one-to-one relationship between two entities. The unique identifier is a globally unique identifier generated by the agent. The agent implements an algorithm to be able to generate such unique identifiers. The details of the algorithm can be found in the “<a href="https://identity.foundation/peer-did-method-spec/">Peer DID Method Specification</a>”</li><li>When generated using a provider — “<em>did</em><strong><em>:</em></strong><em>did-method</em><strong><em>:</em></strong><em>method-specific-unique-identifier</em>”: The “did-method” is usually a service provider that maintains a VDR (Verifiable Data Registry) and implements operations to create and maintain DIDs in the VDR. This is typically used to create public identifiers or when a user wants to share the identifier with more than one entity. The “method-specific-unique-identifier” is unique in the context of he did-method.</li></ul><p>Often, there is a DID Document associated with the DID. The DID Document contains information that can be used to interact with the entity in a private and secured way. It doesn’t or should never contain any personal data, either directly to indirectly like through references.</p><p>A typical digital interaction of a user involves Authentication and Service Access.</p><p>The DID document provides option to specify the cryptographic public key that can be used to authenticate the user. The authentication typically uses a challenge response protocol. The figure1 below shows the use of keys from the DID Document and the challenge response protocol to establish mutual authentication between subject (user, entity, holder) and verifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*ciUsQLeC6sKuE7miFgmvEg.png" /><figcaption>Figure1 DID Authentication</figcaption></figure><p>In the above example, the DIDs for both subject (Alice) and verifier (DPS) were generated using a provider (method). In case the DIDs were generated using agents, the flows would be same except the source of the DID Documents. In that case, the DID Documents could have been already exchanged when the peer relationship was established or could be exchanged while requesting for the service, and stored on their devices.</p><p>This is a basic example of challenge response protocol. There are many variations which can be more complex that provide stronger security. However, all of these variations depend on the public keys specified in the DID Document.</p><p>Let’s suppose DPS provided “Login History” service, and Alice wanted to access it. This service was protected, and could be accessed by authorised users sonly. Let’s suppose that she was provided with an access token after successful authentication. She would need to pass this token to DPS to be able to access “Login History”. The Figure2 below details the Service Access flows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*WaxJLzefBXPjtYrphAjIbw.png" /><figcaption>Figure2 DID Service Access</figcaption></figure><p>Both of these scenarios, Authentication and Service Access, used cryptographic keys to verify the authenticity of the user. The DID Document specification terms the use of cryptographic keys for the purpose of authentication and authorisation as <strong>verification methods</strong>. It recommends the use of different keys (verification methods) for different purposes. These different purposes are termed as <strong>verification relationships</strong>. The specification has defined the following verification relationships.</p><ul><li><strong>Authentication</strong>: It is used to specify how the <a href="https://www.w3.org/TR/did-core/#dfn-did-subjects">DID subject</a> is expected to be <a href="https://www.w3.org/TR/did-core/#dfn-authenticated">authenticated</a>, for purposes such as logging into a website or engaging in any sort of challenge-response protocol. [I covered this in this article.]</li><li><strong>Assertion</strong>: It is used to specify how the <a href="https://www.w3.org/TR/did-core/#dfn-did-subjects">DID subject</a> is expected to express claims, such as for the purposes of issuing a Verifiable Credential. [I will cover this in detail in my next article.]</li><li><strong>Key Agreement</strong>: It is used to specify how an entity can generate encryption material in order to transmit confidential information intended for the <a href="https://www.w3.org/TR/did-core/#dfn-did-subjects">DID subject</a>, such as for the purposes of establishing a secure communication channel with the recipient. [I will cover this in detail in one of my up coming articles.]</li><li><strong>Capability Invocation</strong>: It is used to specify a <a href="https://www.w3.org/TR/did-core/#dfn-verification-method">verification method</a> that might be used by the <a href="https://www.w3.org/TR/did-core/#dfn-did-subjects">DID subject</a> to invoke a cryptographic capability, such as the authorization to update the <a href="https://www.w3.org/TR/did-core/#dfn-did-documents">DID Document</a>. [I covered this in this article.]</li><li><strong>Capability Delegation</strong>: It is used to specify a mechanism that might be used by the <a href="https://www.w3.org/TR/did-core/#dfn-did-subjects">DID subject</a> to delegate a cryptographic capability to another party, such as delegating the authority to access a specific HTTP API to a subordinate.</li></ul><p>The specification provides security recommendations to preserve privacy while securing digital interactions. Here are my takeaways from the document.</p><ul><li>Like stated above, the DID Document should not contain any personal data. Rather the specification provides an option to specify service endpoints that are under the control of the user, and can be used to retrieve additional information or interaction options with the subject. The other option for sharing personal data is using Verifiable Credential (VC). [I will cover VC in my next article. ]</li><li>Like any type of globally unambiguous identifier, <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DIDs</a> might be used for correlation. <a href="https://www.w3.org/TR/did-core/#dfn-did-controllers">DID controllers</a> can mitigate this privacy risk by using pairwise <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DIDs</a> (peer DID) that are unique to each relationship; in effect, each <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DID</a> acts as a pseudonym. The anti-correlation protections of pairwise <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DIDs</a> are easily defeated if the data in the corresponding <a href="https://www.w3.org/TR/did-core/#dfn-did-documents">DID documents</a> can be correlated. For example, using identical <a href="https://www.w3.org/TR/did-core/#dfn-verification-method">verification methods</a> or bespoke <a href="https://www.w3.org/TR/did-core/#dfn-service-endpoints">service endpoints</a> in multiple <a href="https://www.w3.org/TR/did-core/#dfn-did-documents">DID documents</a> can provide as much correlation information as using the same <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DID</a>. Therefore, the <a href="https://www.w3.org/TR/did-core/#dfn-did-documents">DID document</a> for a pairwise <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DID</a> also needs to use pairwise unique information, such as ensuring that <a href="https://www.w3.org/TR/did-core/#dfn-verification-method">verification methods</a> are unique to the pairwise relationship.</li><li>Following the concept of herd privacy, a better strategy for endpoint privacy might be to share an endpoint among a large number of <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DIDs</a> controlled by many different subjects. It is the goal of herd privacy to ensure that the nature of specific <a href="https://www.w3.org/TR/did-core/#dfn-did-subjects">DID subjects</a> is obscured by the population of the whole. To maximize herd privacy, implementers need to rely on one — and only one — service endpoint, with that endpoint providing a proxy or mediator service that the controller is willing to depend on, to protect such associations and to blind requests to the ultimate service.</li><li>External links are to be avoided if they cannot be integrity protected and the <a href="https://www.w3.org/TR/did-core/#dfn-did-documents">DID document</a>’s integrity is dependent on the external link.</li></ul><p>In essence, <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DIDs</a> are an enhanced form of URN (Uniform Resource Name) that provide a cryptographically secure, location-independent identifier for a digital resource, while also providing metadata that enables retrieval. Due to the indirection between the <a href="https://www.w3.org/TR/did-core/#dfn-did-documents">DID document</a> and the <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DID</a> itself, the <a href="https://www.w3.org/TR/did-core/#dfn-did-controllers">DID controller</a> can adjust the actual location of the resource — or even provide the resource directly — without adjusting the <a href="https://www.w3.org/TR/did-core/#dfn-decentralized-identifiers">DID</a>. The use of cryptographic keys allows to preserve privacy while providing security. With advancement of technology, creation and maintenance of cryptographic keys are not major challenges anymore. [This reminds me film photography vs digital photography. In the former case we had to be very careful about taking pictures because of limited resources. Whereas in the later case that constraint is not there almost.]</p><p>In summary, the essential properties of DID Document that facilitate secured interaction with the DID subject while preserving privacy are</p><ul><li>Id: This is the subject identifier (DID)</li><li>verificationMethod</li><li>authentication</li><li>assertionMethod</li><li>keyAgreement</li><li>capabilityInvocation</li><li>capabilityDelegation</li><li>service</li></ul><p>The other properties of the DID Document are: alsoKnownAs, and controller that provide more extensibility and flexibility for the use of DIDs and DID Documents. The specification itself goes into more implementation and technical details.</p><p><strong>References</strong>:</p><p>[1] <a href="https://www.w3.org/TR/did-core/#methods">https://www.w3.org/TR/did-core/#methods</a></p><p>[2] <a href="https://ssimeetup.org/introduction-did-auth-markus-sabadello-webinar-10/">https://ssimeetup.org/introduction-did-auth-markus-sabadello-webinar-10/</a></p><p>[3] <a href="https://github.com/WebOfTrustInfo/rwot6-santabarbara/blob/master/final-documents/did-auth.md">https://github.com/WebOfTrustInfo/rwot6-santabarbara/blob/master/final-documents/did-auth.md</a></p><p>[4] <a href="https://rsksmart.github.io/rif-identity-docs/ssi/specs/did-auth.html">https://rsksmart.github.io/rif-identity-docs/ssi/specs/did-auth.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=888b11875a50" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self Sovereign Digital Identity: How it works?]]></title>
            <link>https://rudrapanda.medium.com/self-sovereign-digital-identity-how-it-works-96ea3f5aa014?source=rss-b56bada386b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/96ea3f5aa014</guid>
            <category><![CDATA[identity-management]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[digital-identity]]></category>
            <category><![CDATA[decentalized-identity]]></category>
            <category><![CDATA[self-sovereign-identity]]></category>
            <dc:creator><![CDATA[Rudra]]></dc:creator>
            <pubDate>Sat, 04 Nov 2023 16:58:11 GMT</pubDate>
            <atom:updated>2023-11-04T16:58:11.010Z</atom:updated>
            <content:encoded><![CDATA[<p>In my last <a href="https://medium.com/@rudrapanda/self-sovereign-digital-identity-the-idea-e4dc71805e57">article</a>, I described the idea of Self Sovereign Digital Identity, and how it will provide both security and privacy that will be in the control of the subject (user, entity) itself. In this article, I will demonstrate how it works. I will use a simple, but closer to real life scenario to show various participants, and use of self sovereign digital identities for obtaining a Real ID¹.</p><p>The participants in my example scenario are</p><ul><li>Alice, who is interested in getting a Real ID.</li><li>Social Security Administrator (SSA), who issues Social Security Card.</li><li>Department of State (DoS), who issues Passport.</li><li>A Bank, who issues Statements for the account that Alice has with them.</li><li>Department of Public Safety (DPS) who will issue Real ID.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*D5lPSlCtmFHpRYMmUhKikQ.png" /></figure><p>In our example, Alice requests for a Real ID from DPS. Let’s suppose that DPS needs to verify the following information about Alice before issuing a Real ID to her.</p><ul><li>16 years or older</li><li>Citizenship</li><li>Name and Address submitted by Alice</li></ul><p>DPS suggests that Alice can use the following form of credentials to meet the above verification requirements.</p><ul><li>Social Security Card for Name verification</li><li>Passport for Name, Age, and Citizenship verification</li><li>Bank statement for Address verification</li></ul><p>Alice needs to establish digital relationship with SSA, DoS, and Bank so that she can receive the suggested credentials from them. Let’s suppose that SSA, DoS, and Bank had created public <a href="https://www.w3.org/TR/did-core/#did-documents">DIDs</a> (Decentralised Identifiers) and stored those in some publicly accessible VDRs (Verifiable Data Registry).</p><p>The figure 2 below demonstrates the creation of public DIDs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3pC6j9t_LLCwDhS_pYAApg.png" /><figcaption>Fig 2: Public DIDs</figcaption></figure><p>Few things to note in the above diagram</p><ul><li>Alice and DPS do not have public DIDs - just to demonstrate that not all participants are required to have public DIDs.</li><li>The VDR represents the network of all VDRs. This is to demonstrate that the public DIDs do not need to be created and stored in the same VDR.</li><li>The issuers of credentials such as SSA, DoS, and Bank also store the definition of credentials in the VDR. The consumer of the credentials use the definitions to interpret the credential. These credentials are called as <a href="https://www.w3.org/TR/vc-data-model/">Verifiable Credentials</a> (VCs).</li><li>The DIDs such as did:gov:ssa, did:gov:dos, and did:vdrA:bank follow a specific naming convention. I will go over this convention in one of my future articles. For the purpose of this article, these are just made up DIDs just to demonstrate the concept.</li></ul><p>The figure 3 below demonstrates the process that Alice follows to establish a Peer DID relationship with the Bank, and obtain Statement as a VC. Though not shown in the diagram, similar steps Alice follows to obtain SS card, and Passport from SSA and DoS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*panCx9uqwnnLZeUaxrefxw.png" /><figcaption>Figure 3: Issuance of VC</figcaption></figure><p><strong>Create Link Secret</strong>: In this article, I will describe what’s it, not how it’s used. The link secret is a large random string that Alice generates. It’s used to prove that the VC was issued to the subject, in this case Alice, without having Alice to reveal her peer DID, which is supposed to be private, to the verifier of the VC. Alice never reveals the link secret to the verifier, rather provides proof that she knows about it — the concept of Zero Knowledge Proof (ZKP). To make it distinct for each issuer, she adds a nonce to the secret link, and wraps it in a cryptographic commitment. This cryptographic commitment of link secret and nonce is termed as Blinded Identifier.</p><p><strong>Create Peer DID Relationship</strong>: Alice generates a unique public &amp; private key pair that will be used to establish a relationship with the Bank. She creates a DID, say did:peer:AliceBank, and associates a <a href="https://www.w3.org/TR/did-core/#did-documents">DID Document</a> with this DID. The DID Document will contain the public key that Alice had generated for this relationship.</p><p>Alice finds the public DID of the bank (did:vdrA:Bank) from wherever the Bank has published it — may be Bank portal, or an API etc. Alice finds the corresponding DID Document from the VDR, and then retrieves the Bank’s public key from the document.</p><p>Alice signs her DID and DID Document using her private key. She encrypts all of these using Bank’s public key, and then shares the encrypted DID and DID Document with the Bank.</p><p>Bank is the only entity that can decrypt the message using its private key. It will retrieve Alice’s DID and DID Document from the message. It will obtain Alice’s public key from the DID Document. It will use Alice’s public key to verify the signature and hence the authenticity of the message.</p><p>Prior to sharing DID and DID Document, Alice may use challenge response to authenticate the bank — it encrypts a challenge response message using Bank’s public key, and then Bank decrypts the message using its private key. The Bank then sends back the response that’s signed using its private key. Alice verifies the signature using Bank’s public key, and the content of the message. This assures Alice the authenticity of the Bank.</p><p>The peer DID relationship is a mutually authenticated and uniquely identifiable relationship between two parties.</p><p><strong>Issue Verifiable Credential</strong>: Alice creates a Blinded Identifier, SB, and requests Bank to issue her a recent Statement that’s bound to SB. The Bank issues the requested Statement in the form of a VC. The exchange of all these communications between Alice and Bank is performed using <a href="https://identity.foundation/didcomm-messaging/spec/v2.0/">DIDComm Messaging</a> Protocol. This protocol assures the confidentiality, integrity, and authenticity of the communication. I will go over this protocol in one of my future articles.</p><p>The VCs that Alice receives for SS, and Passport will be bound to blinded identifiers SS and SU respectively.</p><p>The figure 4 below shows the final step in which Alice provides the required information to verify her eligibility and receive a Real ID from DPS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qnpx36OthW6sBcTEeBMLBQ.png" /><figcaption>Figure 4: Verification of VC</figcaption></figure><p>Alice first establishes a peer DID relationship with DPS. She confirms to DPS that she has the following VCs from different issuers to establish her eligibility for a Real ID.</p><ul><li>SS from did:gov:ssa</li><li>Passport from did:gov:dos</li><li>Statement from did:vdrA:bank</li></ul><p>The DIDs mentioned above are all public DIDs. The second part of these DIDs tell DPS where it can get required information to verify the VCs. It retrieves DID Document and definition for SS from VDR “gov”, for Passport from VDR “gov”, and for Statement from VDR “vdrA”. It retrieves the public keys of these issuers from their respective DID Documents. It uses the public key to validate the signature and hence the authenticity of the VCs. It uses the definition of VCs to understand what information and how to use the information from the VCs.</p><p>Let’s suppose that DPS relies on SS to verify the name that Alice has provided. It asks Alice to retrieve and provide the following information from the VC: name, issuer signature, and the blinded ID that the VC is bound to. Few things to note here</p><ul><li>Alice is sharing only “name” from SS, not the number</li><li>Alice is not sharing her peer DID that she had established with SSA. That’s private and remains unique and known between her and SSA.</li><li>Alice is just sharing the blind ID that the VC is bound to so that DPS is able to verify that the VC was issued to her. Sharing the blind ID doesn’t reveal the link secret, it just proves that she has the knowledge of it.</li><li>DPS uses the issuer public key to validate the signature on the VC to establish the authenticity of the VC.</li></ul><p>Similarly DPS will request Alice to use Passport, and provide the following information only, nothing else.</p><ul><li>Yes or no to the question if she is older than 16. She doesn’t share the date of birth from the Passport.</li><li>Yes or no to the question if she is a valid citizen. She doesn’t share issue or expiry date, issuing authority etc.</li><li>Name that DPS verifies against the name retrieved from SS.</li></ul><p>DPS will request Alice to use Statement, and provide address information only and nothing else.</p><p>The idea of presenting only need to know information from multiple VCs, and not the entire VCs is termed as <a href="https://www.w3.org/TR/vc-data-model/#presentations">Verifiable Presentation</a>. I will go over this concept in detail in one of my future articles.</p><p>Upon validating the eligibility, DPS issues Real ID to Alice. This Real ID can be in the form of a VC or physical one.</p><p>I covered the idea of Self Sovereign Digital Identity, and how it will provide both security and privacy that will be in the control of the subject (user, entity) itself in my first article. In this article, I showed how all it works. I will dive deep into concepts that make it all work in my future articles.</p><ul><li>Decentralised Identifiers (DIDs)</li><li>DID Document</li><li>Verifiable Credentials (VCs)</li><li>Verifiable Presentations</li><li>DIDComm Messaging Protocol</li></ul><p>References:</p><p>[1]: <a href="https://www.dhs.gov/real-id">https://www.dhs.gov/real-id</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96ea3f5aa014" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self Sovereign Digital Identity: The Idea]]></title>
            <link>https://rudrapanda.medium.com/self-sovereign-digital-identity-the-idea-e4dc71805e57?source=rss-b56bada386b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4dc71805e57</guid>
            <category><![CDATA[identity]]></category>
            <category><![CDATA[self-sovereign-identity]]></category>
            <category><![CDATA[decentalized-identity]]></category>
            <dc:creator><![CDATA[Rudra]]></dc:creator>
            <pubDate>Sat, 07 Oct 2023 19:13:59 GMT</pubDate>
            <atom:updated>2023-10-07T19:24:27.802Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*8chguJrsneNdc-ii.jpg" /></figure><p>The promise of self sovereign digital identity is to provide both security and privacy that will be in the control of the subject (user, entity) itself.</p><p>It will achieve this objective by meeting the requirements of Cameron’s seven Laws of Identity as explained below.</p><ol><li><strong>User Control and Consent</strong>: <em>Technical identity systems must only reveal information identifying a user with the user’s consent.</em></li></ol><blockquote>A system meeting this requirement would provide autonomy to the subject to choose who to share their identifying (a.k.a Personal Identifying Information) information with. This would put the control of privacy in the hand of the subject.</blockquote><p>2. <strong>Minimal Disclosure for Constrained Use</strong>: <em>The solution that discloses the least amount of identifying information and best limits its use is the most stable long-term solution.</em></p><blockquote>A system meeting this requirement would provide both improved security and privacy.</blockquote><blockquote>Such a s system will acquire minimum needed information (“need to know”) and would retain it for the minimum needed period (“need to retain”). In case such a system is breached, only minimal information will be exposed. Because of acquiring and retaining minimum needed information such a system also reduces the risk of aggregating identifying information.</blockquote><blockquote>A self sovereign system would let the subject to have autonomy to agree or disagree to provide the identifying information being requested. The subject will be aware of the identifying information being shared. Because of sharing minimal information there is an increased possibility of providing anonymity to the subject.</blockquote><p>3. <strong>Justifiable Parties</strong>: <em>Digital identity systems must be designed so the disclosure of identifying information is limited to parties having a necessary and justifiable place in a given identity relationship.</em></p><blockquote>A system meeting this requirement would make the subject aware of who (a.k.a relying party) they are sharing their identity information with. They would be made aware of the use of such identifying information. They would have a say (autonomy and consent) if they would want to share such information.</blockquote><blockquote>The subject would have autonomy to establish unique identity relationship with each relying party rather sharing a common identity across all parties they interact with. This would improve security and reduce the risk of aggregating information as there is no shared (common or universal) Identity.</blockquote><p>4. <strong>Directed Identity</strong>: <em>A universal identity system must support both “omni-directional” identifiers for use by public entities and “unidirectional” identifiers for use by private entities, thus facilitating discovery while preventing unnecessary release of correlation handles.</em></p><blockquote>A system meeting this requirement would provide autonomy to the subject to use either a public or private identity.</blockquote><blockquote>The public identity is invariant and well know to others, for example name or email address. Such an identity may be used while interacting in social or professional communities.</blockquote><blockquote>The private identity is unique for that specific interaction. For example, while browsing a site the subject doesn’t need to share real data, but need to provide an identity that the site can trust. The private identifiers provide anonymity to the subject.</blockquote><p>5. <strong>Pluralism of Operations and Technologies</strong>: <em>A universal identity system must channel and enable the inter-working of multiple identity technologies run by multiple identity providers.</em></p><blockquote>A subject will have different personas during different interactions, such as, citizen while interacting with government, employee while interacting with an employer, student while interacting with an university, customer while interacting with a service provider etc. All these interactions require different identity information.</blockquote><blockquote>A system meeting this requirement would provide the ability for the subject to have the ability to have identities from multiple identity providers that can interoperate.</blockquote><blockquote>A system meeting this requirement would enable to meet the other laws of identity.</blockquote><p>6. <strong>Human Integration</strong>: <em>The universal identity metasystem must define the human user to be a component of the distributed system integrated through unambiguous human-machine communication mechanisms offering protection against identity attacks.</em></p><blockquote>A system meeting this requirement would be cognizant that the human is an integral part of the system. The system would be aware that the human being is vulnerable to revealing identity information through social engineering attacks, the human being could attack the system to steal identity information. Such a system should protect from these potential threats.</blockquote><p>7. <strong>Consistent Experience Across Contexts</strong>: <em>The unifying identity metasystem must guarantee its users a simple, consistent experience while enabling separation of contexts through multiple operators and technologies.</em></p><blockquote>A system meeting this requirement brings all the laws together and puts the subject at the heart and centre of it. Such a system should make it easy (user friendly) for the subject to know their relying parties, the information that they are sharing, the consent that they are providing, their ability to revoke their consent, etc. This interaction should be consistent irrespective of what identity provider that they are using.</blockquote><p>The idea of self sovereign digital identity can be achieved by creating an identity metasystem (analogous to internet that’s a metasystem of networks) that has the following characteristics.</p><ul><li><strong>Decentralised or polycentric</strong></li></ul><blockquote>There is no single identity authority or administrator. There will be multiple identity providers including the subject having the ability to self certify. This satisfies Cameron’s law for Pluralism of Operations and Technologies. This also gives the choice to the subject for choosing their identity provider.</blockquote><ul><li><strong>Polymorphic</strong></li></ul><blockquote>There is no single definition of identity. Rather the system will support subject’s different personas and interactions. This satisfied Cameron’s law for Pluralism of Operations and Technologies, Minimal Disclosure for Constrained Use, and Directed Identity.</blockquote><ul><li><strong>Heterarchical</strong></li></ul><blockquote>The metasystem provides means for subjects (people, organisations, and things) to have relationships with each other as peers without any central authority or administrators. This satisfies Cameron’s laws for User Control and Consent, Justifiable Parties, and Directed Identity.</blockquote><ul><li><strong>Secured</strong></li></ul><blockquote>The metasystem provides means for subjects to have relationships that are mutually authenticated, secured, and as private as possible. It enables the parties in the metasystem to use messaging to exchange polymorphic identities that assures confidentiality, integrity, and authenticity of these exchanges. This satisfies the Cameron’s laws for Human Integration, User Control and Consent, and Justifiable Parties.</blockquote><blockquote>Like the internet is aware of the metadata to facilitate the communication, the identity metasystem will be aware of metadata only. It will neither store or be aware of the identity data. The identity data will remain with the parties that interact with the metasystem.</blockquote><ul><li><strong>Unified User Experience</strong></li></ul><blockquote>Like the internet provides a unified user experience irrespective of what browsers the subject is using, or what the site they are browsing, this Identity metasystem would provide unified user experience. It would help them to understand what to expect when they are interacting with any of the parties in the metasystem. This satisfies Cameron’s law for Consistent Experience Across Contexts.</blockquote><p>Like internet has IP (Internet Protocol) as its protocol for facilitating communication, the W3C Decentralised Identifiers Working Group has developed Decentralised Identity (DID) specification as a protocol for Identity metasystem. The next article will delve into DID protocol.</p><p>If interested to know how it all works, here is a good article: <a href="https://www.windley.com/archives/2016/10/how_sovrin_works.shtml">https://www.windley.com/archives/2016/10/how_sovrin_works.shtml</a></p><p>References:</p><p>[1] Cameron’s Laws of Identity: <a href="https://www.identityblog.com/?p=352">https://www.identityblog.com/?p=352</a></p><p>[2] Learning Digital Identity by Phillip J. Windley</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4dc71805e57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[REST API Development with Rust]]></title>
            <link>https://rudrapanda.medium.com/rest-api-development-with-rust-c58071831a5c?source=rss-b56bada386b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/c58071831a5c</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[example]]></category>
            <dc:creator><![CDATA[Rudra]]></dc:creator>
            <pubDate>Mon, 30 Aug 2021 18:32:01 GMT</pubDate>
            <atom:updated>2021-08-30T18:32:01.182Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*Clys3dE2TmPhZtknyUuduA.png" /></figure><h3><strong>Abstract</strong></h3><p>This article is an extension of article <a href="https://github.com/a759116/mos_rust#application-development-with-rust">Application Development with Rust</a>. The purpose of this article is to demonstrate development of API using Rust. This article will provide an example design and implementation of REST API to expose functionality already developed. The hope is that the reader will be able to explore the example implementation and extend this to further solidify their understanding of developing API using Rust.</p><h3><strong>Introduction</strong></h3><p>I developed this code as part of my learning Rust. I had already developed code in Rust for memory allocation functionality. I wanted to expose that functionality as REST APIs and wanted to develop it in Rust. With my research, I was able to build a REST API. In this article, I would want to share the use-cases, and the design that I based this code on. My hope is that this will help the reader not to worry about the use-case or design, rather the reader will explore the implementation using Rust, and this may expedite the Rust learning for the reader.</p><p>The use-case section of this article links to the repo with existing memory and virtual memory allocation functionality developed in Rust. The design section details the steps required to build REST API to expose existing functionality. The run and extend section details the process to run this code, and options to extend this code. I hope that this makes it easier for the reader to learn Rust.</p><h3><strong>Use-cases</strong></h3><p>The existing <a href="https://github.com/a759116/mos_rust/tree/main/src">application</a> has two modules. The module Memory has different algorithms for allocating memory for a requesting process. Here are the list of functions implemented in this module.</p><ul><li>best_fit_allocate</li><li>first_fit_allocate</li><li>worst_fit_allocate</li><li>next_fit_allocate</li><li>release_memory</li></ul><p>Similarly, the module Virtual has implementation for a number of algorithms for allocating virtual memory. This article will provide an example implementation of a REST API that is exposing best_fit_allocate functionality. The hope is that the reader will follow the example and expose other functionalities through REST APIs.</p><h3><strong>Design</strong></h3><p>I used <a href="https://actix.rs/">actix</a> framework for developing APIs in Rust.</p><p>This web application is configured to run on localhost (127.0.0.1) and port 8080. This configuration has been implemented in <strong>src/main.rs</strong>. Few other things that have been configure in src/main.rs.</p><ul><li>Tag actix_web::main is exposing this application as a web application.</li><li>HttpServer is instantiating the embedded web server to service web requests.</li><li>There are two service function calls to configure two different paths</li></ul><pre>1. localhost:8080 : This serves the contents common across all modules. I’m calling it home and it’s currently responding text “OS Services …”.</pre><pre>2. localhost:8080/os/memory/ : This serves the contents from Memory module.</pre><p>The intent for this design is to provide different contexts for different resources as per REST best practices. The hope is that the reader will extend this repo and add a new path localhost:8080/os/virtual/ to serve contents from Virtual module.</p><ul><li>The path localhost:8080/os/memory/ requires a path parameter “alloc_method”. The current implementation doesn’t require any specific value for alloc_method, it can be anything. The hope is that the reader will modify this code to call different functions based on allocation method requested. The reader may want to explore Rust enum structure and pattern matching feature “match” to implement this.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*49tdaFsydK9ijAyeWLViyQ.png" /></figure><p>The “src” folder contains code for creating different API endpoints and “tests” folder contains code to test those respective endpoints.</p><p>The folder “src” has a subfolder “services” that contains code for different contexts such as home, memory, and virtual.</p><pre>To achieve this structure in Rust,<br> <br>1. module services is defined in lib.rs</pre><pre>2. different contexts are defined as modules in services.rs. As of this writing, the modules defined are home, memory, and virtual.</pre><pre>After this step, the subfolder services is created and Rust code is written for different modules in files home.rs, memory.rs, and virtual.rs.</pre><p>As of this writing, the code is there only in memory.rs to expose best_fit_allocate functionality as a REST API.</p><ul><li>The API is expecting a JSON payload as input. Here is an example payload</li></ul><pre>{<br>    &quot;memory_map&quot;: {<br>        &quot;memory_blocks&quot;: [<br>            {<br>                &quot;start_address&quot;: 0,<br>                &quot;end_address&quot;: 1023,<br>                &quot;segment_size&quot;: 1024,<br>                &quot;process_id&quot;: 0<br>            }<br>        ]<br>    },<br>    &quot;req_size&quot;: 10,<br>    &quot;req_process_id&quot;: 32<br>}</pre><pre>To achieve this in Rust,</pre><pre>1. Two structs have been defined: MemoryMap, and MemoryAllocationRequest</pre><pre>2. I used <a href="https://serde.rs/">Serde</a> framework for serializing and deserializing these data structures. The use of derive macro from Serde generates implementations of the Serialize and Deserialize traits for these data structure.</pre><pre>3. The <a href="https://docs.rs/actix-web/3.3.2/actix_web/web/struct.Json.html">Json helper</a> from module web in crate actix-web is used to extract MemoryAllocationRequest struct structure from input Json payload.</pre><ul><li>The API calls function best_fit_allocate defined in module memory::memory of crate mos_rust to allocate memory.</li></ul><pre>To achieve this in Rust,</pre><pre>1. I’ve defined a dependency on mos_rust, and I’ve the <a href="https://github.com/a759116/mos_rust">project mos_rust</a> at path ../mos_rust with respect to this project</pre><ul><li>The API response is in Json format.</li></ul><pre>To achieve this, I used the method “json” of “HttpResponseBuilder”. The “Ok” method of HttpResponse returns “HttpResponseBuilder”. This helped to create the response in Json format.</pre><p>With the above explanation, the hope is that the reader can follow this example and expose other functions as REST APIs.</p><h3><strong>Run and Extend</strong></h3><p>There are two options to run and test the existing code.</p><ul><li>Clone or download this code. Then go to the code directory, and run the command “cargo test”. Upon successful run, you will see message like</li></ul><pre>Running tests/test_memory.rs running 2 test <br>test tests::test_index_get ... ok <br>test tests::test_best_fit_allocate_get ... ok&quot;.</pre><ul><li>Use a tool like Postman to test your API. Clone or download this code. Then go to the code directory, and run the command “cargo run”. You will see message like</li></ul><pre>Finished dev [unoptimized + debuginfo] target(s) in 0.26s<br> Running `target/debug/mos_web</pre><p>Now use Postman to send a POST request to localhost:8080/os/memory/best_fit. Here is an example request and response body for the request</p><pre><br>Request Body:<br>{<br>    &quot;memory_map&quot;: {<br>        &quot;memory_blocks&quot;: [<br>            {<br>                &quot;start_address&quot;: 0,<br>                &quot;end_address&quot;: 1023,<br>                &quot;segment_size&quot;: 1024,<br>                &quot;process_id&quot;: 0<br>            }<br>        ]<br>    },<br>    &quot;req_size&quot;: 10,<br>    &quot;req_process_id&quot;: 32<br>}<br><br>Response Body:<br>{<br>    &quot;memory_blocks&quot;: [<br>        {<br>            &quot;start_address&quot;: 0,<br>            &quot;end_address&quot;: 9,<br>            &quot;segment_size&quot;: 10,<br>            &quot;process_id&quot;: 32<br>        },<br>        {<br>            &quot;start_address&quot;: 10,<br>            &quot;end_address&quot;: 1023,<br>            &quot;segment_size&quot;: 1014,<br>            &quot;process_id&quot;: 0<br>        }<br>    ]<br>}</pre><p>Some ideas have been discussed already in this article for extending this repo. I hope the reader can expose other functions implemented in the modules Memory and Virtual as REST APIs.</p><h3><strong>Conclusion</strong></h3><p>This is an attempt to provide an example design and implementation of REST API using Rust to help the reader to learn Rust. The reader is expected to explore the code on their own. This will be revised based on feedback from readers to achieve the goal of making it easier to learn Rust.</p><p>If the reader wishes, they can build a UI using <a href="https://www.rust-lang.org/what/wasm">Rust WebAssembly</a>. This will serve as a frontend to APIs.</p><p><em>Originally published at </em><a href="https://github.com/a759116/mos_web/blob/main/README.md"><em>https://github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c58071831a5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Application Development with Rust]]></title>
            <link>https://rudrapanda.medium.com/application-development-with-rust-4bb0d5f4435b?source=rss-b56bada386b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bb0d5f4435b</guid>
            <category><![CDATA[example]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Rudra]]></dc:creator>
            <pubDate>Mon, 23 Aug 2021 21:33:51 GMT</pubDate>
            <atom:updated>2021-08-30T14:24:59.127Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*xzY8H-QSezRlMw6Lzm6GSQ.png" /></figure><p><strong>Abstract</strong></p><p>The purpose of this article is to make it easier for the reader to learn Rust. This article will provide use-case, and example design and implementation using Rust. The hope is that the reader will be able to explore the example implementation and learn Rust. The reader can extend this and apply their learning to further solidify their understanding of Rust language.</p><p><strong>Introduction</strong></p><p>I developed this code as part of my learning Rust. I had already developed the same in Python and C as part of my class assignment. I didn’t have to spend time in developing logic, rather I focussed on developing the same using Rust. This expedited my learning process. In this article, I would want to sharer the use-cases, and the design that I based this code on. My hope was that this would help the reader not to worry about the use-case or design, rather the reader would explore the implementation using Rust, and this could expedite the Rust learning for the reader.</p><p>The use-case section of this article would link to the page with problem description. The design section would details the code structure. The run and extend section would detail the process to run this code, prerequisites, and options to extend this code. I hope that this would make it easier for the reader to learn Rust.</p><p><strong>Use-cases</strong></p><p>The use-cases have been borrowed from NYU Cyberfellow class Introduction to Operating Systems. This class has a number of assignments to help the students to better understand various concepts in Operating System. This specific code has implementation for two of those assignments:</p><ul><li><a href="https://github.com/a759116/mos_rust/wiki/Use-Cases#memory-allocation-lab">Memory allocation lab</a>: This lab project addresses the various memory-management schemes in an operating system.</li><li><a href="https://github.com/a759116/mos_rust/wiki/Use-Cases#virtual-memory-lab">Virtual lab</a>: This lab project addresses the implementation of page-replacement algorithms in a demand-paging system.</li></ul><p><strong>Design</strong></p><p>The <a href="https://github.com/a759116/mos_rust">solution</a> was developed as two different modules a) Memory for Memory Allocation, and b) Virtual for pagination algorithms in a system that utilized virtual memory.</p><p>In Rust, the modules were to be defined either in main.rs or lib.rs. I had chosen to use lib.rs. You would notice that virtual was prefixed with r# in lib.rs. This was because virtual was a reserved keyword in Rust, and the prefix would help to escape the reserved keywords and use them as identifiers. For both modules defined in lib.rs, I had to create two .rs extension files: a) memory.rs, and b) virtual.rs. I again defined two modules in each of these files. For example, I had memory and test_memory in memory.rs. With this, I had a folder named memory that contained memory.rs, and test_memory.rs. The memory.rs had implementation code for Memory Allocation lab use-case. As the name suggests, test_memory had unit test code for the functionality implemented in memory.rs. Similar design was followed for implementing Virtual Lab use-case.</p><p>I used “serde” crate to enable serialization and desearilization of structured data. This was done to be able to expose the functionality as REST APIs. The details for API development would be covered in a separate <a href="https://github.com/a759116/mos_web">repo</a>. These external libraries or crates were to be defined as dependencies in Cargo.toml.</p><p><strong>Run and Extend</strong></p><p>Before downloading and running this code, please follow <a href="https://www.rust-lang.org/learn/get-started">Rust Getting Started</a>. At this point, you would have Rust installed. Now you would clone or download this code. Then go to the code directory, and run the command “cargo test — lib”. Upon successful run, you would see a message like “test result: ok. 37 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s”.</p><p>If you would want to extend this code, you could implement functionality for <a href="https://github.com/a759116/mos_rust/wiki/Use-Cases#disk-scheduling-lab">Disk Scheduling</a> and <a href="https://github.com/a759116/mos_rust/wiki/Use-Cases#cpu-scheduling-lab">CPU Scheduling</a> labs following the design stated above and using the code here as examples. The implementation code for these use-cases in C and Python are available at <a href="https://github.com/a759116/mos">GitHub Repo</a>.</p><p><strong>Conclusion</strong></p><p>This was an attempt to provide an example design and implementation to help the reader to learn Rust. The reader was expected to explore the code on their own. This would be revised based on feedback from readers to achieve the goal of making it easier to learn Rust.</p><p><em>Originally published at </em><a href="https://github.com/a759116/mos_rust/blob/main/README.md"><em>https://github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bb0d5f4435b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predict CVSS score for new CVEs]]></title>
            <link>https://rudrapanda.medium.com/predict-cvss-score-for-new-cves-49c48442576b?source=rss-b56bada386b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/49c48442576b</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[cvss]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vulnerability]]></category>
            <dc:creator><![CDATA[Rudra]]></dc:creator>
            <pubDate>Sat, 28 Nov 2020 16:24:04 GMT</pubDate>
            <atom:updated>2020-11-28T16:24:04.636Z</atom:updated>
            <content:encoded><![CDATA[<p>This article demonstrates the use of text analysis and linear regression to predict score for a newly identified software vulnerability. The solution demonstrated here relies on the historical CVE (Common Vulnerability Enumeration) data stored in NVD (National Vulnerability) database, and scikit-learn libraries. It uses the “summary” text of the vulnerability to extract features, and then predicts CVSS score (both V2 and V3 scores).</p><p>The RMSE (root mean squared error) for the current experiment, which measures how close are the predictions to the actuals, is 1.33.</p><h3>Introduction</h3><p>CVSS (common vulnerability system) is a vulnerability scoring system for software vulnerabilities that has been developed to help software industry to prioritize and remediate vulnerabilities promptly and reduce the number of attacks on systems. There are two versions: V2.0 and V3.x standards. The scores range from 0.0 to 10.0, and their qualitative severity rankings differ from V2 to V3. For V2, Low is 0.0–3.9, Medium is 4.0–6.9, and High is 7.0–10.0. For V3, Low is 0.1–3.9, Medium is 4.0–6.9, High is 7.0–8.9, and Critical is 9.0–10.0.</p><p>After disclosure of a vulnerability, the CVSS score may not be available for few days. In that situation, the proposed solution can be used to get an estimation of possible score for the newly identified vulnerability.</p><p>The rest of the paper describes data, its source, approach for extracting features, and use of LinerRegression to predict scores for different sets of features. The complete code from Jupyter Notebook is provided at the end of the article.</p><h3>Data</h3><p>Figure 1 gives a pictorial view of the dataset. The columns used for this experiment are “summary” and “cvss”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*evXd300YZpr2ZtSgSWkBLw.png" /><figcaption>Figure 1 CVE Data</figcaption></figure><p>This dataset was available on <a href="https://www.kaggle.com/andrewkronser/cve-common-vulnerabilities-and-exposures">Kaggle</a>. It had 89,660 records. It had CVE data published between Jan, 1999 and Nov, 2019.</p><h3>Feature Extraction and Selection</h3><p>The summary column is textual data that provides a high-level summary of a CVE when it gets disclosed for the first time. I broke down this text into a bag of words using “CountVectorizer” function available in “sklearn”. Here is the code extract</p><pre><em># will convert text to number vector</em></pre><pre><em>vectorizer = CountVectorizer(stop_words=stop_words)</em></pre><pre><em># convert text into word features</em></pre><pre><em>X_train = vectorizer.fit_transform(X_train)</em></pre><pre><em>X_test = vectorizer.transform(X_test)</em></pre><p>For the dataset used, there were 82,934 words after eliminating common English stop words. I used the “SelectKBest” function from “sklearn” to identify the most influential features. I used “f_regression” as the score function that uses F value to determine statistically significant features. Figure-2 shows the relationship between actual and predicted values for different number of features (words) selected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c1Vf7HHD_-3Z3V9MEz_PjA.png" /><figcaption>Figure 2 Actual vs. Predicted CVSS Scores</figcaption></figure><p>As can be seen from the picture above, there was large variation between prediction and actual when all features (words) were used. Then I experimented with multiple of 100 features (words) at a time. The variation improved with 100 words, 200, and 300 words, and started staying the same or diminished as the number of features (words) were increased.</p><h3>Model Training</h3><p>The goal was to predict CVSS score for the newly identified vulnerability that would fall between 0 and 10. Since this was a continuous target, I used multivariate regression for building the model.</p><p>For model training, I divided the total dataset into 70 and 30 ratios for training and test data. I used “LinearRegression” from “sklearn” to create the model. I evaluated the model using all, 100, 200, 300, 400, and 500 features. Figure-3 shows the comparison of RMSE¹ (Root Mean Squared Error) and R-square² values between these different experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*jfMnfle-dQeNQQyP6buCMw.png" /><figcaption>Figure 3 Model Scores</figcaption></figure><h3>Model Prediction</h3><p>I gathered 21 CVEs from NVD site those were latest CVEs to be scored as of 18th Nov 2020. I used the model to predict CVSS scores for these vulnerabilities and compared those scores against the actual scores given by NVD. Figure-4 shows the comparison for V2 scores, and Figure-5 shows for V3.x scores. The x-axis represents the difference between predicted and actual scores. The y-axis represents the frequency of this difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9HtBRMRQN9A8FEPQIYlecw.png" /><figcaption>Figure 4 CVSS V2.0 (Prediction — Actual) Distribution</figcaption></figure><p>The greater number of data closer to zero on x-axis is better, meaning the model is predicting more CVEs closer to their actual values. As can be seen from the figure above, the experiment with 300 features provides better prediction. However, this is not the case for CVSS V3.x prediction (Figure 5). The possible explanation is that CVSS V3.x score started in Sept, 2019 whereas the dataset used for this experiment has CVSS V2.0 scores only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VvT0lsBGrGsbXs4LSD9Img.png" /><figcaption>Figure 5 CVSS V3.x (Prediction — Actual) Distribution</figcaption></figure><p>Figure 6 shows a comparison between the predicted and actual values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*ENJgMnENzYvSckcEJCXYew.png" /><figcaption>Figure 6 Prediction vs. Actuals</figcaption></figure><h3>Conclusion and Future Work</h3><p>The current model can be used to estimate a possible score for the vulnerability as soon as it has been disclosed. This model can be further improved using other techniques and additional features. Here are some ideas for future work</p><ul><li>Use combination of words (n-grams) to predict score</li><li>Use CPE (Common Platform Enumeration) if available</li><li>Create separate models for CVSS V2 and V3 prediction</li><li>Use mutual info regression function to score and select features</li><li>Use Neural Network for predicting score</li></ul><p>[1] Root Mean Squared Error (RMSE): The mean squared error tells you how close a regression line is to a set of points. The lower is better.</p><p>[2] Coefficient Determination (R-squared): The coefficient of determination effectively compares the variance in the data to the variance in the residual. The residual is the difference between the predicted and observed value and its variance is the sum of squares of this difference. The closer to 1 is better.</p><h3><strong>Code</strong></h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c758b6b7d24098d9b058d2e06e2fd799/href">https://medium.com/media/c758b6b7d24098d9b058d2e06e2fd799/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49c48442576b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tutorial: neo4j, neomodel, python]]></title>
            <link>https://rudrapanda.medium.com/tutorial-neo4j-neomodel-python-bf26022921f0?source=rss-b56bada386b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf26022921f0</guid>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Rudra]]></dc:creator>
            <pubDate>Fri, 22 May 2020 16:40:11 GMT</pubDate>
            <atom:updated>2020-05-22T16:40:11.401Z</atom:updated>
            <content:encoded><![CDATA[<p>Example code for creating models with relationships, and interacting with data using both object and cypher queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*fY7J8aAKPqU_j72iK62IJw.png" /></figure><h3>Introduction:</h3><p>This tutorial will step you through the use-case, design, and development of the solution using Python, neo4j, and neomodel. There will be code examples along the way.</p><h4>Pre-requisites:</h4><p>Though not required, it will be helpful if you would have gone through <a href="https://neomodel.readthedocs.io/en/latest/">neomodel documentation</a>. You could download code for this tutorial from <a href="https://github.com/a759116/teamskill">my repo</a>.</p><p>Install Neo4J and run it with default configurations (bolt://localhost:7687). Change the password to “password”</p><p>Install Python. Install neomodel using command pip install neomodel</p><h3>Use-case:</h3><p>As a leader, I want the ability to track and manage the <strong>skills</strong>, and <strong>proficiencies</strong> on those skills over years, for the <strong>team members</strong> in a project.</p><h3>Database Design:</h3><p>The problem statement above gives indication about few entities and their relationship — all bolded. The database should have an entity for storing information about team member, another one for different skills. Proficiency is a relationship between team member and skill, and it needs to capture the proficiency level for a year.</p><p>A graph database like neo4j fits very well for this scenario. And neomodel makes it easier for Python developers to create these structures in neo4j. Here is the example code (taken from model.py). Please refer the code in repo for comments explaining important parts of the code.</p><pre><br>class Skill(StructuredNode):<br>    name = StringProperty(required=True, unique_index=True) </pre><pre># structure to define relationship attributes between Team Member and Skill<br>class ProficiencyRel(StructuredRel):<br>    year = IntegerProperty(required=True)<br>    score = FloatProperty(required=True)<br>    name = AliasProperty(to=&#39;year&#39;)<br>class TeamMember(StructuredNode):<br>    first_name = StringProperty(required=True)<br>    last_name = StringProperty(required=True)<br>    name = StringProperty(required=True)<br>    email: object = EmailProperty(required=True, unique_index=True)     <br>    skill =  Relationship(Skill,<br>                          &#39;PROFICIENCY_FOR&#39;,<br>                           cardinality=ZeroOrMore,<br>                           model=ProficiencyRel)<br>    def pre_save(self):<br>        self.email = self.first_name + &quot;.&quot; + self.last_name + &quot;@&quot; +<br>                     &quot;acme.com&quot;<br>        self.name = self.last_name + &quot;,&quot; + self.first_name</pre><h3>Application:</h3><p>The code in app.py shows how to interact with neo4j database using options provided by neomodel and cypher. There are almost 100 lines of code showing different features of neomodel and python to save and retrieve data from neo4j. I’m not copying the code here to keep it concise, and hope that you find the comments in code useful to understand the different features.</p><h3>Conclusion:</h3><p>I was looking for some programming options to work with neo4j. Based on my situation, I had to go for Python. I found that neomodel made it easy to work with neo4j. I created some examples. I hope you found it useful and it introduced you to developing applications using neo4j, Python, and neomodel.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf26022921f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science: Data Profiling]]></title>
            <link>https://rudrapanda.medium.com/data-science-data-profiling-a7183f1dd27e?source=rss-b56bada386b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7183f1dd27e</guid>
            <category><![CDATA[profiling]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Rudra]]></dc:creator>
            <pubDate>Wed, 06 Nov 2019 21:16:13 GMT</pubDate>
            <atom:updated>2019-11-06T21:16:13.227Z</atom:updated>
            <content:encoded><![CDATA[<p>Understanding your data</p><h4>Abstract:</h4><p>The first step in any data science project is to understand your data, a.k.a. data profiling. I came across two useful resources to help with this. One is an article, Profiling Big Data in distributed environment using Spark: A Pyspark Data Primer for Machine Learning, published by <a href="https://medium.com/u/b05845aaa911">Shaheen Gauher</a> on Medium. The other one is a python module called “pandas_profiling”.</p><p>In this blog post, I will provide an example of using “pandas_profiling” to profile data.</p><h4>Assumption:</h4><p>I’m making an assumption that you are already familiar with python development and installing python modules. This requires installing “pandas_profiling” module if not installed in your environment already.</p><h4>Example Code:</h4><p>The code can be found in the repo <a href="https://github.com/a759116/python-data-profiling">data-profiling</a>.</p><p><strong>Import python modules: </strong>The code is straightforward. We need pandas to read a csv file and create a pandas data frame<strong>.</strong></p><pre>import pandas as pd<br>import pandas_profiling as dprof</pre><p><strong>Load Data: </strong>I had downloaded the example dataset from <a href="https://crime-data-explorer.fr.cloud.gov/">FBI Crime Data Explorer site</a>. This file has data about number of law enforcement officers employed at different regions in US from 1960 to 2018. The file is about 197 MB and has about 1.4 M records. Click on <a href="http://s3-us-gov-west-1.amazonaws.com/cg-d4b776d0-d898-4153-90c8-8336f86bdfec/pe_1960_2018.csv">download </a>to download this data file.</p><p>The code below reads the csv file and creates a pandas data frame named as pe_data.</p><pre># load original police employment data<br>pe_file_path = &#39;./pe_1960_2018.csv&#39;<br>pe_data = pd.read_csv(pe_file_path)</pre><p><strong>Profile Data: </strong>The function “ProfileReport()” in “pandas_profiling” takes a pandas data frame as input, and creates a profile. The profile is saved as an html file that you can open in any browser.</p><pre>#data profiling police employment data<br>data_prof = dprof.ProfileReport(pe_data)<br>data_prof.to_file(&#39;./pe_data.html&#39;)</pre><h4>Performance:</h4><p>I’ve observed the profiling taking anywhere from few minutes to few hours, please be patient.</p><p>If you are performing any data clean up steps like removing duplicate records, or removing missing records etc., please reset the index before running the profiling function.</p><h4>Conclusion:</h4><p>With a few line of code, we are able to get deeper insight into the data. The report provides</p><ul><li>Dataset summary: { Number of variables, Number of observations, Total Missing (%), Total size in memory, Average record size in memory}</li><li>For every field: {Numeric or categorical, Distinct Count, Unique %, Missing %, Mean, Minimum, Maximum</li><li>Sample data, correlation, and many other useful information</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7183f1dd27e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>