<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arun on Medium]]></title>
        <description><![CDATA[Stories by Arun on Medium]]></description>
        <link>https://medium.com/@cleanpegasus?source=rss-ad035839866------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*HKMVbPCDMJQKgGANpNehQQ.jpeg</url>
            <title>Stories by Arun on Medium</title>
            <link>https://medium.com/@cleanpegasus?source=rss-ad035839866------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:52:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cleanpegasus/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Implementing KZG Commitment Scheme]]></title>
            <link>https://levelup.gitconnected.com/implementing-kzg-commitment-scheme-a18bf8ec057a?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/a18bf8ec057a</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[zero-knowledge-proofs]]></category>
            <category><![CDATA[cryptography]]></category>
            <category><![CDATA[web3]]></category>
            <category><![CDATA[ethereum]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Sat, 06 Jul 2024 13:01:30 GMT</pubDate>
            <atom:updated>2024-07-11T01:09:58.257Z</atom:updated>
            <content:encoded><![CDATA[<h4>Kate Polynomial Commitment in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w9KGs29zPPKfZbuO" /><figcaption>Photo by <a href="https://unsplash.com/@flyd2069?utm_source=medium&amp;utm_medium=referral">FlyD</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>KZG Commitment (aka Kate Polynomial Commitment) is a polynomial commitment scheme that allows a prover to compute a commitment to a polynomial such that it can be opened later at any position.</p><h4>What is a Commitment?</h4><p>A commitment is a technique that allows one party to commit to a chosen value while keeping it hidden from others, with the ability to reveal the committed value later. A commitment has two properties:</p><ol><li><strong>Binding:</strong> The value cannot be changed once it is committed</li><li><strong>Hiding:</strong> The value cannot be known until it’s revealed.</li></ol><p><strong>Hashing</strong> is an example of a commitment scheme. Once a value is hashed (committed), it cannot be changed, and the value cannot be known from the hash (commitment).</p><h4>Polynomial Commitment</h4><p>A polynomial commitment lets a prover commit to a polynomial p(x)and later prove a point (X, Y) to a verifier such that p(X) = Y without revealing the polynomial itself.</p><p>E.g.: You can commit a polynomial p(x) = x^3 -6x^2 + 11x -6 and later prove that p(1) = 0 without revealing the polynomial.</p><h4>Advantages of KZG commitment</h4><ul><li><strong>Constant-sized commitment and proof</strong>: The commitment to the polynomial and the proof are constant regardless of the polynomial’s size. Even a polynomial with degree 2⁵⁰ will have the same commitment and proof size as a polynomial with degree 2¹. This helps with scalability, reducing the amount of data that needs to be transmitted and verified.</li><li><strong>Efficient proof verification: </strong>Since the commitment and the proof are constant, the proof verification is also very efficient with only a few operations.</li><li><strong>Aggregate multiple proofs: </strong>KZG commitments support the aggregation of multiple proofs into a single one. eg: you can prove (X_1, Y_1), (X_2, Y_2).....(X_N, Y_N) lies on the polynomial p(x) with a single proof.</li></ul><p>Because of these advantages, KZG commitments are employed in verkle trees (vector commitment + Merkle Tree), Ethereum’s proto-danksharding and Scroll’s proving system. I’m currently working on an article on implementing verkle trees. So stay tuned.</p><h3>Implementation</h3><p>This implementation assumes prior knowledge of finite fields and elliptic curves. I highly recommend Rareskill’s articles on <a href="https://www.rareskills.io/post/finite-fields">Finite Field</a> and <a href="https://www.rareskills.io/post/elliptic-curve-addition">Elliptic Curves</a> if you are not familiar with these concepts.</p><p>I will not dive too deep into the math behind KZG commitment. I highly recommend <a href="https://medium.com/p/a18bf8ec057a/edit">Dankrad Feist’s article</a> if you are interested. It’s pretty straightforward.</p><p>You can find the Github repository <a href="https://github.com/CleanPegasus/kzg-commitment-python">here</a>.</p><h3><strong>Imports and initializing the finite field</strong></h3><p>This implementation uses the BLS12-381 curve.</p><p>We initialize the KZGCommitment class by setting the curve order of the BLS12–381 as the field modulus of our finite field (GF).</p><pre>import galois<br>from py_ecc.bls12_381 import G1, G2, G12, add, multiply, curve_order, pairing, eq, neg, Z1<br>import random<br>from functools import reduce<br><br>class KZGCommitment:<br>    def __init__(self, curve_order, trusted_setup_degree = 10):<br>        print(&quot;Initializing Galois Field with BLS12_318 curve order. This may take a while.&quot;<br>        self.GF = galois.GF(curve_order)<br>        print(&quot;Galois Field Initialized&quot;)<br>        (self.trusted_setup_g1, self.trusted_setup_g2) = self.generate_trusted_setup(trusted_setup_degree)</pre><h3><strong>Generating trusted setup</strong></h3><p>A one-time trusted setup has to be performed before computing KZG commitments. We pick a random element τ ∈ Finite Field. We compute <strong>[τ⁰ * G1, τ¹ * G1, τ² * G1,….. τ^n * G1]</strong> and <strong>[τ⁰*G2, τ¹*G2, τ²*G2,……τ^n * G2]</strong> where G1 and G2 are the generator points of the BLS12–381 curve.</p><p>Note that <strong>τ</strong> here is <strong>toxic waste</strong>. It needs to be discarded. If a prover knows τ, they can generate valid proofs for invalid values.</p><p>Ideally, this needs to be computed via multi-party computation (MPC). For the sake of simplicity, we will generate the trusted setup inside our class.</p><pre># generate [tau^i * G1] and [tau^i * G2]<br>def generate_trusted_setup(self, degree):<br>  print(&quot;Generating trusted setup&quot;)<br>tau = self.GF(random.randint(1, curve_order))<br>  return ([multiply(G1, int(tau ** i)) for i in range(degree + 1)], [multiply(G2, int(tau ** i)) for i in range(degree + 1)])</pre><h3><strong>Vector to Polynomial using Lagrange Interpolation</strong></h3><p>As mentioned above, KZG commitment is a polynomial commitment scheme. In that case, how can we commit to a list of values? We use a simple math trick called Lagrange interpolation.</p><p>Lagrange interpolation takes a set of points and returns a polynomial that passes through all these points. If you have a list [10, 20, 30] , you can pass the points (0, 10), (1, 20), (2, 30) . The resulting polynomial p(x) will be such that p(0) = 10, p(1) = 20 and p(2) = 30.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*IU55Rkr6H4WU0Z7vvPZ69w.png" /><figcaption>Lagrange interpolation of p(x)</figcaption></figure><p>Note that the Lagrange interpolation is computed here in a finite field to avoid precision errors.</p><pre># interpolate the given points to return a galois polynomial<br>def lagrange_interpolation_galois(self, xs, ys):<br>  assert len(xs) == len(ys), &quot;Length mismatch&quot;<br>  length = len(xs)<br><br>  def pi(j, y):<br>      p_result = self.GF(1)<br>      for k in range(length):<br>          if k == j:<br>              continue<br>          q, _ = divmod(galois.Poly([self.GF(1), -xs[k]], self.GF), (xs[j] - xs[k])<br>          p_result = p_result * q<br>      return y * p_result<br><br>  result = self.GF(0)<br>  for i in range (length):<br>      result = result + pi(i, ys[i])<br><br>  return result<br><br><br># convert an integer to a finite field value<br>def convert_int_to_gf(num):<br>  if num &lt; 0:<br>    return -GF(-num)<br>  else:<br>    return GF(num)<br><br># lagrange interpolation to convert vector to a galois polynomial<br>def vector_to_polynomial(vector):<br>  xs = [convert_int_to_gf(i) for i in range(len(vector))]<br>  vector_gf = [convert_int_to_gf(x) for x in vector]<br>  return lagrange_interpolation_galois(xs, vector_gf)<br></pre><h3><strong>Committing the polynomial</strong></h3><p>A polynomial evaluated at the trusted setup is its commitment.</p><p><strong>Commitment C = [p(τ)]₁</strong></p><p>eg: Commitment to polynomial p(x) = x^3 -6x^2 + 11x -6 would be</p><p><strong>C = [p(τ)]₁ =</strong> <strong>τ³G1– 6 * τ²G1 + 11 * τG1–6*G1</strong></p><pre>def evaluate_at_trusted_setup(self, polynomial, trusted_setup):<br>    trusted_setup = trusted_setup[:polynomial.degree + 1]<br>    return reduce(add, (multiply(point, int(coeff)) for point, coeff in zip(trusted_setup, polynomial.coeffs[::-1])), Z1)<br><br>def commit_polynomial(self, polynomial):<br>    return self.evaluate_at_trusted_setup(polynomial, self.trusted_setup_g1)</pre><p>Let’s see if our commitment scheme has both <strong>binding</strong> and <strong>hiding</strong> properties.</p><h4>Hiding</h4><p>If you notice closely, you can see that our commitment is a G1 elliptic curve point. It’s common knowledge that you cannot find the discrete logarithm of an elliptical curve point. So our polynomial cannot be revealed from the commitment.</p><h4>Binding:</h4><p>Before we dive into the binding property of our commitment scheme, I want to introduce you to the Schwartz-Zippel lemma<strong>.</strong></p><blockquote><strong>Schwartz–Zippel lemma</strong></blockquote><blockquote>Any two polynomials can intersect each other at most <em>d</em> points where d is the degree of the larger polynomial. If you have polynomials <em>p(x)</em> with degree <em>10</em> and <em>q(x)</em> with degree 3, these two polynomials will intersect at most <em>10</em> different points.</blockquote><p>For a polynomial commitment to be binding, once the prover has committed to a polynomial,p(x) he should not be able to change the polynomial to q(x). Therefore, <strong>[p(τ)]₁!= [q(τ)]₁</strong>.</p><p>Since the prover does not know <strong>τ </strong>(note that the prover only knows [<strong>τ</strong>]<strong>₁, </strong>[<strong>τ²</strong>]<strong>₁, </strong>[τ³]₁,<strong> </strong>etc.; they do not know the scalar value <strong>τ</strong>), for <strong>[q(τ)]₁</strong> to have a high probability of being equal to <strong>[p(τ)]₁, </strong>both of these polynomials have to intersect at as many places as possible. But we know from the Schwartz-Zippel lemma that two polynomials can intersect at most d points, where d is the degree of the larger polynomial.</p><p>Suppose we have 2 polynomials with degrees 2²⁰. These polynomials will intersect at at most 2²⁰ points. The curve order of our BLS12–381 curve is ~2²⁵⁶. The probability of collision here is 2²⁰ / 2²⁵⁶= ~0.</p><p>Thus, our KZG commitment scheme is binding.</p><h3><strong>Generate Proof</strong></h3><blockquote>A polynomial is divisible by (x-z) if it has a root at z . The converse is also true. If a polynomial has a root at z it is divisible by (x — z) . Let’s look at an example. Take the polynomial <strong>p(x) = x³ — 6x² + 11x — 6</strong>. One of the roots of this polynomial is 3. When you simplify this polynomial, you get <strong>p(x) = (x — 1)(x — 2)(x — 3)</strong>. As you can see it is divisible by <strong>(x — 3)</strong>.</blockquote><p>We need to prove a point (z, y) exists on the polynomial such that p(z) = y . We can see that p(x) — y is 0 at z. This means p(x) — y is divisible by (x — z).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/0*BhJDpoH80a00663S" /><figcaption>q(x)</figcaption></figure><p>The proof π of (z, y) in polynomial p(x) is the evaluation of q(x) at <strong>[τ]₁.</strong></p><blockquote>π = [q(τ)]₁</blockquote><p>Let&#39;s go a little further and see how we can prove multiple points with a single proof.</p><p>Given the points (z_0, y_0), (z_1, y_1),……(z_n, y_n)such that p(z_i) = y_i we first compute a polynomial I(x) that passes through all these points using Lagrange interpolation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*b2U7AN3QB8lO0DTihNJe6w.png" /><figcaption>Lagrange Interpolation of I(X)</figcaption></figure><p>The polynomial p(x) — I(x) is 0 at z_0, z_1, z_2,.... z_n. Therefore, it must be divisible by (x — z_0)(x — z_1)(x — z_2)....(x — z_n)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/0*29NfzLDY6b4zVBbh" /><figcaption>z(x)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/0*uHHA9tSaPp9hpGpS" /><figcaption>q(x)</figcaption></figure><p>The proof π of (z_0, y_0), (z_1, y_1),……(z_k-1, y_k-1) in polynomial p(x) is the evaluation of q(x) at [τ]<strong>₁.</strong></p><blockquote>π = [q(τ)]₁</blockquote><pre># generate a proof for a given set of point<br>def generate_proof(self, polynomial, points):<br>  x_s = [self.convert_int_to_gf(x) for x, y in points]<br>  y_s = [self.convert_int_to_gf(y) for x, y in points]<br><br>  points_polynomial = self.lagrange_interpolation_galois(x_s, y_s)<br>  numerator = polynomial - points_polynomial<br>  denominator = self.GF(1)<br>  for x in x_s:<br>      denominator = denominator * galois.Poly([self.GF(1), -x], self.GF)<br><br>  q, r = divmod(numerator, denominator)<br>  assert r == 0, &quot;This is not a valid proof.&quot;<br><br>  return self.evaluate_at_trusted_setup(q, self.trusted_setup_g1)</pre><h3><strong>Verify Proof</strong></h3><p>We have our commitment <strong>C = [p(τ)]₁ </strong>and proof <strong>π = [q(τ)]₁</strong>. The verifier needs to check the below pairing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/0*pjrHQS7dD2oMVxf5" /><figcaption>Proof verification</figcaption></figure><p>The proof essentially checks the below equation with pairing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/0*4H2mrmw8eELJz9g2" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/0*tUkqGfkmwrOaKODY" /><figcaption>Verification</figcaption></figure><pre># Verify proof<br>def verify_proof(self, commitment, points, proof):<br>  x_s = [self.convert_int_to_gf(x) for x, y in points]<br>  y_s = [self.convert_int_to_gf(y) for x, y in points]<br>  points_polynomial = self.lagrange_interpolation_galois(x_s, y_s)<br><br>  z = galois.Poly([1], self.GF)<br>  for x in x_s:<br>      z = z * galois.Poly([self.GF(1), -x], self.GF)<br>  z_s = self.evaluate_at_trusted_setup(z, self.trusted_setup_g2)<br>  lhs = pairing(z_s, proof)<br>  i_s = self.evaluate_at_trusted_setup(points_polynomial, self.trusted_setup_g1)<br>  rhs = pairing(G2, add(commitment, neg(i_s)))<br>  print(&quot;isProofValid: &quot;, eq(lhs, rhs))<br>  assert eq(lhs, rhs), &quot;The proof is invalid.&quot;</pre><p>A good proof mechanism has two properties.</p><ul><li><strong>Correctness: </strong>If the prover follows the defined steps, they should be able to produce a valid proof. This is evident from the above equation.</li><li><strong>Soundness: </strong>A prover cannot provide incorrect proof as valid proof.</li></ul><p>If a prover wants to provide an invalid proof p(z) = y&#39; where p(z) is not equal toy’, he needs to divide p(x) — y&#39; by (x-z) without a remainder. But this division is clearly not possible since p(z) != y&#39; .</p><p>Let’s try to create a fake proof from our verification equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/0*4H2mrmw8eELJz9g2" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/0*OFm5Myr0Ru_J4e_q" /></figure><p>To provide a fake proof, the prover has to know the scalar value of <strong>τ </strong>so he can change z(x) such that it will create a valid proof. Since the prover does not know <strong>τ</strong>, it is impossible to create a fake proof.</p><p>Thus, our proof is both correct and sound.</p><h3>KZG Commitment Test</h3><p>Now that we have completed all parts of our commitment scheme (commitment, proof generation, and verification), let’s test it on a list of values.</p><p>The below code commits a vector [10, 20, 36, 50, 90] as a polynomial using KZG commitment scheme. We create a proof for two points (0, 10), (1, 20) and verify if the proof is valid.</p><pre>if __name__ == &#39;__main__&#39;:<br>    kzg = KZGCommitment(curve_order)<br>    vector = [10, 20, 36, 50, 90]  # example vector<br>    print(&quot;Commiting Vector: &quot;, vector)<br>    polynomial = kzg.vector_to_polynomial(vector)<br>    commitment = kzg.commit_polynomial(polynomial)<br>    points = [(0, 10), (1, 20)]  # element 10 at index 0 and element 20 at index 1<br>    proof = kzg.generate_proof(polynomial, points)<br>    print(&quot;Proof: &quot;, proof)<br>    kzg.verify_proof(commitment, points, proof)</pre><h3>About me</h3><p>I’m a blockchain engineer with an interest in cryptography and zero knowledge proofs. Hit me up on <a href="https://x.com/CleanPegasus">Twitter</a> if you want to chat about web3/crypto/zkp.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a18bf8ec057a" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/implementing-kzg-commitment-scheme-a18bf8ec057a">Implementing KZG Commitment Scheme</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing Decentralised Identity with ZK-SNARKS(Part-1)]]></title>
            <link>https://levelup.gitconnected.com/implementing-decentralised-identity-with-zk-snarks-68f9fca32c47?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/68f9fca32c47</guid>
            <category><![CDATA[solidity]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[zero-knowledge-proofs]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[web3]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Mon, 16 Oct 2023 13:56:06 GMT</pubDate>
            <atom:updated>2023-10-19T02:19:04.560Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building On-chain Identity</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bJOi-xmjEDgnTJv1" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=medium&amp;utm_medium=referral">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction to Decentralized Identity</h3><p>In the digital age, our identities are often managed by centralized entities, leaving our personal data vulnerable and out of our direct control. Decentralized identity offers a paradigm shift: it empowers individuals by giving them ownership of their identity. This innovative approach not only enhances security but also returns control to the individual, ensuring that identity is both private and user-centric.</p><h3>Problems with centralized identity</h3><ol><li><strong>Data Vulnerability</strong>: Centralized databases become prime targets for cyberattacks. A breach in one system can expose the personal data of millions, leading to potential misuse.</li><li><strong>Lack of User Control</strong>: In centralized systems, users typically have little say over how their data is used. Institutions or corporations decide the terms, often leading to unauthorized sharing or selling of personal data.</li><li><strong>Single Point of Failure</strong>: Centralized systems are vulnerable to both technical and administrative failures. If the system goes down, every user is affected, and if the administrative body makes an error, it can have wide-reaching consequences.</li><li><strong>Inefficiency and Cost:</strong> Centralized identity verification often requires manual processing or repeated verification across different platforms, resulting in both time and cost inefficiencies.</li><li><strong>Privacy Concerns:</strong> Centralized entities might have access to more information than necessary, leading to potential misuse or unwarranted surveillance.</li><li><strong>Interoperability Issues:</strong> Data stored in one centralized system may not be easily compatible or accessible with another, hindering seamless user experiences across platforms.</li></ol><h3>Understanding Zero-Knowledge Proofs (ZKPs):</h3><p>At its essence, a zero-knowledge proof is a cryptographic method that allows one party (the prover) to prove to another party (the verifier) that a statement is true without revealing any specific information about the statement itself. Imagine proving that you are over 21 years old without providing your date of birth.</p><p>Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge (zk-SNARKs) is a specific type of zero-knowledge proof that has three distinct qualities: zero-knowledge, succinctness, and non-interactive. Let’s break it down.</p><ol><li><strong>Zero-Knowledge</strong>: Like all zero-knowledge proofs, zk-SNARKs enable one party to prove to another that a statement is true without revealing any information about the statement itself.</li><li><strong>Succinctness</strong>: zk-SNARKs are incredibly efficient. Both the proof and the verification process are short and quick, making it practical for blockchain applications where efficiency is a priority.</li><li><strong>Non-Interactive</strong>: The prover can create a proof without any interaction from the verifier. Once the proof is generated, any verifier can confirm its accuracy independently.</li></ol><h4><strong>How it works</strong></h4><ol><li><strong>Setup Phase</strong>: The first step in creating a zk-SNARK is the setup phase. This phase is crucial for generating the cryptographic keys used in the process. The setup generates a proving key (for the prover) and a verification key (for the verifier).</li><li><strong>Creating the Proof</strong>: The prover uses the proving key to produce proof. The prover takes the statement to be proven (let’s say you’re over 21), applies it to a mathematical function called a “circuit,” and generates a proof using their secret information and the proving key. Importantly, the proof doesn’t contain the secret (your age), but it does mathematically encapsulate the truth of the statement.</li><li><strong>Verification</strong>: The verifier uses the verification key from the setup phase to check the proof. The verification process is computationally light, making zk-SNARKs incredibly efficient for blockchain systems. If the proof checks out, the verifier can be certain that the statement is true, all without knowing the statement.</li></ol><h3>Implementation</h3><p>Here we have it, our first step into decentralized identity with the Soul Bond Token (SBT). This contract is just version one. As I work on it, there will be improvements and added features. Bear with me here.</p><h4>Storing Identity On-Chain using Soul Bond Tokens:</h4><p>The contract is designed to manage decentralized identities using Soul Bond Tokens (SBTs) on the blockchain. At its core, the contract provides a mechanism for specific verified entities like the government to issue, or &quot;mint&quot;, SBTs for users. Each token contains crucial identity parameters: a UID and cryptographic hashes of the user&#39;s name and age.</p><p>This system goes a step further by allowing additional entities, such as banks or educational institutions, to add profiles to a user’s SBT. For instance, a bank might add a profile reflecting a credit score, or a university might append a degree verification. These added profiles, represented as cryptographic hashes, maintain user privacy while still enabling verification. And with the integration of zero-knowledge proofs, it’s possible to verify the truthfulness of the data without revealing the actual information itself. In essence, theDIdentity contract offers a more structured approach to decentralized identity, blending the benefits of blockchain with modern cryptographic techniques.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/071be6cf887cbfab594c81599794a287/href">https://medium.com/media/071be6cf887cbfab594c81599794a287/href</a></iframe><p><strong>Deploy the contract and mint an identity for the user:</strong></p><p>The provided JavaScript code delves into deploying and minting an identity for a user. Once deployed, we can mint an identity by hashing a unique user ID (UID), name, and date of birth (DoB) and then saving this information on-chain. Lastly, it features a specialized hashing function, poseidonHashthat uses the Poseidon hash algorithm, a zk-SNARK-friendly function, to hash given inputs securely.</p><pre>const hre = require(&quot;hardhat&quot;);<br>const ethers = require(&quot;ethers&quot;);<br>const circomlibjs = require(&quot;circomlibjs&quot;);<br>const snarkjs = require(&quot;snarkjs&quot;);<br>const fs = require(&quot;fs&quot;);<br><br>// Deploy the contracts<br>async function deploy() {<br>  const dIdentityContract = await hre.ethers.deployContract(&quot;DIdentity&quot;);<br>  await dIdentityContract.waitForDeployment();<br>  console.log(&quot;DIdentity deployed to:&quot;, dIdentityContract.target);<br>  return dIdentityContract;<br>}<br><br>// Mint an Identity<br>async function mintIdentity(signer, name, DoB, dIdentityContract) {<br>  const UID = ethers.sha256(ethers.toUtf8Bytes(signer.address + name + DoB));<br>  const nameHash = ethers.sha256(ethers.toUtf8Bytes(signer.address + name));<br>  const DoBHash = await poseidonHash([signer.address, DoB]);<br><br>  const identity = {<br>    UID: UID,<br>    nameHash: nameHash,<br>    dobHash: DoBHash,<br>  }<br>  <br>  const tx = await dIdentityContract.mint(signer.address, identity);<br>  await tx.wait();<br>}<br><br>// Poseidon Hash<br>async function poseidonHash(inputs) {<br>  const poseidon = await circomlibjs.buildPoseidon();<br>  const poseidonHash = poseidon.F.toString(poseidon(inputs));<br>  return poseidonHash;<br>}</pre><h4><strong>Creating ZK-SNARK Circuits with Circom:</strong></h4><p>In this example, we use Circom to create the circuits for ZK-SNARKs. Since Circom is written in Rust, you’ll have to install the Rust environment.</p><pre>curl --proto &#39;=https&#39; --tlsv1.2 https://sh.rustup.rs -sSf | sh</pre><p>Once Rust is installed, you need to build the Circom compiler.</p><pre>git clone https://github.com/iden3/circom.git<br>cd circom<br>cargo build --release<br>cargo install --path circom</pre><p>The Circomlib library has a ton of predefined circuits. This saves us the headache of writing our own circuits for complex things like hashing. The Snarkjs library allows us to do a trusted setup, and generate and verify proofs.</p><pre>npm install circomlib<br>npm install snarkjs</pre><p><strong>Age Proof</strong></p><p>The provided circuit describes a zero-knowledge proof mechanism for age verification without revealing the exact birth date. In the age of data privacy, we&#39;d love to validate a user&#39;s age without revealing their exact birth date, and this code aims to do just that.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e7a533aa173de6586939671aac02f32/href">https://medium.com/media/7e7a533aa173de6586939671aac02f32/href</a></iframe><p>Here’s what each section does:</p><ol><li>Imports: The code starts by importing two essential circuit components from the circomlib library - Poseidon, a zk-SNARK-friendly hash function, and a set of comparators for numeric checks.</li><li>Template AgeProof: This is where the magic happens. (<strong>Format this better</strong>)<br>- Inputs: It takes in the doBTimestamp (a timestamp of the user&#39;s date of birth), address (Ethereum address of the user), currentTimestamp (the current time for age calculation), ageThreshold (the age limit we&#39;re checking against, say 18 years converted to seconds), and hash (a combination of the address and DoB for validation).<br>- Age Calculation: It computes the user’s age by subtracting the doBTimestamp from the currentTimestamp.<br>- Age Check: Using the LessThan comparator, the code ensures the computed age surpasses the ageThreshold. This is our gatekeeper, ensuring the age requirement is met.<br>- Hash Validation: The Poseidon hashing mechanism is then employed to verify the provided hash against the computed hash of the user&#39;s address and DoB. This ensures that the data being presented hasn&#39;t been tampered with.</li><li>Component Main: The main component is an instantiation of the AgeProof template, taking in the public parameters needed to generate or validate the proof.</li></ol><p>In essence, this circuit can be used to confirm if someone’s age exceeds a specific threshold (like being over 18) without revealing their actual age, making it a powerful tool in the realm of decentralized identity and privacy.</p><p><strong>Credit Proof</strong></p><p>The below code is designed to determine the creditworthiness of a user. The bank can set a profile with the hash of the credit score stored in the user profile.</p><pre>async function createProfile(signer, address, creditScore, dIdentityContract) {<br>  const hash = await poseidonHash([address, creditScore]);<br>  const profile = {<br>    entity: &quot;Bank Credit Score&quot;,<br>    dataHash: hash,<br>    timestamp: Date.now(),<br>  }<br>  const tx = await dIdentityContract.connect(signer).setProfile(address, profile);<br>  await tx.wait();<br>}</pre><p>The below circuit helps the user prove that their credit score is within a certain range.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c892302e9e79f3ee77f00191ab06de6b/href">https://medium.com/media/c892302e9e79f3ee77f00191ab06de6b/href</a></iframe><p>Let’s break down this circuit:</p><ol><li>Imports: We’re just using some handy tools from the circomlib library for hashing and numeric checks.</li><li>Template CreditProof:<br>- Inputs: We’ve got the user’s creditScore, a minCreditScore and maxCreditScore to define the acceptable range, the user&#39;s address, and a hash.<br>- Range Check: The code checks if the given creditScore is within the set range.<br>- Hash Check: We’re using Poseidon, a fancy name for a hash function, to double-check that the data hasn’t been tampered with.</li><li>Component Main: It’s the “public interface”, meaning these are the bits others will use to communicate with our circuit.</li></ol><p>In a nutshell, using this zk-SNARK circuit, lenders in the DeFi space can validate if someone meets their credit requirements without ever knowing the person’s exact credit score, showcasing the blend of privacy and trust in decentralized systems.</p><h4>Compiling the Circuits</h4><p>At first, we need to compile the circom circuit and generate the wasm and r1cs files for the circuit.</p><pre>circom age_proof.circom --wasm --r1cs -o ./build</pre><pre>circom credit_proof.circom --wasm --r1cs -o ./build</pre><p>We need a proving key to generate the proof for the circuit. We need a ptau file to generate a proving key for this circuit. We can either generate a new one using Snarkjs or download a pre-generated one. In this example, we’ll download a pre-generated ptau file.</p><pre>wget https://hermez.s3-eu-west-1.amazonaws.com/powersOfTau28_hez_final_12.ptau</pre><p>We can use this ptau file to generate a proving key for our circuits.</p><pre>npx snarkjs groth16 setup build/age_proof.r1cs powersOfTau28_hez_final_12.ptau circuit_age.zkey</pre><pre>npx snarkjs groth16 setup build/credit_proof.r1cs powersOfTau28_hez_final_12.ptau circuit_credit.zkey</pre><h4>Creating Zero Knowledge Proofs (ZKPs)</h4><p>Let’s delve into a snippet that deals with creating a zero-knowledge proof for credit scores. The below function accepts four parameters — signer, creditScore, minCreditScore, and maxCreditScore. The signer is the user for whom the proof is being created, and their credit score along with the acceptable minimum and maximum scores are passed as arguments. First, it calculates a hash of the signer&#39;s address and their credit score using Poseidon, a zk-SNARK-friendly hash function. This hash is then fed, along with the other parameters, into the zk-SNARK circuit snarkjs.groth16.fullProve to generate the proof and public signals for the given arguments. This proof and public signals are used for on-chain verification, thereby ensuring that your creditworthiness can be proven without revealing the score itself.</p><pre>// Create a credit proof<br>async function createCreditProof(signer, creditScore, minCreditScore, maxCreditScore) {<br>  const hash = await poseidonHash([signer.address, creditScore]);<br>  const {proof, publicSignals} = await snarkjs.groth16.fullProve(<br>    { creditScore: creditScore, minCreditScore: minCreditScore, maxCreditScore: maxCreditScore, address: signer.address, hash: hash },<br>    &quot;build/credit_proof_js/credit_proof.wasm&quot;,<br>    &quot;circuit_credit.zkey&quot;);<br><br>  const creditProof = proof;<br>  const creditPublicSignals = publicSignals;<br>  return { creditProof, creditPublicSignals };<br>}</pre><p>In the same vein as our credit score verification, we’ve also implemented a function createAgeProof that performs a similar function—but this time for age verification. Imagine you want to confirm that someone is over 18 for a service you offer, but you don&#39;t want to know their exact age.</p><pre>// Create age proof<br>async function createAgeProof(signer, doBTimestamp, currentTimestamp, ageThreshold) {<br>  const hash = await poseidonHash([signer.address, doBTimestamp]);<br>  const { proof, publicSignals } = await snarkjs.groth16.fullProve(<br>    { doBTimestamp: doBTimestamp, address: signer.address, currentTimestamp: currentTimestamp, ageThreshold: ageThreshold, hash: hash }, <br>    &quot;build/age_proof_js/age_proof.wasm&quot;, <br>    &quot;circuit_age.zkey&quot;);<br><br>  return { proof, publicSignals };<br>}</pre><p>The createAgeProof the function takes in four parameters: signer, doBTimestamp, currentTimestamp, and ageThreshold. The signer is the user&#39;s Ethereum address, doBTimestamp is the timestamp for the user&#39;s date of birth, currentTimestamp is the current time, and ageThreshold is the minimum age requirement (all in seconds). First off, we use Poseidon to hash the signer&#39;s address and the date of birth timestamp. Next, the function takes this hash, along with the other data points, and creates a zk-SNARK proof using snarkjs.groth16.fullProve. Similar to the previous function, we use the proof and public signals for on-chain verification.</p><h4>Verification of ZKPs</h4><p>We need to create verification keys from the proving key to verify the generated proof.</p><pre>npx snarkjs zkey export verificationkey circuit_age.zkey verification_key_age.json</pre><pre>npx snarkjs zkey export verificationkey circuit_credit.zkey verification_key_credit.json</pre><p><strong>Verifying Age proof</strong></p><p>When it comes to putting our age-proving function to the test, we need a way to verify these zero-knowledge proofs. That’s exactly what the verifyAgeProof function is designed to do. This function accepts four parameters: address, proof, publicSignals, and dIdentityContract.</p><pre>// Verify Age zk-proof<br>async function verifyAgeProof(address, proof, publicSignals, dIdentityContract) {<br>  const id = await dIdentityContract.getID(address);<br>  const vKey = JSON.parse(fs.readFileSync(&quot;verification_key_age.json&quot;));<br>  const res = await snarkjs.groth16.verify(vKey, publicSignals, proof);<br>  return (res &amp;&amp; (id.dobHash == publicSignals[3]));<br>}</pre><p>First, it fetches the decentralized identity of the user associated with the provided address by invoking the getID method of our dIdentityContract. This gives us a user&#39;s identity information, including their hashed date of birth (dobHash).</p><p>Next up, the function reads the verification key from a JSON file. You might be wondering why we need a verification key. Well, it’s like the public part of a lock-and-key system for zk-SNARKs, ensuring that the proof can be publicly verified.</p><p>Now comes the moment of truth: the function used snarkjs.groth16.verify to verify the provided zk-SNARK proof against the publicSignals and the verification key. If the verification is successful, we&#39;re not done just yet! We also make sure that the hashed date of birth in the identity contract matches one of the publicSignals.</p><p>So, if everything lines up — the zk-SNARK proof is verified and the date of birth hash matches — we return true, confirming the user&#39;s age without ever revealing what it is.</p><p><strong>Verifying Credit Proof</strong></p><pre>// Verify Credit zk-proof<br>async function verifyCreditProof(profiler, address, proof, publicSignals, dIdentityContract) {<br>  const profile = await dIdentityContract.getProfile(profiler, address);<br>  const dataHash = profile.dataHash;<br>  const vKey = JSON.parse(fs.readFileSync(&quot;verification_key_credit.json&quot;));<br>  const res = await snarkjs.groth16.verify(vKey, publicSignals, proof);<br>  return (res &amp;&amp; (dataHash == publicSignals[3]));<br>}</pre><p>This function takes in five arguments, including the profiler (eg, bank), the user&#39;s address, a zk-SNARK proof, publicSignals, and the decentralized identity contract (dIdentityContract).</p><p>The function starts by fetching the user’s profile data from the smart contract and then reads the verification key from a stored JSON file. With these pieces in place, it calls on the snarkjs.groth16.verify function to authenticate the zk-SNARK proof. If the proof holds up and matches the hash stored in the user&#39;s profile, the function returns true, confirming that the credit data is both authentic and matches the stored record. It’s a secure yet straightforward way to maintain the integrity of financial data on a decentralized network.</p><h3>Conclusion</h3><p>So, that’s where we are with this project for now. I’m working on expanding it to be a more complete decentralized identity system on-chain. It’s a step-by-step process, and there’s a lot more to come. I’ll continue to blog about my progress, so stay tuned for Part 2.</p><p>You can find this project’s GitHub repository <a href="https://github.com/CleanPegasus/Decentralized-Identity">here</a>.</p><p>If you want to chat about web3, zk proofs, quantum computing, or AI, feel free to hit me up on <a href="https://twitter.com/CleanPegasus">Twitter</a>. I’m always open to a good tech discussion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68f9fca32c47" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/implementing-decentralised-identity-with-zk-snarks-68f9fca32c47">Implementing Decentralised Identity with ZK-SNARKS(Part-1)</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding DeFi — The Future of Finance]]></title>
            <link>https://medium.com/coinmonks/understanding-defi-the-future-of-finance-75aee54efb15?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/75aee54efb15</guid>
            <category><![CDATA[decentralized-finance]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Tue, 09 Nov 2021 07:01:19 GMT</pubDate>
            <atom:updated>2021-11-09T12:57:27.545Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding DeFi — The Future of Finance</h3><h4>Finance on Blockchain</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zbsGWvLahd8Zva37" /><figcaption>Photo by <a href="https://unsplash.com/@etiennemartin?utm_source=medium&amp;utm_medium=referral">Etienne Martin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The finance industry has grown exponentially over the years with the growth of technology. Decentralized Finance(DeFi) is a new financial system that aims to be trust-free, open to everyone, permissionless and doesn’t require middlemen like banks. To achieve this, DeFi uses Blockchain technology and smart contracts. Currently, most DeFi projects are built on the Ethereum blockchain(and its Layer 2 chains) due to its high security, robust programming language and strong developer ecosystem.</p><p>How it started?</p><p>One of the first DeFi projects was MakerDAO, founded in 2015. It allowed users to lock in ETH as collateral and generate DAI. DAI is a stablecoin that is algorithmically pegged to the US dollar such that 1 DAI always equals 1 USD. DAI can also be used for savings on MakerDAO’s oasis platform. This recreates traditional finance’s ability to lend and borrow assets.</p><h3>How is DeFi better than Traditional Finance(TradFi) or Centralized Finance(CeFi)?</h3><p>Traditional finance consists of the traditional banks that you come across in your everyday life and Centralized finance consists of any system that has its orders routed through a single exchange system.</p><p>Some examples of TradFi — Bank of America, Wells Fargo, JP Morgan Chase</p><p>Some examples of CeFi — Coinbase, Nexo, BlockFi, Celcius</p><p>Some examples of DeFi — Aave, Compound, Uniswap</p><p>Unlike banks, DeFi protocols are permissionless, secure and anonymous. Anyone with enough collateral can take a loan with DeFi. While banks regulate the interest percentage on loans and savings to match with their agenda, DeFi protocols offer interest percentages based on supply and demand.</p><p>In most countries with unstable economies, the banks close up often due to losses, taking the users funds along with them. Both traditional banks and CeFi companies suffer from Hacks, Bad loans, Inside jobs and Rug pull.</p><p>As of writing this blog, the most popular DeFi protocols are AAVE and Compound with over 26.8 billion US dollars worth of Total Value Locked(TVL) in AAVE and over 19 billion USD of TVL in Compound.</p><h3>How does DeFi Work?</h3><p>DeFi mainly works by creating money markets for specific tokens like ETH, DAI, USDC, USDT, LINK, etc. Users supply tokens to these money markets and earn interest on their savings.</p><p>When a user deposits their token, they receive an alternate token that is equivalent to the supplied token + interest. For example, Compound provides CETH tokens when ETH is deposited. The CETH tokens are just like any other ERC20 tokens and can be transferred between different wallets. CETH is required to withdraw the underlying ETH locked in the money market. The exchange rate between ETH and CETH tokens will keep increasing with each block. Thus, when you exchange your CETH for ETH at a later time, you’ll get the total deposited ETH and the interest on top of it.</p><p>To borrow money from a DeFi protocol, collateral must be placed. Currently, all loans are overcollateralized. The borrower must supply tokens worth more than the actual loan. Every token deposited as collateral has a collateral factor that determines how much you can borrow. For example, ETH and DAI have a 75% collateral factor on Compound. This means that, for every ETH that is collateralized, you can borrow 0.75 ETH.</p><p>The big question now is, why would someone take a loan when they can just liquidate their funds and get money? People generally take loans to avoid/delay paying their capital gains taxes or they don’t want to sell their tokens. There is a limit on how much can be borrowed. This limit depends on the amount of funds in the money market and the quality of your collateral.</p><p>The borrowed funds should always be less than the collateral times the collateral factor.</p><blockquote>Borrowed Amount &lt; Collateral * Collateral Factor</blockquote><p>If this condition fails, the collateral will be automatically liquidated and the protocol will repay the difference to the borrower.</p><p>The interest rates for lenders and borrowers are determined by the ratio between supply and the borrowed tokens in a particular market. The interests are calculated in every block, thus it is always a variable. Protocols like AAVE offers a stable APY in the short term but changes over the long term based on supply and demand.</p><h3>Risks in DeFi</h3><p>The major risk in DeFi comes from poorly written smart contracts. A poorly written smart contract can be susceptible to hacks where the money market can lose all its funds. The other risks with DeFi are the volatility of cryptocurrencies and the quickly changing APYs.</p><h3>Conclusion</h3><p>DeFi is hugely disrupting traditional finance. The permissionless, open and censorship-resistant nature of DeFi is crucial in the present condition where government policies and traditional banks are not built to serve the common people. I’m incredibly bullish on DeFi. Always DYOR.</p><h3>About Me</h3><p>I have been developing on Ethereum for a while now and have been investing in various cryptocurrencies as well. I mostly work with smart contracts and solidity programming. If you have a fun project in mind and want to collaborate with a solidity developer, hit me up :)</p><p>You can find me on <a href="https://twitter.com/CleanPegasus">Twitter</a> where I post about blockchain development and crypto in general.</p><blockquote>Join Coinmonks <a href="https://t.me/coincodecap">Telegram Channel</a> and <a href="https://www.youtube.com/c/coinmonks/videos">Youtube Channel</a> learn about crypto trading and investing</blockquote><h4>Also, Read</h4><ul><li><a href="https://medium.com/coinmonks/best-crypto-signals-telegram-5785cdbc4b2b">Best Crypto Trading Signals Telegram</a> | <a href="https://medium.com/coinmonks/moonxbt-review-6e4ab26d037">MoonXBT Review</a></li><li><a href="https://medium.com/coinmonks/coinswitch-kuber-review-1a8dc5c7a739">Coinswitch Kuber Review</a> | <a href="https://blog.coincodecap.com/grid-trading">Grid Trading Bots</a> | <a href="https://medium.com/coinmonks/coinbase-fees-831e77d4f2c5">Coinbase Fees</a></li><li><a href="https://blog.coincodecap.com/bitget-review">Bitget Review</a> | <a href="https://blog.coincodecap.com/gemini-vs-blockfi">Gemini vs BlockFi</a> | <a href="https://blog.coincodecap.com/okex-futures-trading">OKEx Futures Trading</a></li><li><a href="https://blog.coincodecap.com/okex-kucoin">OKEx vs KuCoin</a> | <a href="https://blog.coincodecap.com/celsius-alternatives">Celsius Alternatives</a> | <a href="https://blog.coincodecap.com/buy-vechain">How to Buy VeChain</a></li><li><a href="https://blog.coincodecap.com/binance-futures-trading">Binance Futures Trading</a> | <a href="https://blog.coincodecap.com/mudrex-3commas-etoro">3Commas vs Mudrex vs eToro</a></li><li><a href="https://blog.coincodecap.com/crypto-arbitrage-in-india">Earn Passive Income Using Crypto Arbitrage In India</a></li><li><a href="https://blog.coincodecap.com/crypto-exchanges-in-germany">Best Crypto Exchanges in Germany</a> | <a href="https://blog.coincodecap.com/wazirx-p2p">WazirX P2P</a></li><li><a href="https://blog.coincodecap.com/buy-monero">How to buy Monero</a> | <a href="https://blog.coincodecap.com/idex-review">IDEX Review</a> | <a href="https://blog.coincodecap.com/bitkan-trading-bot">BitKan Trading Bot</a></li><li><a href="https://blog.coincodecap.com/buy-shiba-bitbns">How to buy Shiba(SHIB) Coin on Bitbns?</a> | <a href="https://blog.coincodecap.com/binance-in-india">Binance in India</a></li><li><a href="https://blog.coincodecap.com/binance-vs-bitstamp">Binance vs Bitstamp</a> | <a href="https://blog.coincodecap.com/bitpanda-coinbase-coinsbit">Bitpanda vs Coinbase vs Coinsbit</a></li><li><a href="https://blog.coincodecap.com/buy-ripple-india">How to buy Ripple (XRP)</a> | <a href="https://blog.coincodecap.com/crypto-exchange-africa">Best Crypto Exchanges in Africa</a></li><li><a href="https://blog.coincodecap.com/crypto-exchange-africa">Best Crypto Exchanges in Africa</a> | <a href="https://blog.coincodecap.com/hoo-exchange-review">Hoo Exchange Review</a></li><li><a href="https://blog.coincodecap.com/etoro-robinhood">eToro vs Robinhood</a> | <a href="https://blog.coincodecap.com/bybit-bityard-moonxbt">MoonXBT vs Bybit vs Bityard</a></li><li><a href="https://blog.coincodecap.com/stormgain-review">Stormgain Review</a> | <a href="https://blog.coincodecap.com/bexplus-review">Bexplus Review</a> | <a href="https://blog.coincodecap.com/binance-vs-bittrex">Binance vs Bittrex</a></li><li><a href="https://blog.coincodecap.com/bookmap-review-2021-best-trading-software">Bookmap Review</a> | <a href="https://blog.coincodecap.com/crypto-exchange-usa">5 Best Crypto Exchanges in the USA</a></li><li><a href="https://blog.coincodecap.com/ftx-futures-trading">How to trade Futures on FTX Exchange</a> | <a href="https://blog.coincodecap.com/okex-vs-binance">OKEx vs Binance</a></li><li><a href="https://blog.coincodecap.com/buy-domain-on-unstoppable-domains">How to Buy Domain Name on Unstoppable Domains?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75aee54efb15" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coinmonks/understanding-defi-the-future-of-finance-75aee54efb15">Understanding DeFi — The Future of Finance</a> was originally published in <a href="https://medium.com/coinmonks">Coinmonks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to TinyML]]></title>
            <link>https://medium.com/data-science/an-introduction-to-tinyml-4617f314aa79?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/4617f314aa79</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Tue, 10 Nov 2020 12:27:28 GMT</pubDate>
            <atom:updated>2020-11-10T15:23:23.972Z</atom:updated>
            <content:encoded><![CDATA[<h4>Machine Learning meets Embedded Systems</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nNDwamhwR_SNJJIY5dNdlw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@alexkixa?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Alexandre Debiève</a> on <a href="https://unsplash.com/s/photos/electronics?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>We are currently living in a world surrounded by Machine Learning models. Over the course of your day, you are utilising these models more than you realize. Day to day tasks like scrolling through social media, taking a picture, checking the weather, all depend on machine learning models. You might even see this blog because a machine learning model suggested this to you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*poXRRZdZAElrrP9C3ZQLoQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@tvick?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Taylor Vick</a> on <a href="https://unsplash.com/s/photos/data-center?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>We all know that training these models are computationally expensive. But most of the times, running inference on these models are computationally expensive as well. The rate at which we are using machine learning services, we need computing systems that are fast enough to handle it. Thus most of these models run on huge data centres with clusters of CPUs and GPUs(even TPUs in some cases).</p><blockquote>Big is not always better</blockquote><p>When you take a picture, you want the machine learning magic to happen instantly. You don’t want to wait for the image to be sent to a data centre where it is being processed and sent back again. In this case, you want the machine learning model to run locally.</p><p>When you say “Alexa” or “Ok, Google”, you want your devices to respond to you instantly. Waiting for the device to send your voice to the servers where it is processed, and the information is retrieved. This takes time and cripples the user experience. Again, in this case, you want the machine learning model to run locally.</p><h4>What is TinyML</h4><p>TinyML is a field of study in Machine Learning and Embedded Systems that explores the types of models you can run on small, low-powered devices like microcontrollers. It enables low-latency, low power and low bandwidth model inference at edge devices. While a standard consumer CPUs consume between 65 watts and 85 watts and standard consumer GPU consumes anywhere between 200 watts to 500 watts, a typical microcontroller consumes power in the order of milliwatts or microwatts. That is around a thousand times less power consumption. This low power consumption enables the TinyML devices to run unplugged on batteries for weeks, months, and in some cases, even years, while running ML applications on edge.</p><blockquote>The Future of ML is Tiny and Bright</blockquote><h4>Advantages of TinyML</h4><ol><li><strong>Low Latency: </strong>Since the model runs on the edge, the data doesn&#39;t have to be sent to a server to run inference. This reduces the latency of the output.</li><li><strong>Low Power Consumption:</strong> As we discussed before, microcontrollers consume very little power. This enables them to run without being charged for a really long time.</li><li><strong>Low Bandwidth:</strong> As the data doesn’t have to be sent to the server constantly, less internet bandwidth is used.</li><li><strong>Privacy: </strong>Since the model is running on the edge, your data is not stored in any servers.</li></ol><h4>Applications of TinyML</h4><p>By summarizing and analyzing data at the edge on low power devices, TinyML offers many unique solutions. Even though TinyML is an emerging field, it has been used in production for years. The “OK Google”, “Alexa”, “Hey Siri” wake words are an example of TinyML. Here, the devices are always on and are analyzing your voice to detect the wake word. I’ll add some more applications of TinyML here.</p><ol><li><strong>Industrial Predictive Maintenance: </strong>Machines are prone to fault. Using TinyML on low powered devices, it is possible to monitor the machine and predict faults ahead of time constantly. This predictive maintenance can lead to significant cost savings. Ping Services, an Australian startup, has introduced an IoT device that autonomously monitors wind turbines by magnetically attaching to the outside of the turbine and analyzing detailed data at the edge. This device can alert the authorities regarding potential issues even before it occurs.</li><li><strong>Healthcare: </strong>The Solar Scare Mosquito project uses TinyML to curb the spread of mosquito-borne diseases like Dengue, Malaria, Zika Virus, Chikungunya, etc. It works by detecting the mosquito breeding conditions and agitates the water to prevent mosquito breeding. It runs on solar power and can thus run indefinitely.</li><li><strong>Agriculture: </strong>The Nuru app helps farmers detect diseases in plants just by taking a picture of it by running Machine Learning models on the device using TensorFlow Lite. Since it works on the device, there is no need for an internet connection. This is a crucial requirement for remote farmers since they might not have proper internet connection in their place.</li><li><strong>Ocean Life Conservation: </strong>Smart ML-powered devices are used to monitor whales in real-time in waterways around Seattle and Vancouver to avoid whale strikes in busy shipping lanes.</li></ol><h4>How can I get started?</h4><ol><li><strong>Hardware:</strong> The <strong>Arduino Nano 33 BLE Sense</strong> is the suggested hardware for deploying Machine Learning models on edge. It contains a 32-bit ARM Cortex-M4F microcontroller running at 64MHz with 1MB of program memory and 256KB RAM. This microcontroller provides enough horsepower to run TinyML models. The Arduino Nano 33 BLE Sense also contains colour, brightness, proximity, gesture, motion, vibration, orientation, temperature, humidity, and pressure sensors. It also contains a digital microphone and a Bluetooth low energy(BLE) module. This sensor suite will be more than enough for most applications.</li><li><strong>Machine Learning Framework: </strong>There are only a handful of frameworks that cater to TinyML needs. Of that, <strong>TensorFlow Lite</strong> is the most popular and has the most community support. Using TensorFlow Lite Micro, we can deploy models on microcontrollers.</li><li><strong>Learning Resources: </strong>Since TinyML is an emerging field, there aren’t many learning materials as of today. But there are a few excellent materials like Pete Warden and Daniel Situnayake’s book, “TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power”, <a href="https://www.edx.org/professional-certificate/harvardx-tiny-machine-learning">Harvard University’s Course on TinyML by Vijay Janapa Reddi</a>, and <a href="https://www.digikey.in/en/maker/projects/intro-to-tinyml-part-1-training-a-model-for-arduino-in-tensorflow/8f1fc8c0b83d417ab521c48864d2a8ec">Digikey’s blogs and videos on TinyML</a>.</li></ol><h4>Conclusion</h4><p>Microcontrollers are everywhere, and they collect a vast amount of data. With TinyML, we can leverage this data to build better products. There are over 250 Billion microcontroller units today, and this number is only going to rise in the future. This will lead to a decrease in price(because, economics! Duh!). Enabling Machine Learning in microcontrollers will open up new opportunities.</p><h4>About Me</h4><p>I’m a student interested in AI research and development. I like to write about some lesser talked about topics in AI like Federated Learning, Graph Neural Networks, Fairness in AI, Quantum Machine Learning, TinyML, etc. Follow me to stay updated on all my future blogs.</p><p>You can find me on <a href="https://twitter.com/CleanPegasus">Twitter</a>, <a href="https://www.linkedin.com/in/arunkumar-l/">LinkedIn</a>, and <a href="https://github.com/CleanPegasus">Github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4617f314aa79" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-introduction-to-tinyml-4617f314aa79">An Introduction to TinyML</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI is Flawed — Here’s Why]]></title>
            <link>https://medium.com/data-science/ai-is-flawed-heres-why-3a7e90c48878?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a7e90c48878</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Sat, 31 Oct 2020 10:52:11 GMT</pubDate>
            <atom:updated>2020-10-31T14:47:54.175Z</atom:updated>
            <content:encoded><![CDATA[<h3>AI is Flawed — Here’s Why</h3><h4>Fairness in AI and AI bias</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xqElYYwIds8NChYEdGxH7Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@bill_oxford?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Bill Oxford</a> on <a href="https://unsplash.com/s/photos/criminal?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Artificial Intelligence has become an integral part of everyone’s lives. Starting from simple tasks like YouTube recommendations to complex life-saving tasks like generating drugs to cure illness, it has become omnipresent. It is influencing our lives in more ways than we realize.</p><blockquote>But, is AI fair? No, it definitely isn’t.</blockquote><p>It is hard to define a fair AI. Here is the best explanation I could come up with. A given AI model is fair if the outputs are independent of sensitive parameters(e.g., gender, race, sexuality, religious faith, disability, etc.) for a specific task that is already affected by social discrimination.</p><p>In this blog, I will be writing about AI bias, real-life examples of AI bias, and ways to fix it.</p><h4><strong>The Problem</strong></h4><p>AI bias is caused due to the inherent prejudice in the data used to train the model, leading to social discrimination. This leads to a lack of equal opportunities.</p><p>For instance, let&#39;s say I was tasked with creating a model to calculate a person&#39;s credit score with the location as a parameter. Certain ethnic groups are concentrated in certain locations. This will make my model racially biased towards those ethinic groups, affecting them while getting credit cards and bank loans.</p><blockquote>Biased AI models exacerbate current social discrimination and paves way for oppression.</blockquote><h4>Real-life examples of AI bias</h4><p>Here are some real-life examples of AI bias</p><ol><li><strong>COMPAS: </strong>COMPAS(Correctional Offender Management Profiling for Alternative Sanctions) is a software used by the US courts to judge the probability of a defendant(the person accused of committing a crime) becoming a recidivist(act of repeating the previously committed crime). Due to the heavily biased data, the model predicted twice as many false positives for recidivism in black offenders than white offenders.</li><li><strong>Amazon’s Hiring</strong>: In 2014, Amazon developed an AI recruiting system to streamline their hiring process. It was found to be discriminatory against women as the data used to train the model was from the past 10 years, where most selected applicants were men due to the male dominance in the tech industry. Amazon scraped this system in 2018.</li><li><strong>US Healthcare: </strong>The US healthcare system used an AI model that assigned a lower risk rate for black people than white people for the same disease. This was because the model was optimized for cost, and since black people were perceived as being less able to pay, the model ranked their health risk lower than their white counterparts. This resulted in lower healthcare standards for black people.</li><li><strong>Twitter Image Cropping: </strong>In September of 2020, Twitter users found out that the image cropping algorithm favored white faces over black faces. I.e., When an image with a different aspect ratio than the preview window is posted on Twitter, the algorithm crops parts of the image and shows only a certain portion of the image as the preview. This AI model often showed white faces in the preview window in a picture with white and black faces.</li><li><strong>Facebook’s Advertisement Algorithm: </strong>In 2019, Facebook allowed advertisers to target people based on their race, gender, and religion. This led to jobs like nursing and secretary being targeted to women, while jobs like janitor and taxi drivers targeted men, especially men of color. The model also learned that real estate ads had a better click-through rate when shown to white people, resulting in a lack of real-estate advertisements to minority people.</li></ol><p>These are just a few common examples of AI bias. There are many instances of unfair AI practices with or without the knowledge of the developer.</p><h4>How Can You Fix It?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RpdCB8ssHVk6dpHL31rKbA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@yancymin?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Yancy Min</a> on <a href="https://unsplash.com/s/photos/code?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>The first step towards a fair AI is admitting the problem. AI is imperfect. Data is imperfect. Our algorithms are imperfect. Our technology is imperfect. It is impossible to find a solution when we pretend there is no problem.</p><p>Secondly, ask yourself if this solution requires AI.</p><blockquote>Don’t be afraid to launch a product without Machine Learning — Google</blockquote><p>There are a few problems that do not depend on data. Tasks like finding the probability of recidivism in the accused depend more on emotions than data.</p><p>Third, follow responsible AI practices. I have added the points from Google’s responsible AI practices guide below.</p><p><strong>Responsible AI practices:</strong></p><ol><li><strong>Use a human-centered design approach:</strong> Design models with appropriate disclosure built-in and incorporate feedback from the testers before deployment.</li><li><strong>Identify multiple metrics to assess training and monitoring: </strong>Use different metrics appropriate for the task to understand the tradeoffs between different errors and experiences. These metrics can be feedback from the consumers, false positive and false negative rates, etc.</li><li><strong>Examine your raw data if possible: </strong>An AI model reflects the data being used to train the model. If the data is faulty, the model will be faulty as well. Try to have balanced data.</li><li><strong>Understand your model&#39;s limitations: </strong>A model trained to detect correlations does not necessarily help make causal interfaces. For example, a model might learn that people buying basketball shoes are generally taller on average, but this does not mean that a user buying basketball shoes will become taller as a result.</li><li><strong>Test: </strong>Conduct rigorous unit tests to determine faults in the model.</li><li><strong>Continue monitoring and updating your model after deployment: </strong>Consider user feedback and update your model based on that regularly after deployment.</li><li><strong>Designing a model with concrete goals for fairness and inclusion: </strong>Engage with experts from the fields of ethics and social studies to understand and account for various perspectives. Try to make your model as fair as possible.</li><li><strong>Using representative datasets to train and test your model: </strong>Try to assess the fairness of your data. I.e., look for prejudicial or discriminatory correlations between features and labels.</li><li><strong>Check for unfair biases: </strong>Get the unit tests inputs from a diverse background of testers. This can help identify which group of people might be affected by the model.</li><li><strong>Analyze performance: </strong>Take different metrics into account. An improvement in one metric might hurt another metric’s performance.</li></ol><h4>Tools for developing a Fair AI</h4><ol><li><strong>FATE: </strong>Fairness, Accountability, Transparency, and Ethics in AI(FATE), by Microsoft offers tools to assess visualization dashboards and bias mitigation algorithms. It is majorly used to compare trade-offs between fairness and performance of the system.</li><li><strong>AI Fairness 360: </strong>AI Fairness 360 is an open-source toolkit offered by IBM that helps you examine, report, and mitigate discrimination and bias in Machine Learning models.</li><li><strong>ML Fairness Gym: </strong>ML Fairness Gym is a tool offered by google for exploring the long-term impacts of Machine Learning systems concerning AI bias.</li></ol><h4>Conclusion</h4><p>Over the past few years, companies and governments have started taking AI bias seriously. Many companies have developed tools to assess AI fairness and are doing their best to fight AI bias. While AI has a huge potential, it is now more than ever important for us to keep in mind the potential discriminatory dangers of an AI system and help develop fair AI models.</p><h3>About Me</h3><p>I like to write about some lesser talked about topics in AI like Federated Learning, Graph Neural Networks, Fairness in AI, Quantum Machine Learning, TinyML, etc. Follow me to stay updated on all my future blogs.</p><p>You can find me on <a href="https://twitter.com/CleanPegasus">Twitter</a>, <a href="https://www.linkedin.com/in/arunkumar-l/">LinkedIn</a>, and <a href="https://github.com/CleanPegasus">Github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a7e90c48878" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/ai-is-flawed-heres-why-3a7e90c48878">AI is Flawed — Here’s Why</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Must-Have Tool for Every Data Scientist]]></title>
            <link>https://medium.com/data-science/a-must-have-tool-for-every-data-scientist-5e7c76f1916f?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e7c76f1916f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Wed, 21 Oct 2020 10:32:43 GMT</pubDate>
            <atom:updated>2020-10-22T03:22:32.088Z</atom:updated>
            <content:encoded><![CDATA[<h4>TensorDash — An app to remotely monitor your Machine Learning models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpyrgqwMjfclV2oN1U2VIA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Luke Chesser</a> on <a href="https://unsplash.com/s/photos/monitoring?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Let&#39;s face it; training a machine learning model is time-consuming. Even with the advancement in computing prowess over the past few years, training machine learning models takes a lot of time. Even the most trivial models have more than a million parameters. On a bigger scale, these models have over a billion parameters(GPT-3 has over 175 billion parameters!), and training these models takes days, if not weeks. As a Data Scientist, we would want to keep an eye on the model’s metrics to know if the model performs as per expectations. But it doesn’t make sense to sit next to your computer, monitoring the metrics for hours. Won’t it be nice if we could get all this data on your phone?</p><p>When was the last time you let your model train for hours and left the place, but your model had crashed halfway when you came back? This is frustrating and a complete waste of time. Since most people do not have the horsepower to train machine learning models locally, cloud services like GCP, Google Colab, AWS, and Azure are preferred where users pay as per their usage. In this case, if the model has crashed in-between training, you are paying for the service without using it. It will help you avoid FOMO if you are notified about your model status.</p><h4>TensorDash</h4><p>TensorDash is a free to use open-source app that lets you remotely monitor your machine learning models through detailed graphs of all your metrics. It also notifies you when the training has been completed or if the model has crashed. It supports all the major machine learning frameworks like TensorFlow, Pytorch, Keras, and Fastai.</p><p>Now you don’t have to be stressed sitting in front of your computer, praying for the accuracy to increase. You can do that from the comfort of your home.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yaielDm2rwnr49698gerLg.png" /><figcaption>TensorDash Sample (Image by Author)</figcaption></figure><h4>Using TensorDash</h4><ol><li>Install the <a href="https://play.google.com/store/apps/details?id=tech.tensordash.tensordash&amp;hl=en_AU">TensorDash</a> app from the play store. Support for iOS devices is coming soon.</li><li>Create an account.</li><li>Install the TensorDash python package using the command pip install tensor-dash.</li><li>Follow the instructions below for using TensorDash with your respective framework.</li></ol><h4>Keras/tf.keras Support</h4><p>Remote monitoring with Keras and tf.keras works using the callback function. Import Tensordash. Define a TensorDash object using the model name, your account email ID, and password as the parameters. Pass the TensorDash object via callbacks in the fit() function. To get notified if your model crashes, add the fit() function under exception handling, and in the except, call the sendCrash() method.</p><p><strong>Below is the sample code for using TensorDash on Keras/tf.keras</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e42ecdc03ca253040e13dc1c25ca0c1/href">https://medium.com/media/0e42ecdc03ca253040e13dc1c25ca0c1/href</a></iframe><h4>PyTorch Support</h4><p>Import Torchdash from the Tensordash package. Create a Torchdash object with the model name, email ID, and password as the parameters. Use the sendLoss() method of the object inside the training loop to send the model metrics to your account. Pass the loss, accuracy, validation loss, and validation accuracy metrics as parameters(Note that you have to add at least one metric). To get notified if your model crashes, add the training loop under exception handling, and in the except, call the sendCrash() method.</p><p><strong>Below is the sample code for using TensorDash on PyTorch</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0559f72400088886f428c2f3921cdf52/href">https://medium.com/media/0559f72400088886f428c2f3921cdf52/href</a></iframe><h4>Fast.ai Support</h4><p>Remote monitoring with fast.ai works through callbacks. Import Fastdash. Define a Fastdash object using the model name, your account email ID, and password as the parameters. Pass the Fastdash object via callbacks in the fit() function. To get notified if your model crashes, add the fit() function under exception handling, and in the except, call the sendCrash() method.</p><p><strong>Below is the sample code for using Fast.ai</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f216f488526f0a83eaf7b65fb30b6dc/href">https://medium.com/media/4f216f488526f0a83eaf7b65fb30b6dc/href</a></iframe><h4>TensorFlow Support</h4><p>Import Customdash from the Tensordash package. Create a Customdash object with the model name, email ID, and password as the parameters. Use the sendLoss() method of the object inside the training loop to send the model metrics to your account. Pass the loss, accuracy, validation loss, and validation accuracy metrics as parameters(Note that you have to add at least one metric). To get notified if your model crashes, add the training loop under exception handling, and in the except, call the sendCrash() method.</p><p>Note that Customdash can be used with any framework where you specify a custom training loop.</p><p><strong>Below is the sample code for using TensorDash on a TensorFlow custom loop</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/244645f708b50aecd341a1d6370e24ae/href">https://medium.com/media/244645f708b50aecd341a1d6370e24ae/href</a></iframe><h4>Conclusion</h4><p>TensorDash is completely free and open-source. Issues and contributions are welcome. Check out the repository <a href="https://github.com/CleanPegasus/TensorDash">here</a> and the detailed docs <a href="https://cleanpegasus.github.io/TensorDash/">here</a>.</p><p><a href="https://medium.com/u/7992c83feb72">Harshit Maheshwari</a> and I built this tool to solve our monitoring problems during AI research. I hope this helps you just as well as it helped us.</p><h4>About Me</h4><p>I like to write about some lesser talked about topics in AI. Follow me to stay updated.</p><p>You can find me on <a href="https://twitter.com/CleanPegasus">Twitter</a>, <a href="https://www.linkedin.com/in/arunkumar-l/">LinkedIn</a>, and <a href="https://github.com/CleanPegasus">Github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e7c76f1916f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-must-have-tool-for-every-data-scientist-5e7c76f1916f">A Must-Have Tool for Every Data Scientist</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Graph Neural Networks (Part 1)]]></title>
            <link>https://medium.com/data-science/an-introduction-to-graph-neural-networks-part-1-57335bca9a79?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/57335bca9a79</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Wed, 14 Oct 2020 10:02:53 GMT</pubDate>
            <atom:updated>2020-10-14T12:39:22.423Z</atom:updated>
            <content:encoded><![CDATA[<h4>Graph Neural Networks and their applications</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YECeOxlko9KoOJNw8RNm3A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@alinnnaaaa?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Alina Grubnyak</a> on <a href="https://unsplash.com/s/photos/network?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Over the past three years, Graph Neural Networks(GNN) has grown exponentially in popularity. This is due to their ability to learn from graph data efficiently. There is an intricate and cohesive network connection behind most systems that defines the interaction between the network components. This interaction can be represented as a graph. To model these systems, it is essential to understand the network behind them. Graph Neural Networks help us take advantage of the network&#39;s relational structure to model it better and predict the outcome.</p><h4><strong>What is a Graph?</strong></h4><p>A graph is a structured non-linear datatype with nodes(also called vertices) and edges. It is mathematically represented as G(V, E). For example, a molecule of NO₂ can be considered a graph where the nitrogen atom and two oxygen atoms are considered the node, and the bonds between the atoms are considered to be edges. Another example of a graph can be your family, where each person is a node, and the relationship between two people is the edge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*6kCXfcSSmUrbOmr5T7mJgw.png" /><figcaption>NO₂ as a graph (Image by Author)</figcaption></figure><p>A graph can have labels on both its nodes and edges. It can either be numerical or textual. Each node will have some features that define the node. In a graph of NO₂, elements like the atomic number, atomic weight, and the number of valency electrons of each node can be its respective features. The edges might or might not have features depending on the type of graph. In NO₂, the edges&#39; features can be the bond strength, bond type(single bond or double bond), etc.</p><p>Graphs are classified on many different bases. The most common one is based on the edges of the graph. These are directed and undirected graphs. In a directed graph, the edges from one node to another are directional, while in an undirected graph, the nodes are connected through edges, and no direction is present.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*Nh9PPYO-G3dkCCdTJ7Fayw.png" /><figcaption>Directed Graph (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*mxs1PfMxE1SPd14wzqrFcQ.png" /><figcaption>Undirected Graph (Image by Author)</figcaption></figure><p>A practical example of a directed graph is Instagram. When you follow someone, they don&#39;t necessarily follow you back. In a sense, this is unidirectional. On the other hand, Facebook friend requests are an example of an undirected graph. Once the friend request is accepted, both of you can see each other’s content.</p><h4><strong>Examples of Graphs</strong></h4><ol><li><strong>Social Networks: </strong>A social network is a graph where nodes represent people, and the relationship between two people is the edge. This relationship can be anything ranging from a simple acquaintance to a family.</li><li><strong>Molecules</strong>: A molecule can be represented as a graph where the nodes represent the atoms, and the edges represent the bond between them.</li><li><strong>Internet</strong>: The internet is a graph where the devices, routers, website, and the servers are the nodes and the internet connections are the edges.</li></ol><h4><strong>Analyzing Graphs (Elaborate)</strong></h4><ol><li><strong>Node Classification </strong>— Predict the type of a given node. For example, a given person in a social network can be classified based on their interests, beliefs, or characteristics.</li><li><strong>Link Prediction </strong>— Predict if and how two nodes are linked. For example, finding if two given people(nodes) have any relationship between them.</li><li><strong>Community Detection</strong> — Identify densely linked clusters of nodes. For example, finding if a group of people have any topic similarities.</li><li><strong>Network Similarity</strong> — Measure the similarity of two nodes/networks. Here you can find if two people or two different groups of people are similar to one another.</li></ol><h4><strong>Some applications of Graphs Neural Networks</strong></h4><ol><li><strong>Recommendation systems</strong>: The abilities of a recommendation system can be increased exponentially using GNNs. With GNNs, the recommendations will be based on borrowing information from the nearby nodes, thus making the node embedding more accurate. Pinterest uses a GNN based recommender system.</li><li><strong>Tackling misinformation</strong>: A hoax article can be identified using its links. Real articles link more coherently, while fake ones will have loose ends. According to a 2016 paper titled “Disinformation on the web,” humans can detect a hoax article 66 out of 100 times while a GNN can detect it 86 out of 100 times.</li><li><strong>Drug development</strong>: All molecules can be represented as a graph. Using GNNs, it is possible to model complex networks like protein-protein interaction(PPI) networks and metabolic networks. This model helps in developing better and stable drugs for diseases.</li><li><strong>Polarization on Twitter</strong>: Depending on the posts a person likes and people they follow, it is possible to detect if a person is polarized towards a specific view on a topic(political, environmental, etc.) or not.</li><li><strong>Social Circle detection: </strong>Using GNNs, it is possible to detect a person&#39;s social circle based on his interaction with others. This circle can be co-workers, college friends, family members, people in the same class, etc.</li></ol><p><strong>Why can convolutions not be applied to graphs?</strong></p><p>Images have a fixed size and grid-based structure data with a defined spatial locality. On the other hand, graphs have arbitrary sizes, complex topology, and a non-euclidean structure. It also does not have a fixed node order. As we all know, neural networks are defined for specific sizes, grids, and structures. Thus convolutions cannot be directly applied to graphs.</p><h4>Conclusion</h4><p>I’ll write about transforming the graphs into matrices like the Adjacency matrix, Incidence matrix, Degree matrix, and Laplacian matrix in the upcoming blogs. Further, I’ll discuss different approaches to using convolutions on graphs and their advantages and disadvantages, creating a node embedding of the graph, and forward propagation in GNN.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57335bca9a79" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-introduction-to-graph-neural-networks-part-1-57335bca9a79">An Introduction to Graph Neural Networks (Part 1)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Federated Learning]]></title>
            <link>https://medium.com/data-science/understanding-federated-learning-99bc86a0d026?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/99bc86a0d026</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Tue, 06 Oct 2020 13:13:41 GMT</pubDate>
            <atom:updated>2020-10-06T16:45:00.346Z</atom:updated>
            <content:encoded><![CDATA[<h4>A focus on privacy in Machine Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*J7lJxuE8Ruy_Zpgg0VaG3A.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/madartzgraphics-3575871/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1944688">Darwin Laganzon</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1944688">Pixabay</a></figcaption></figure><p>With an increasing focus on privacy, Federated Learning has become one of the essential concepts in modern machine learning. Federated learning is geared towards training a model without uploading personal information or identifiable data to a cloud server. As you might already know, a machine learning model needs a lot of data to train. But there are times when the training data is sensitive, and people are growing reluctant to share their personal data with a third party. With growing concerns for privacy, federated learning is now essential in most machine learning applications.</p><blockquote>Data is born at the edge</blockquote><p>Over a billion edge devices like phones, tablets, and IoT devices worldwide continually generate data. For companies and developers, this data can make their product, and the user experiences better by training a better model. Often, the client sends data to the server where the model runs inference and returns the prediction. Once the model returns the prediction, the client sends feedback, using which the model corrects itself. While this presents an advantage in collecting the data and reducing the computational strain on the edge device, it faces problems regarding offline usage, latency, and, more importantly, privacy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*WxVuYdNzlUIjfiovkrNERA.png" /><figcaption>Model Inference on the cloud. Image by the Author</figcaption></figure><p>While the model can be trained and inferred locally, it too has its limitations. A single user will provide too little data for a model to be adequately trained. In this case, the other devices&#39; data aren’t contributing, which leads to having a non-generalized data. It has become a fight between privacy and better intelligence with almost no solutions until recently.</p><p><strong>How does Federated Learning work?</strong></p><p>In federated learning, the server distributes the trained model(M1) to the clients. The clients train the model on locally available data. These models are then sent back to the server instead of the data, where they are averaged to produce a new model(M2). This new model(M2) now acts as the primary model and is again distributed to the clients. This process is repeated until the model achieves a satisfactory result. In every iteration, the model gets a little better than it already was. Thus, federated learning gives birth to better intelligence, while the user&#39;s personal data is secure in their device.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*GjjtEeO4OmzQ5RPx-OiDbA.png" /><figcaption>Federated Learning. Image by the Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*ryfwiLVxzLpT5_Q0xjQHXQ.jpeg" /><figcaption>Gboard predicting my next word.</figcaption></figure><p>The most common example of federated learning is Google’s keyboard app, Gboard. Machine learning models improve user experience like swipe typing, auto-correct, next-word predictions, voice to text, etc. Federated learning plays a huge role here because what you type is very personal to you, and you wouldn’t want to send your data to a server. The local model is trained with your data and sent to the server, like numerous other models from different clients. The server takes the average of these new models to produce a new model. The server now distributes the latest model to the clients. This process repeats forever.</p><p><strong>Security Protocols</strong></p><p>Even though the user data is not uploaded to the server, there is potential for the model to be reverse engineered to obtain user data. Model aggregation and client-side data encryption are used to combat this problem.</p><p>The Federated learning protocol combines and sums the model output, and the server has access only to the aggregate model and not the individual models. Here, the devices report only the data that is required for the calculation. The server distributes this model to all the clients.</p><p>Masking is used to cancel out opposite data points from the clients during aggregation. Since the masked value is sent to the server, it is difficult for anyone to intercept the values and reverse-engineer any personal data.</p><p><strong>TensorFlow Federated</strong></p><p>TensorFlow Federated(TFF) is an open-source framework for federated learning on decentralized data made by Google’s TensorFlow team. TFF is still in its infancy and has a lot to improve. At the time of writing, TFF only provides local simulation runtime and no options for deployment.</p><p>You can install TensorFlow Federated using the pip package manager.</p><p>pip install tensorflow_federated</p><p>TensorFlow Federated offers two sets of interfaces, namely, Federated Learning(FL) and Federated Core. Using the federated learning interface, the developer can implement federated training or federated evaluation, and developers can apply federated learning to existing TensorFlow models. The federated core interface is used to test and express new federated algorithms and run local runtime simulations.</p><p>I will write more on TensorFlow Federated in an upcoming blog.</p><p><strong>Conclusion</strong></p><p>This blog is a quick rundown on federated learning, and I will publish more in-depth blogs soon. I’m planning to write more blogs on some lesser-known concepts of Machine Learning. Follow me to get the latest updates.</p><p>You can find me on <a href="https://twitter.com/CleanPegasus">Twitter</a>, <a href="https://www.linkedin.com/in/arunkumar-l/">LinkedIn</a>, and <a href="https://github.com/CleanPegasus">Github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99bc86a0d026" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/understanding-federated-learning-99bc86a0d026">Understanding Federated Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with TensorFlow Serving]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/getting-started-with-tensorflow-serving-b03c130bdb5c?source=rss-ad035839866------2"><img src="https://cdn-images-1.medium.com/max/2600/1*WSEePtBQ5Zwgfupi2dQTAw.jpeg" width="2968"></a></p><p class="medium-feed-snippet">TensorFlow Serving is a part of TensorFlow Extended(TFX) that makes deploying your machine learning model to a server more comfortable&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/data-science/getting-started-with-tensorflow-serving-b03c130bdb5c?source=rss-ad035839866------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/getting-started-with-tensorflow-serving-b03c130bdb5c?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/b03c130bdb5c</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Mon, 28 Sep 2020 10:18:41 GMT</pubDate>
            <atom:updated>2020-09-28T21:57:23.273Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Quantum Gates-Explained]]></title>
            <link>https://medium.com/incode-lab/quantum-gates-explained-e79a20ab2b14?source=rss-ad035839866------2</link>
            <guid isPermaLink="false">https://medium.com/p/e79a20ab2b14</guid>
            <category><![CDATA[quantum]]></category>
            <category><![CDATA[quantum-computing]]></category>
            <category><![CDATA[quantum-physics]]></category>
            <dc:creator><![CDATA[Arun]]></dc:creator>
            <pubDate>Sun, 08 Dec 2019 12:21:32 GMT</pubDate>
            <atom:updated>2019-12-08T12:21:32.375Z</atom:updated>
            <content:encoded><![CDATA[<p>Quantum computing can be hard and intimidating in the beginning. But it will get easier with practice. With this article, I’m hoping to get you started with quantum computing with ease. First, you have to understand qubits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tDSaGSaCu2BJJojM8nqGpQ.jpeg" /><figcaption>credits: roche.com</figcaption></figure><p><strong>Classical bit vs Qubit</strong></p><p>In classical computers, the bit values can either be 0 or 1. In the case of a quantum computer, the “qubits” can take the coherent superposition of both 0 and 1 simultaneously. The qubit is represented with the probabilities of it being a 0 and 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*oeb1yslqmqHLsXgzO1evpQ.jpeg" /><figcaption>credits: researchgate.net</figcaption></figure><p>This property of qubit is what makes quantum computers exponentially faster than a classical computer. For a given number of bits(n), a quantum computer will perform a calculation 2^n times faster than a classical computer.<br>Example: With 3 qubits of data, the Quantum computer can store all the 8 possible combinations of 0 and 1 simultaneously. As a result, 4 Qubit based Quantum computer does calculations 16 times faster compare to a 4-bit classical computer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*B5MKeASbOH3GJnubfLXR6g.png" /></figure><p>Now let’s take a look into quantum gates.</p><p><strong>Hadamard Gate</strong></p><p>The Hadamard gate is the most common and the most used quantum gate in quantum computing. It is a single qubit operation that maps the basis state ∣0＞ to ( |0＞ + |1＞)/√2 and ∣1＞ to ( |0＞ — |1＞)/√2, thus creating an equal superposition of the two basic states. This means the measurements will have equal probabilities to become 1 or 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*v1a2gmjJALXYFfbPweV1lg.png" /><figcaption>credits: researchgate.net</figcaption></figure><p><strong>Pauli-X gate</strong></p><p>The Pauli-X gate is a single qubit operation gate that is similar to the classical not gate. It takes a value and flips it to the opposite one. It maps |0＞ to |1＞ and |1＞ to |0＞.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*XMQ4-FoRR8LOih7EZ6_3LA.png" /><figcaption>credits: MSDN Blogs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/1*ACiT6ef43BqKZmJfWuRbiA.png" /></figure><p><strong>Pauli-Y gate</strong></p><p>The Pauli-Y gate is a single qubit operation. It maps |0＞ to -i|1＞ and |1＞ to i|0＞. It equates to a rotation around the Y-axis of the Bloch sphere by pi radians.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/1*N5EeobQLFsQwoQT_pfapZg.png" /></figure><p><strong>Pauli-Z gate</strong></p><p>The Pauli-Z gate is a single qubit operation. It maps |1＞ to -|1＞ and it leaves |0＞ unchanged. It equates to a rotation around the Z-axis of the Bloch sphere by pi radians.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/1*R6A1P06ZmGd9PXmm_lwk5g.png" /></figure><p><strong>Controlled gates</strong></p><p>The controlled gate is a 2 or more qubit operation, where 1 or more qubits act as a control for some operation on a qubit. (eg: cx, cy, cz gates)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/1*P_OZjvLezKYHhlSQwKN5_A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/1*33EXEzgKcTvFzkknXlVpHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/1*QLQ0Wl2A6FYO_-VD16_fCA.png" /><figcaption>cx, cy, and cz gates</figcaption></figure><p><strong>Controlled X gate</strong></p><p>The controlled-x gate acts on 2 qubits and performs the NOT operation on the second qubit only when the first qubit is |1＞.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/1*P_OZjvLezKYHhlSQwKN5_A.png" /></figure><p><strong>Conclusion</strong></p><p>I have attached some links to learn quantum mechanics and quantum computing. I hope it helps.</p><p>Quantum Computing: <a href="https://www.youtube.com/playlist?list=PLOFEBzvs-Vvp2xg9-POLJhQwtVktlYGbY">https://www.youtube.com/playlist?list=PLOFEBzvs-Vvp2xg9-POLJhQwtVktlYGbY</a></p><p>Quantum Mechanics: <a href="http://ocw.mit.edu/8-04S16">http://ocw.mit.edu/8-04S16</a></p><p>Github: <a href="https://github.com/CleanPegasus/Quantum_Computing">https://github.com/CleanPegasus/Quantum_Computing</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e79a20ab2b14" width="1" height="1" alt=""><hr><p><a href="https://medium.com/incode-lab/quantum-gates-explained-e79a20ab2b14">Quantum Gates-Explained</a> was originally published in <a href="https://medium.com/incode-lab">InCode Lab</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>