<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Satish VT on Medium]]></title>
        <description><![CDATA[Stories by Satish VT on Medium]]></description>
        <link>https://medium.com/@satish.vt?source=rss-c460585cbaec------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Satish VT on Medium</title>
            <link>https://medium.com/@satish.vt?source=rss-c460585cbaec------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:18:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@satish.vt/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chapter 3: Transaction Process and Lifecycle Management]]></title>
            <link>https://medium.com/@satish.vt/chapter-3-transaction-process-and-lifecycle-management-182039af12f4?source=rss-c460585cbaec------2</link>
            <guid isPermaLink="false">https://medium.com/p/182039af12f4</guid>
            <category><![CDATA[payments-technology]]></category>
            <category><![CDATA[payment-tokenization]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[network-tokenization]]></category>
            <category><![CDATA[payments]]></category>
            <dc:creator><![CDATA[Satish VT]]></dc:creator>
            <pubDate>Thu, 09 Jul 2020 18:22:26 GMT</pubDate>
            <atom:updated>2020-07-09T18:22:26.072Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/@satish.vt/payment-tokenization-chapter-1-data-obfuscation-types-tokens-and-types-73777f3bdf24">last</a> <a href="https://medium.com/@satish.vt/chapter-2-tokens-as-payment-instruments-concept-stakeholders-token-issuance-process-8e9a46f8d916">two</a> blog posts, basics of Payment Tokenization, Token Issuance Process. In this last chapter, i intend to cover the Transaction Process for Token based transactions and how Life cycle events are handled.</p><h3>Transaction Process — Tokens</h3><p>Mobile NFC based — Tokenized transactions, work very similar to that of normal EMV Chip Contactless card, with few changes at the scheme layer. Again for ease of understanding, I’ve considered TSP to be Scheme itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*yPfkVVcXgHBqNqDkfnJX4g.png" /><figcaption>Tokenized Transacton Flow</figcaption></figure><p>The flow is as follows,</p><p><strong>Step 1: </strong>Customer opens the Mobile app, selects the card profile and taps over the POS. The interaction is very similar to that of EMV Chip Card Contactless transaction. The same protocol, and command exchange is performed between Mobile and the POS</p><p><strong>Step 2</strong>: Acquirer receives the message from the POS.</p><p><strong>Step 3: </strong>Acquirer forwards the message to Scheme</p><p><em>For Tokenized Transactions, there is no/very minimal change to Acquiring infrastructure i.e. POS or Acquiring switching infrastructure.</em></p><p><strong>Step 4:</strong> Scheme receives the Tokenized Transaction, and performs the following,</p><ul><li>Perform the cryptogram validation. Scheme/TSP has the EMV keys for the cryptogram validation for the Token transaction</li><li>Performs de-tokenization, passes the original PAN number (basically DE2 will real PAN), EMV cryptogram validation, token number</li></ul><p><strong>Step 5:</strong> The updated message (with real PAN), is passed to Issuer</p><p><strong>Step 6: </strong>Issuer performs standard authorization validations and responds to scheme</p><p><strong>Step 7:</strong> Scheme replaces the real PAN number with Token details</p><p><strong>Step 8: </strong>Scheme passes the results to Acquirer</p><p><strong>Step 9: </strong>Acquirer further passes the details to POS, which further final decision process.</p><h3>Life cycle Management</h3><p>As we all know that, Token(s) are tied with a given PAN. In the lifetime of a given PAN, there can be many changes to a given PAN like, reissue of a PAN, PAN blocking, Account closure. Any such changes, should also result in corresponding change to Token. for example, When a PAN is lost, and is further blocked. All the associated Tokens should also be blocked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*Fosyfp-JowZ06sT8rlWQ_A.png" /><figcaption>Token Life-cycle updates — Issuer end</figcaption></figure><p>Step 1: Issuer intimates the Schemes for any update on the PAN. It could be a real-time update via an API, or via a portal, or an ISO8583 message or by a batch feed</p><p>Step 2: Token Service Provider, retrieves the Token(s) attached to the given PAN, from the Token vault and updates the status based on the information received from Issuer</p><p>Step 3: Corresponding Token status updates are pushed to the Token device.</p><p><em>Similarly, any updates initiated from the mobile device, or the Token requestor will also be propagated back.</em></p><p><strong>Concluding Remarks:</strong></p><ul><li>In the case of Tokenized transactions, there’s no much change to the Acquiring infrastructure</li><li>Apart from the PAN/Token number, Schemes also maintain a separate identifier called as PAR (Payment Account Reference)</li><li>Schemes/TSP does the key aspects of cryptogram validation and passes the information to Issuer (like a on-behalf transaction)</li><li>Life cycle events at Issuer end are pushed to TSP which are further pushed to Mobile device.</li></ul><p>This concludes the (final) Chapter 3 .</p><p>— Thank You!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=182039af12f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chapter 2: “Tokens as Payment Instruments” — Concept, Stakeholders & Token Issuance Process]]></title>
            <link>https://medium.com/@satish.vt/chapter-2-tokens-as-payment-instruments-concept-stakeholders-token-issuance-process-8e9a46f8d916?source=rss-c460585cbaec------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e9a46f8d916</guid>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[payment-tokenization]]></category>
            <category><![CDATA[network-tokenization]]></category>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[payments-technology]]></category>
            <dc:creator><![CDATA[Satish VT]]></dc:creator>
            <pubDate>Wed, 08 Jul 2020 20:26:47 GMT</pubDate>
            <atom:updated>2020-07-10T13:23:46.324Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chapter 2: “Tokens as Payment Instruments” — Concept, Stakeholders &amp; Token Issuance Process</h3><p>This blog post is a sequel to my <a href="https://medium.com/@satish.vt/payment-tokenization-chapter-1-data-obfuscation-types-tokens-and-types-73777f3bdf24">previous post</a> on Payment Tokenization</p><p>Traditional card payment transactions involve PAN for transactions. PAN is displayed on the physical plastic (embossed). It is also stored inside the EMV Chip application. (EMV Chip application is a payment application that resides on the chip on the plastic, which enables the payment process).</p><p>To avoid the exposure of PAN, a surrogate value of Token is created. A unique Token is created for every transacting device i.e. if a PAN holder, has a Samsung Pay, Apple Pay, then there would be 2 tokens for the same PAN.</p><p>This article aims to provide how Tokens are used as Payment Instruments.</p><p><strong><em>Note: </em></strong><em>Some of the details provided may appear abstracted OR expects the reader to have some background of EMV. Should you have any questions. Do a comment, and i shall try to respond.</em></p><h4>Typical EMV Card flow</h4><p>Traditionally the Issuer Banks sends a Card Personalization file which consists of the PAN number, demographic details, EMV chip application details. Card Personalization factory and Issuer Bank agree on the keys, risk parameters, CAM/CVM methods that are to be populated on the EMV chip.</p><ol><li>Card Personalization/Printing vendor prints the physical plastic with the corresponding EMV payment applications, and mails the final transacting device i.e. an EMV chip card to the customer</li><li>Customer gets the physical plastic and performs a transaction at point-of-sale (POS).</li><li>POS builds a transaction message typically an ISO8583 and sends it to Acquirer bank.</li><li>Acquirer Bank does pre-edits on the message and forwards to the schemes</li><li>Schemes does its own pre-edits and forwards the message to Issuer Bank</li><li>Issuer Bank sends the response to Schemes</li><li>Schemes forwards the responds to Acquirer Bank</li><li>Acquirer Bank forwards the message to POS. POS evaluates the response and updates the EMV chip as required and processes the final responses.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*fy0eG4emsE_kzdjxDBabqw.png" /><figcaption>Typical EMV Card Transaction Flow</figcaption></figure><h4>Advent of NFC on mobile</h4><p>Traditionally, EMV Card was THE predominant transacting device provided by banks, with a well-defined POS acceptance infrastructure.</p><p>Thanks to advent of NFC technology on Mobile (especially Android 4.4 — KitKat HCE technology, Secure Element based payments), Mobiles could become a transacting device like a EMV Chip card.</p><p>Now the question arises, How could the mobile be “personalized” like a EMV Chip Card.</p><p>Earlier PAN was only available with the customer earlier in the physical plastic, and if PAN is made available across all transacting device, there is a risk of exposure of PAN and transacting devices can’t be managed easily.</p><p><strong>Welcome to “Tokens”, </strong>Tokens were designed as alternate for PAN.</p><h3>Tokenization Ecosystem</h3><p>Before we start into Token Issuance process, lets look into various entities in the Token Issuance Process.</p><p>I have taken Mobile Payment Application as the end transacting device. However the end transacting device need not necessarily a mobile. It could be a smartwatch, a key for Tokenization.</p><p><strong>Mobile Application:</strong></p><p>End mobile payment application which would be used by the customer as a transacting device. Mobile payment application can be SE (Secure element) based payments (like Apple Pay) or standard-HCE based payments (like Samsung Pay). These payment applications allow the consumers to add their payment card for transacting.</p><p><strong>Token Requestor (TR) / Wallet Server:</strong></p><p>This is the server component of the Mobile Application. TR can be a bank wallet application, or an OEM payment application. TR takes care of providing the tokens to the end application.</p><p>A Token Requestor should register with the Token Service Provider (TSP). TSP provides each Token Requestor with a unique code. TSP liaises with TSP for any Token provisioning, and any other life cycle management activities.</p><p><strong>Token Service Provider (TSP):</strong></p><p>TSP is the most important entity in the eco-system of Tokenization, which performs majority of heavy-lifting given below. Each TSP has to register with EMVCo, and would be assigned a unique code.</p><p><strong><em>Schemes also act as a TSP, in majority of the implementations. Hence in this post, i have considered the same for providing the overview and examples.</em></strong></p><ul><li><strong>Issuer Interfaces:</strong></li></ul><pre>* Issuers onboard with TSP<br>* TSP holds the mapping between Real and PAN Ranges<br>* TSP maintains a Token Vault and maintains the life-cycle of the <br>  Token<br>* TSP has an issuer interface to receive any life-cycle events from     <br>  Issuer (PAN block, re-issue etc.)<br>* TSP uses this interface to push any messages to Issuer (say during <br>  Token Assurance etc.)</pre><ul><li><strong>EMV Data Prep: </strong>In a physical plastic EMV card, card personalizations generates and populates on the EMV chip. In case of a Tokenization, TSP generates the EMV data and transmits the data to the Mobile. TSP holds the necessary keys and card profiles to generate the EMV personalization data.</li><li><strong>Token Requestor Interface: </strong>Needless to say, TSP maintains an interface with TR to manage the lifecycle of Tokens to the end device.</li><li><strong>Transaction Processing: </strong>TSP maintains an interface with the schemes. Every transaction performed using a Token. TSP performs the cryptogram validations, de-tokenizes the transaction to be further passed to Issuer. This is further detailed in subsequent sections</li></ul><h3>Tokenization Issuance Process</h3><p>Overall Tokenization Issuance process is split into 4 steps</p><ol><li>Token Assurance: Is the process, of validating if the person adding the PAN is the rightful owner of the PAN.</li><li>Token Generation: This is the process of generating a Token number and Token Expiry date for a given PAN number</li><li>Token Issuance: Issuance is the process of generating all associated data for a Token like the personalization data, including keys (say EMV data)</li><li>Token Provisioning: Is the process of delivering Token and associated data to the Token Location i.e. Mobile Payment Application</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*max-GR6Sc2dgyEs2JYj0Hw.png" /><figcaption>Token Issuance Process</figcaption></figure><p><strong>Step 1: </strong>Customer enters the PAN details (PAN, Expiry Date, CVC, Name) OR Customer scans the card</p><p><strong>Step 2 and 3: </strong>PAN details are passed to the TSP/Scheme which are further passed to Issuer for validation</p><p><strong>Step 4: </strong>Issuer sends the validation result to the TSP</p><p><strong>Step 5: </strong>Based on the policies set-up for additional authentication a.k.a Step-up process which is a part of the overall ID&amp;V process (Identification &amp; Verification process) at the TSP end, additional authentication is requested. Additional authentication can be in the form of OTP which the customer would need to validate.</p><p><em>Steps 1–5 form a part of the Token Assurance Process.</em></p><p><strong>Step 6: </strong>The TSP generates the Token number, expiry date and other EMV data. This is a part of the Token Generation and Token Issuance Process</p><p><strong>Step 7 &amp; 8: </strong>This is the Token Provisioning process, where the Token data is delivered to the client.</p><h4>Concluding Remarks:</h4><ul><li>In this post, have covered the overall flow for Token Issuance. As mentioned above, i have written this post in a way that Schemes act as a TSP, which is generally the case, however not mandatory</li><li>End Token Consuming/Requesting application, need not be always mobile. However for ease of understanding i have taken mobile as an example</li><li>In the next post, I’ll cover how a typical Token based transaction works and LCM (Life Cycle Management) works.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e9a46f8d916" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Payment Tokenization — Chapter 1 Data Obfuscation types, Tokens and Types]]></title>
            <link>https://medium.com/@satish.vt/payment-tokenization-chapter-1-data-obfuscation-types-tokens-and-types-73777f3bdf24?source=rss-c460585cbaec------2</link>
            <guid isPermaLink="false">https://medium.com/p/73777f3bdf24</guid>
            <category><![CDATA[payment-tokenization]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[payments-technology]]></category>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[network-tokenization]]></category>
            <dc:creator><![CDATA[Satish VT]]></dc:creator>
            <pubDate>Sat, 04 Jul 2020 21:07:50 GMT</pubDate>
            <atom:updated>2020-07-09T18:42:31.649Z</atom:updated>
            <content:encoded><![CDATA[<h3>Payment Tokenization — Chapter 1 Data Obfuscation types, Tokens and Types</h3><p>The word “Tokenization” or “Tokens” has different meanings in different domains,</p><ul><li>For an Information security professional, it’s more of obfuscation of data i.e. making unreadable ensuring no business loss due to breach of data</li><li>In the field of NLP, it’s all about breaking sentences into words, sub-words, characters</li><li>In Crypto-currency, “tokens” are crypto assets are special virtual currency assets</li></ul><p>In this blog series, I publish about “Payment Tokenization” (a.k.a Network Tokenization) related to Payments i.e. where Tokens are used as “Payment Instruments”.</p><p>This blog post is the first of 3 posts in this blog series,</p><ul><li>Chapter 1: Data Obfuscation types, Tokens and Types</li><li><a href="https://medium.com/@satish.vt/chapter-2-tokens-as-payment-instruments-concept-stakeholders-token-issuance-process-8e9a46f8d916">Chapter 2: “Tokens as Payment Instruments” — Concept, Stakeholders &amp; Token Issuance Proces</a>s</li><li><a href="https://medium.com/p/182039af12f4?source=post_stats_page---------------------------">Chapter 3: Transaction Process and Lifecycle Management</a></li></ul><p><strong><em>Start of Chapter 1</em></strong></p><h3>Introduction:</h3><p>Tokens are used as Payment Instruments to hide the underlying PAN.</p><ul><li>Payment industry which started with physical plastic cards are now having mobile device or watch as their transacting device.</li><li>To ensure that the PAN is not exposed to all those transacting devices. Tokens are used as a surrogate values and as a replacement of PAN</li><li>Even if one device gets compromised, the Issuer of the Card need not replace the PAN. Just block the given Token.</li><li>So one PAN can have multiple Tokens</li></ul><p>Tokens are a form of data obfuscation, and in this post i introduce you to basics/types of encryption and types of Tokens and what type of Tokens do we use in Payment Tokens.</p><h3>Section 1: Data Obfuscation types</h3><p><strong><em>Why do we need to obfuscate:</em></strong> Assume you have very sensitive data say “Credit card information”, “Customer Personal Information” stored in a file and the file/storage gets exposed. Such data breaches intentional and accidental can lead to various consequences like revenue loss, reputation loss etc.</p><h4><strong>Methods of “Obfuscation”:</strong></h4><h4>Method 1: “Encryption”</h4><p>Encryption is a technique where the data is converted into an “unreadable format OR make data useless (encrypted)” using typically complex cryptographic algorithms. This process is called “Encryption”. The process of retrieving back the data from “obfuscated” data is called as “Decryption”. This method is a “<strong>reversible</strong>” process, where data is encrypted and retrieved back.</p><p>Encryption (cryptography) can be broadly categorized into two types, “Symmetric” and “Asymmetric”.</p><p>Symmetric is where the sender and recipient of the data share a common (same) key; Data Sender encryption obfuscates data with the key, and any recipient with the same key would be able to decrypt and restore the original data. Algorithms like DES, 3DES, AES are symmetric. <em>Payment Card PIN, CVC etc use Symmetric key cryptography.</em></p><p>Asymmetrical encryption is where the data sender would encrypt with a public key, which is available to any sender, and only the recipient with private key would be able to decrypt and restore the original data. This is also called as a PKI (public key infrastructure). Algorithms like RSA, EC, D-H are asymmetric.</p><p>Data-on-Rest OR Data-on-Motion are encrypted using the aforementioned encryption techniques. Data-on-Rest refers to data that is stored in files, databases, or any other file-storages. Data-on-Motion refers to data that is being transmitted.</p><p>In any system where PII (Personally Identifiable Information) OR any other sensitive information is present, Encryption is used.</p><p><em>I am not delving into encryption methods in this blog post, it is “vast” topic and i would leave it to your research.</em></p><h4>Method 1.1: “Tokenization”</h4><p>I call “Tokenization” also as a subset of “Encryption”. Tokenization is also a reversible process. In Payment systems, Tokenization obfuscates the PAN number with a surrogate Token number. Like Encryption, Tokens can be generated using other randomizer algorithms. In the case of Randomizers, the Tokens are stored in a Token vaults to retrieve the original value.</p><p>I’ve explained the various methods, in the subsequent section “Types of Tokenization”</p><h4>Method 2: “Masking”</h4><p>Masking is an irreversible method, where data is obfuscated with some random data. It is only one-way process. Masking can be done via techniques like Hashing OR any other randomizers. There are 2 types of Masking. Static Data Masking where the Data-on-Rest is masked, where as Dynamic data masking where sensitive data is masked by application based on the need (say credit card number should not be displayed for a normal operator, where as a supervisor should be able to see it).</p><p>Generally masking is used when transferring data from a production environment to a test environment.</p><h4>Method 3: “Redaction”</h4><p>This is a brute-force method of deleting OR replacing sensitive text with some special characters any sensitive information on the file storage</p><h3>Section 2: Tokenization Methods</h3><p>In Payment Tokenization, as mentioned earlier, PAN is the key element that would be Tokenized.</p><p>As we all might be aware ,</p><ul><li>PAN can be 15 or 16 digit in length</li><li>first 5 or 6 digits can be BIN</li><li>last digit can be check-digit (not mandatory)</li></ul><p>There are different types of token number options, listed in the below table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*O97PPGtStrv617DqQtwlZQ.png" /><figcaption>Courtesy: Annex H of PCI — Tokenization Product Security Guidelines</figcaption></figure><p>Since all the payment systems are built around 15/16 digit (numeric) PAN number. The Token value is supposed to be a surrogate and any the Token number should mimic a real PAN number. This would ensure minimalistic changes to payment eco-system and applications.</p><p>Hence the option 1–5 are not used for Payment Tokenization.</p><p>Options 7 and 9 retain some part of original PAN, and would allow the original PAN (or part of PAN) to be guessed. Hence it can not be used.</p><p>Option 6 — changes the leading digit of the PAN, which might switch the BINs, and hence this Token method is not used.</p><p><strong><em>Option 8</em></strong><em> is the typical type of tokenization method used as payment tokens, where every real PAN range would have a mapping token range i.e. every PAN BIN would have a corresponding Token BIN.</em></p><h3>Section 3: Token Types (from PCI)</h3><p>PCI classifies Tokenization into 4 different types of categories. Refer <a href="https://www.pcisecuritystandards.org/documents/Tokenization_Product_Security_Guidelines.pdf">Tokenization Product Guidelines documen</a>t for further information</p><ul><li><strong>Reversible Cryptographic: </strong>these Tokens are derived based on certain mathematical function (cryptographic function), which would be used. The Token or PAN is not stored by the Tokenization application. <em>Only the cryptographic keys that is used to generate the Tokens is stored.</em></li><li><strong>Reversible Non-Cryptographic:</strong> these Tokens are generated via some randomizer, and the Tokens are maintained in a separate vault i.e. mapping between the PAN and Token are maintained in separate vault. There would be logic (mathematical logic) with which the Token number can be generated from the PAN, for these types of Tokens. <em>This is the defacto approach for Tokens as Payment Instruments.</em></li><li><strong>Irreversible Authenticatable: </strong>Typically this is a Data “Masking” technique detailed above, where it is a one-way method to mask the PAN number. it is not reversible. These type of Tokens are used during chargebacks, and PAN can be encrypted one-way and the rest of the details of the transaction can be presented as-is. if the other party/recipient has the PAN, that entity can also run a hash to find out the validity of the Token.</li><li><strong>Irreversible Non-Authenticatable: </strong>These types of tokens are not linked to the original PAN and can not authentication or map to a given PAN whatsoever.</li></ul><p><strong>Concluding Remarks:</strong></p><ul><li>Payment Tokens i.e. Tokens as Payment Instruments fall into the category of “Reversible Non-Cryptographic” Token</li><li>Tokens are usually 15/16 digit number — same length as that of the PAN number. This would ensure no/minimal impact as any change to the format and length of PAN/Token will have considerable changes.</li><li>A Token vault is maintained to maintain PAN-Token Mapping, and Token can not be mathematically derived using PAN and vice-versa</li><li>A separate BIN range can be maintained for the Token numbers, which would have a mapping to the corresponding PAN range.</li></ul><p><em>In the </em><a href="https://medium.com/@satish.vt/chapter-2-tokens-as-payment-instruments-concept-stakeholders-token-issuance-process-8e9a46f8d916"><em>next chapter</em></a><em>, let us look into “Tokens as Payment Instruments” — Concept, Stakeholders &amp; Token Issuance Process</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73777f3bdf24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Sqoop to PostgreSQL — Some Learnings]]></title>
            <link>https://medium.com/@satish.vt/apache-sqoop-to-postgresql-some-learnings-964066c29ac?source=rss-c460585cbaec------2</link>
            <guid isPermaLink="false">https://medium.com/p/964066c29ac</guid>
            <category><![CDATA[sqoop]]></category>
            <category><![CDATA[sqoop-postgres-issues]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Satish VT]]></dc:creator>
            <pubDate>Tue, 02 Jun 2020 16:16:43 GMT</pubDate>
            <atom:updated>2020-06-02T16:29:58.565Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Sqoop to PostgreSQL — Some Learnings</h3><p>Apache Sqoop is a (famous) tool which is used to bulk transfer data between Hadoop and structured databases or Mainframes.</p><p>Was trying to install Sqoop in my Mac and connect it to a PostgreSQL database. In this pursuit, i had faced few issues/challenges connecting Sqoop to a PostgreSQL database. This blog post is to mention the issues i faced, and how i went about resolving them.</p><p>Hope this would be useful ! for people who might face similar issues</p><p><em>Disclaimer:</em> This is not a detailed installation guide.</p><p><em>Environment</em>: MacOS Catalina 10.15.4; Installed “SQOOP” using Brew</p><ol><li><strong>PostgreSQL JAR:</strong></li></ol><ul><li>Download PostgreSQL JAR file from <a href="https://jdbc.postgresql.org/download.html">https://jdbc.postgresql.org/download.html</a></li><li>Copy the file to $SQOOP_HOME/lib (<em>in my case it was /usr/local/Cellar/sqoop/1.4.7/libexec/lib/</em>)</li><li>Create a file “postgresql” in $SQOOP_HOME/conf and copy the text <strong>org.postgresql.Driver=/usr/local/Cellar/sqoop/1.4.7/libexec/lib/postgresql-42.2.6.jar </strong>(basically $SQOOP_HOME/lib/&lt;postgresql_jar&gt;)</li></ul><p><strong>2. ACCUMULO</strong> <strong>Download</strong></p><ul><li>Download the Accumulo software (binary) from <a href="https://accumulo.apache.org/downloads/">https://accumulo.apache.org/downloads/</a></li><li>Unzip the binary using “tar zxf”</li><li>Define the ACCUMULO_HOME in bash_profile to the unzipped folder</li></ul><p><strong>3. Sqoop environment variables:</strong></p><ul><li>Copy the file sqoop-env-template.sh in $SQOOP_HOME to sqoop-env.sh and set-up the following parameters HADOOP_HOME, HBASE_HOME, HIVE_HOME, ACCUMULO_HOME, ZOOCFGDIR</li><li>Also define the same variables in bash_profile</li><li>export ACCUMULO_HOME=/Users/<strong>&lt;user-name&gt;</strong>/_softwares/accumulo-2.0.0</li><li>export SQOOP_VERSION=1.4.7</li><li>export SQOOP_HOME=/usr/local/Cellar/sqoop/1.4.7/libexec</li><li>export SQOOP_CONF_DIR=$SQOOP_HOME/conf</li><li>export SQOOP_LIB_DIR=$SQOOP_HOME/lib</li><li>export PATH=$SQOOP_HOME/bin:$PATH</li><li>export HADOOP_HOME=/usr/local/Cellar/hadoop/3.2.1_1/libexec</li><li>export HBASE_HOME=/usr/local/Cellar/hbase/1.3.5/libexec</li><li>export HIVE_HOME=/usr/local/Cellar/hive/3.2.1_1/libexec</li><li>export ZOOCFGDIR=/usr/local/etc/zookeeper</li><li>export ZOOKEEPER_HOME=/usr/local/etc/zookeeper</li></ul><p><strong>References:</strong></p><ul><li><a href="https://gist.github.com/tilakpatidar/87f61c8ac62f018110db5f3a4db4db0e">https://gist.github.com/tilakpatidar/87f61c8ac62f018110db5f3a4db4db0e</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=964066c29ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Incremental Web Data Extraction (with an intentional Complex Architecture to learn/experiment)]]></title>
            <link>https://medium.com/@satish.vt/incremental-web-data-extraction-with-an-intentional-complex-architecture-to-learn-experiment-622696c09ba3?source=rss-c460585cbaec------2</link>
            <guid isPermaLink="false">https://medium.com/p/622696c09ba3</guid>
            <category><![CDATA[web-data-extraction]]></category>
            <category><![CDATA[data-collection]]></category>
            <dc:creator><![CDATA[Satish VT]]></dc:creator>
            <pubDate>Fri, 29 May 2020 18:11:43 GMT</pubDate>
            <atom:updated>2020-06-21T03:11:17.004Z</atom:updated>
            <content:encoded><![CDATA[<h3>Incremental Website Data Extraction (with an intentional Complex Architecture to learn/experiment)</h3><h3>Background:</h3><p>For a ML/NLP project idea of mine (Information Extraction and Modelling), I required some text data precisely “News” data. I have written a lot of web parsers and used crawlers in the past which extract the text fully. <em>However, I had always a thought of how to incrementally extract text from a website/page.</em> Then I had a simple idea of storing the <em>“content hash”</em>. Use this hash later on, to check for any updates on the page, and then extract &amp; store the content if and only if hash has changed.</p><p>In addition to this simple idea above, I wanted to build modular application where I have web extracts (content producers) separately and consumers separately which clean the text and store in the filesystem and use some sort of message broker between producers and consumers. <em>I know it’s an over-kill for this kind of application, However I wanted to explore Kafka (again). Kafka Python client to be precise.</em></p><p><strong><em>In this article, i want to share the design, learnings and the link for the code</em></strong></p><h3>Objectives/Requirements:</h3><ul><li>Write a web data extractor, which identifies pages which have content-change</li><li>Write separate producer and consumer modules for extraction and storing the data</li><li>Use Kafka as a Message Broker (I’ve used Kafka earlier. But never Python Kafka client, so wanted to use this)</li></ul><h3>Design:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*kdzY8owJb6yaXgNDePnIPg.jpeg" /></figure><p>Other Considerations:</p><ul><li>Schedule the entire process and run it as a background process</li></ul><h3>Facts &amp; Inputs:</h3><ul><li>I used “www.firstpost.com” news website for scraping. This is one of my favourite news sites. I wanted to scrap news articles from about 9 categories within the news website.</li><li>Language of choice: Python</li><li>Environment used: My Mac</li></ul><h3>Code URL:</h3><p><a href="https://github.com/satishvt/incremental-web-extractor">https://github.com/satishvt/incremental-web-extractor</a></p><h3>Deep Dive:</h3><p><strong>Component 1:</strong> PostgreSQL db and a table to store “news article” and “hash”</p><ul><li>Create a new table called “firstpost” which has two columns the article and hash</li><li>“hash” is a 32 byte varchar field. It holds MD5 hash of the entire article text</li></ul><p><strong>Component 2:</strong> Kafka and Topics</p><ul><li>Create 9 topics for each category from the news site</li><li>For ease of use, i created a single partition and replication factor as 1. These configurations can be re-looked at based on the need</li><li>Commands used:</li></ul><blockquote>kafka-topics — zookeeper 127.0.0.1:2181 — topic <strong>&lt;topic-name&gt;</strong> — create — partitions 1 — replication-factor 1</blockquote><p><strong>Component 3: </strong>Web page extractor (all blue boxes in the Design)</p><p><strong><em>Component 3.1: Extract web links from home page</em></strong></p><ul><li>Extract links from the home page of the category. Simple plain BeautifulSoup script</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Wjeq13iI3anZqVhIricOw.png" /></figure><p><strong><em>Component 3.2: Get Content</em></strong></p><ul><li>Extracting unique id from each article</li><li>Extract the content from the news article (again using BeautifulSoup), Assume reader understands/knows how text is extracted from HTML page source</li><li>Lessons Learnt/Solution: <strong>“json.loads/dumps”</strong> function in Python fails if the encoded utf-8 string has newline strings in it. Hence when you use a BeautifulSoup, don’t pass the encoding as a utf-8. Directly replace the “\r” and “\n” in the string, and strip spaces.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u6OE2Tma0wXDKj0nKbCV9A.png" /></figure><p><strong><em>Component 3.3: Check if content is present OR if updated</em></strong></p><ul><li>Created a DB class which can be instantiated later on</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K2xo46G9H9pEV8FO6D-A1g.png" /></figure><p>Below function checks if the article is already scraped earlier.</p><ul><li>If already, check if the new hash is different than already scraped one. If any changes, than update the new content</li><li>If already scraped and no change in hash, do NOT update it again</li><li>If article not present, then update now</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SGvBvWpH8prdYmF8Xq2yvA.png" /></figure><p>Below is the calling code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zJqAD3N_wVjFUtdexamuqA.png" /></figure><p><strong><em>Component 3.4: Kafka Producer</em></strong></p><p>Below code acts as a Kafka producer to send the code to the corresponding topic. (in the below snippet, it sends the message to a topic “tech”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seWOz3D1vlo5H_Aoo6ER2A.png" /></figure><p><strong><em>Component 4: Kafka Consumer</em></strong></p><ul><li>Below is the simple code, for consuming the data from the Kafka and it also stores the data into a folder</li><li>use “group_id” to ensure you incrementally read the messages, along with parameter “enable_auto_commit” to True</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7s1ESglX8dV8tpgBs63sYw.png" /></figure><p><strong><em>Component 5: Cron (Auto-scheduling)</em></strong></p><ul><li>I wanted to schedule all of this as a Python code as an automatic used, i found a python package (crontab) to schedule it as Cron. This was very helpful to schedule it via code, instead of editing in crontab command</li><li>Lesson Learnt: OS X comes with a bundled with bundled Python, and i used ananconda. So my Python path was not picked properly. Hence i changed “/etc/paths” to pick Python from ananconda. Also i had to source the bash_profile in command as shown below</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RC-Cey05hkusRXyGdMRsPw.png" /></figure><h3>Closing notes:</h3><ul><li>While the overall project seems to be a simple web extractor, i hope that the simple logic hash/article would certainly help</li><li>Hope you find this article and code useful, for simple web extractor</li><li><strong><em>Fun fact: </em></strong><em>The number of lines in the article are much higher than actual lines for a single Python producer (was just 95 lines), and a single Python consumer was 18 lines.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=622696c09ba3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Primer/FAQ on EMV Scripting]]></title>
            <link>https://medium.com/@satish.vt/primer-faq-on-emv-scripting-3e521a088517?source=rss-c460585cbaec------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e521a088517</guid>
            <category><![CDATA[emv-chip]]></category>
            <category><![CDATA[emv-cards]]></category>
            <category><![CDATA[payments]]></category>
            <dc:creator><![CDATA[Satish VT]]></dc:creator>
            <pubDate>Sat, 20 Apr 2019 12:02:51 GMT</pubDate>
            <atom:updated>2019-04-20T12:02:51.458Z</atom:updated>
            <content:encoded><![CDATA[<p>EMV Script processing, is one of the most important feature that is unused by many EMV chip card issuers/processors. It provides a greater flexibility &amp; control for issuers over the cards issued by them.</p><p>In this post, the terminology card or device are used interchangeably. Since the EMV chip can be issued as a physical plastic OR as a wearable. Also in this post, technical details are not covered. It is a primer/FAQ on EMV Scripting.</p><p><strong><em>What is so significant about this EMV scripting?</em></strong></p><p>Once the card (device) is issued by Issuer to the customer and delivered, traditionally the data on the card can <strong>NOT </strong>be updated later on.</p><p>For EMV chip cards, “<em>script processing</em>” provides an option to the issuer for updating the data in the chip card, as a part of transaction processing.</p><p><strong><em>When the issuer can update the data on the EMV scripting?</em></strong></p><p>This is possible only during an “online transaction” is received by the issuer. Online transaction here means, when the transaction done at the terminal is passed on to the issuer for decisioning.</p><p>EMV scripting is possible for Chip Contact and Contactless transactions. However if it is a Chip contact transaction, the chip card is immersed in the terminal and the terminal has time to pass any actions on the chip (pass the commands on to the chip card) (the card is present inside the terminal, and can<strong>not</strong> be removed till the transaction is complete)</p><p>In case of Contactless transactions, the card is held by the customer in hand and so long as the card/device is present RF zone (terminal RF zone), the script processing can be performed successfully. If the card is removed from the RF zone, then the script processing can NOT be executed. Refer <strong>Section 3.11 — Issuer Script</strong> <strong>Processing</strong> of “<strong>Book C-5”</strong> of “<strong>EMV Contactless Specifications for Payment Systems”</strong> for further details.</p><p>Hence in case of a Chip contact transaction, the success rate of a Script processing is higher and most certain than that of a Contactless transaction.</p><p><strong><em>What data/actions can be updated/performed using Script Processing?</em></strong></p><ul><li>Card parameters that are defined at the time of personalisation. for example, the LCOL, UCOL parameters for offline</li><li>PIN Try Counter Reset</li><li>Offline PIN change</li><li>Blocking/Unblocking the application/card</li></ul><p>Refer EMV — <strong>Book 3 — Application Specification — Integrated Circuit Card Specifications for Payment Systems</strong> for further details on the commands, and <strong>Issuer-to-Card Script Processing </strong>(Section 10.10) process</p><p>Each of the actions/command has a specific code and format.</p><p>Refer the aforementioned book, for the script/command number and format. Also please refer <a href="https://flylib.com/books/en/4.365.1.54/1/">https://flylib.com/books/en/4.365.1.54/1/</a> for more technical information on the Issuer Script (i.e. post-issuance scripts before &amp; after 2nd GENERATE AC command)</p><p><strong><em>How does script processing work?</em></strong></p><p>Issuer inserts the EMV script, as a part of the authorization response message in authorization ISO8583 data elements. For example, for a MasterCard authorization — script is inserted in DE55, and DE142 for Visa authorization message.</p><p><strong><em>How many scripts can the issuer send?</em></strong></p><p>The number of scripts that an issuer can send in the authorization response, is limited to 128 bytes. Any number of commands which can be sent within 128 bytes (sum of all commands’ length should be less than 128 bytes).</p><p>Priority can be defined by the issuer, as to the order in which the commands are to be executed.</p><p><strong><em>Use-cases for Issuer Scripting</em></strong></p><p><strong><em>Blocking the card</em></strong></p><ul><li>Customer calls the issuer loss-of-card &amp; the issuer blocks the card</li><li>The fraudster tries to mis-use the card in a terminal</li><li>The transaction comes online to the Issuer</li><li>Issuer realizing the card is blocked, declines the transaction and also sends a “Card Block” script as a part of the authorization response</li><li>Terminal passes the command to the chip which further blocks the card</li><li>Card becomes unusable</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e521a088517" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Indian Payment Systems — Classification & Statistics]]></title>
            <link>https://medium.com/@satish.vt/indian-payment-systems-classification-statistics-86e7401b18fb?source=rss-c460585cbaec------2</link>
            <guid isPermaLink="false">https://medium.com/p/86e7401b18fb</guid>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[indian-payment-systems]]></category>
            <category><![CDATA[indian-payment-statistics]]></category>
            <category><![CDATA[npci-statistics]]></category>
            <dc:creator><![CDATA[Satish VT]]></dc:creator>
            <pubDate>Wed, 13 Mar 2019 11:32:42 GMT</pubDate>
            <atom:updated>2019-03-13T11:32:42.820Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://mycardpaymentblog.wordpress.com/2019/03/13/indian-payment-systems-classification-statistics/">Indian Payment Systems — Classification &amp; Statistics</a></h3><p><strong><em>In this blog post, I do NOT intend to educate the reader on the various payment options. However the idea is to present the statistics on the growth of the amount &amp; number of transactions of these payments over the years.</em></strong></p><p>NPCI &amp; RBI must be greatly appreciated for creating a robust and resilient payment systems backbone for making India, a true “Digital Economy” state. A decade earlier, the predominant mode of payments for individual/business, small/large value payments used to cash, card, cheque, ECS, NEFT &amp; RTGS. However, in the recent years, introduction of IMPS, AEPS, UPI (BHIM &amp; USSD), NETC, phasing out ECS with NACH, BBPS have spiked up geometrically the number of digital transactions, and more importantly more options to its customers.</p><p>The way RBI/NPCI classifies our payment systems/options is (a) Paper (b) Electronic ( c) Card Based Payments.</p><p>However, I have chosen to classify this a little differently,</p><p><strong>(a) Fast Payments &amp; Retail:</strong> These payment options are typically used by individuals to perform P2P payments (or payments-to-merchants) and most importantly low-ticket transactions. The settlement of these transactions happens instantaneously.</p><p><strong>(b) Clearing &amp; Large value Payments:</strong> These options are used by individuals or businesses to perform large value payments, and usually the money transfer across in a delayed fashion. Delayed here implies, that the settlement between the accounts/banks happen in at defined settlement timings. ACH &amp; Cheque are a classic example of delayed settlements</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_7Ve4mcDUKdELeUT79Zr2w.png" /><figcaption>Payment Systems Classification</figcaption></figure><h3><strong>Statistics on Large Value &amp; Clearing based payment systems</strong></h3><p>Below table provides details of the Amount of Transactions across various Large Value &amp; Clearing based payment systems</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pbo4nqytuIQal3DaUzxhuw.png" /><figcaption>Statistics of Large Value &amp; Clearing Payment Systems</figcaption></figure><p>Inferences:</p><ul><li>RTGS is the largest payment system and is way ahead of all the other payment systems. There is a double-digit growth (10–20%) in this payment systems YoY</li><li>NEFT is rapidly growing at 40–50% YoY</li><li>As expected, paper-based cheque payment system is declining since</li><li>ECS system is being slowly phased out and being replaced by NACH, which is also seeing a constant growth</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KpYLbuQHNtq0WijQSDXRLA.png" /><figcaption>Trend of Clearing &amp; Large Value Payment systems’ volume</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E2fpV5DdCq3VaNFBMI5_KA.png" /><figcaption>Trend of Clearing &amp; Large Value Payment systems’ volume (without RTGS)</figcaption></figure><h3><strong>Statistics on Fast Payments &amp; Retail</strong></h3><p>Below table provides details of the Amount of Transactions across various Fast Payments &amp; Retail Payment Systems</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EvfGs9XqgTEp7hQTXFf9rQ.png" /><figcaption>Statistics of Fast Payments &amp; Retail Payment Systems</figcaption></figure><p>Inferences:</p><ul><li>Every Fast payment has seen a tremendous increase in the last years</li><li>UPI in specific has seen an exponential increase, Thanks to various wallets’ adoption of UPI</li><li>IMPS has doubled till FY 2018, and the growth is still at around 50% this FY2018–19</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-i4GCOgWIfDB_1FG5mpouQ.png" /><figcaption>Trend of Fast Payments &amp; Retail Payment Systems</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86e7401b18fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science on Audio — An Introduction.]]></title>
            <link>https://medium.com/@satish.vt/data-science-on-audio-an-introduction-e3766c78bc5c?source=rss-c460585cbaec------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3766c78bc5c</guid>
            <category><![CDATA[data-science-on-audio]]></category>
            <category><![CDATA[audio-emotion]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[emotion-identification]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Satish VT]]></dc:creator>
            <pubDate>Sun, 20 Jan 2019 09:39:46 GMT</pubDate>
            <atom:updated>2019-01-20T09:39:46.087Z</atom:updated>
            <content:encoded><![CDATA[<h3>Machine Learning for Audio: Introduction, Use-cases &amp; Technology</h3><h3><strong>Section 1: Introduction</strong></h3><p>Few months back I was a part of team who worked on interesting project on emotion identification given an audio file. In a data scientist’s language, we had to “classify” the emotion of the speaker.</p><p>In this post, I want to share my learning on various aspects of audio file processing.</p><p>What this blog is and what this blog is <strong>not.</strong></p><p><em>What this blog is:</em></p><ul><li>Understand features of audio files (or DSP processing) that a data scientist should learn before working on audio-data science projects</li><li>Provide links for further reading</li><li>Provide various aspects (use-cases) of Audio in Data Science</li><li>Provides list of packages that can be used for audio files</li><li>What are the typical pitfalls (OR i should say the issues i faced, and their mitigations)</li></ul><p>Is NOT for:</p><ul><li>Detailed explanation of the audio features</li><li>Technology or approach (code) for any audio file processing</li></ul><h3><strong>Section 2: Audio Files’ 101</strong></h3><p>Understand audio format, codec, sampling rate, bit depth</p><p><strong>Audio format &amp; Audio Codec:</strong> In a layman terms audio <strong>format</strong> is the file format that we see on an audio file. Like wav, mp3 etc. <strong>Codec</strong> is the short format of compression-decompression which is the technical algorithm that is used for compression of the audio file. Some of the sample codecs like raw, PCM etc.</p><p>Understanding the formats and codecs are very important because, even the most-used and the leading Python packages like <strong>librosa</strong>, <strong>pyaudio </strong>don’t support all the codecs, while they might support all major formats.</p><p><strong>Sampling rate and Bit-depth: </strong>Sampling rate signifies the number of samples taken per unit of time. Bit depth signifies the number of bits used to store the sample.</p><p>Understanding the source audio file’s sampling rate is very important. The sampling rate also relates to the quality of the audio sample. Higher the sampling rate, higher the quality. However it also means the size of the audio file increases.</p><p>Many of the python packages default to a sampling rate say 22500 if you don’t specify the sampling rate. Hence understanding and mentioning the sampling rate of an audio file is important.</p><p>Apart from the above features, what says without going is to understand the frequency, amplitude etc.</p><ul><li><a href="http://www.jilion.com/best-audio-format.html">Most popular audio formats: which to choose for Mac?</a></li><li><a href="https://medium.com/@MicroPyramid/understanding-audio-quality-bit-rate-sample-rate-14286953d71f">Understanding Audio Quality: Bit Rate, Sample Rate</a></li><li><a href="https://community.plm.automation.siemens.com/t5/Testing-Knowledge-Base/Digital-Signal-Processing-Sampling-Rates-Bandwidth-Spectral/ta-p/402991">Digital Signal Processing: Sampling Rates, Bandwidth, Spectral Lines, and more...</a></li></ul><h3><strong><em>Section 3: Audio features (contd.)</em></strong></h3><p>As we all know, audio files are a time series data across frequency and amplitude domain. However these features wouldn’t directly help us to perform much specific audio file related applications.</p><blockquote>There are two important features of audio called as Spectral and Cepstral features of the audio files. These features have a variety of applications in data sciences starting from language identification, speech-to-text, emotion identification etc.</blockquote><p><strong>Spectral features</strong> are obtained by performing a fourier transform of the audio signal. In simple terms, the audio files is split by small time chunks, and then apply fourier transforms (also called as DFTs and FFTs, DFT is a discrete version of Fourier transforms, and FFTs are a faster version of obtaining DFTs). Applying the fourier transforms has a variety of uses like understanding the peaks, fundamental frequency of the wave etc. Spectrogram is one of the major audio feature that is used for any audio based applications and can be obtained using packages <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.spectrogram.html"><strong>scipy</strong></a><strong>, </strong><a href="https://librosa.github.io/librosa/generated/librosa.feature.melspectrogram.html"><strong>librosa</strong></a><strong>. </strong>these packages provide the output as a numpy array, which can be used for any further processing.</p><p>Applying an inverse transform on the spectral features is called as the <strong>Cepstral features</strong>, which would provide a real representation of the audio file. MFCC (Mel-frequency Cepstral coefficients) is one of the very major acoustic features in Cepstral features, used in speech related processing. Packages like <a href="https://librosa.github.io/librosa/generated/librosa.feature.mfcc.html"><strong>librosa</strong></a> again provide functions to extract the Cepstral features. One important aspect is the number of co-efficients to choose int he MFCC, Playing around the number of co-efficients is important for any data science related analysis. You might want to play with the number of coefficients to test the accuracies of any model that you want to develop.</p><p>Other important techniques that are important to be learnt are HMM (Hidden Markovian Models) and Gaussian Mixture Models (GMM) for any segmentation and other purposes in audio.</p><p><a href="https://dsp.stackexchange.com/questions/28898/mfcc-significance-of-number-of-features">https://dsp.stackexchange.com/questions/28898/mfcc-significance-of-number-of-features</a></p><p><a href="https://pdfs.semanticscholar.org/b119/86fdff3c924e96c5e7cc96eb562eab5254b1.pdf">https://pdfs.semanticscholar.org/b119/86fdff3c924e96c5e7cc96eb562eab5254b1.pdf</a></p><p><a href="http://www.speech.cs.cmu.edu/15-492/slides/03_mfcc.pdf">http://www.speech.cs.cmu.edu/15-492/slides/03_mfcc.pdf</a> <a href="https://mycourses.aalto.fi/pluginfile.php/276358/course/section/77509/L4-Analysis.pdf">https://mycourses.aalto.fi/pluginfile.php/276358/course/section/77509/L4-Analysis.pdf</a></p><h3>Section 4: Typical Use-cases on Audio files and Approach</h3><p>General requirements on audio files are</p><ul><li>Language Identification</li><li>Emotion Classification</li><li>Speech-to-Text</li></ul><p>For all the typical use-cases the approach the standard approach for any of the requirements is to extract the audio features, into a “numpy” array using packages like “librosa” “pyaudio” “pyAudioAnalysis” “scipy” and do standard processing i.e. apply deep learning algorithms like NN, CNN, RNN, LSTM etc.</p><p>The important aspects is always to extract the right set of features as mentioned in Section 2 &amp; 3 i.e. preprocessing of the audio files based on the audio features and also extracting the features like spectrograms, MFCC etc</p><p>While working on the emotion classification, couple of challenges that we faced</p><ul><li>Silence Removal</li><li>Speaker Diarization</li></ul><p>These two steps are also very important pre-processing before performing any further use-cases</p><h3>Section 4.1: Silence Removal</h3><p>In all typical audio conversation, we all know that there would silences or pauses. These generally don’t bear any significance until you are taking the silence as a parameter into a model.</p><p>Now i want to introduce one of the <strong>best</strong> audio processing packages that i’ve seen and used “<a href="https://github.com/tyiannak/pyAudioAnalysis"><strong>pyAudioAnalysis</strong></a>”. This is an open-source package, This package provides a <a href="https://github.com/tyiannak/pyAudioAnalysis/wiki/5.-Segmentation#fixed-segment-segmentation--classification"><strong>Segmentation function</strong></a> to identify the silences in the audio for any further processing i.e. provide the start-end seconds of the audio file with silences such that they can be removed.</p><h3>Section 4.2: Speaker Diarization</h3><p>In a conversational audio file, there would be more than two speakers, an another requirement would be identify</p><ul><li>how many speakers</li><li>from which time to which time frame which speaker spoke, is called as “Speaker Diarization”.</li></ul><p><em>for example, for 0–15 seconds — silence, 15–24 seconds — speaker 1, 25–33 — speaker 2, 34–50 seconds speaker 3. Seeing an output like this is called as Speaker Diarization</em></p><p>Again “pyAudioAnalysis” provides a “<a href="https://github.com/tyiannak/pyAudioAnalysis/wiki/5.-Segmentation#speaker-diarization"><strong>Speaker Diarization</strong></a>” function. This package uses HMM based segmentation, which works fairly well. However i would suggest to prepare some training data and train this package before putting for any application. While this function works well., it doesn’t give an 100% accuracy.</p><h3><strong>Section 5: Emotion Identification</strong></h3><p>Before you embark on any emotion identification, first finalize which emotion classification groupings you want to pick. Below are some emotion groupings you want to pick.</p><ul><li>Emotion annotation and representation language (EARL) proposed by the Human-Machine Interaction Network on Emotion (HUMAINE)</li><li>Parrott’s emotions by groups</li><li>Plutchik’s wheel of emotions</li><li>Six Basic Emotions</li></ul><p>In the project that we worked on for Emotion identification a simple MLP worked well for us with MFCC — 100 coefficients as features.</p><p>Two major databases for audio files for emotion classification are:</p><p>SAVEE datbase: <a href="http://kahlan.eps.surrey.ac.uk/savee/">http://kahlan.eps.surrey.ac.uk/savee/</a></p><p>Berlin Database of Emotional Speech: <a href="http://emodb.bilderbar.info/docu/#home">http://emodb.bilderbar.info/docu/#home</a></p><h3>Section 6: Speech-to-Text</h3><p>Speech to Text is one of the major needs when doing audio file processing. In our case, we wanted to extract the text to find the emotion from the audio-to-text translation.</p><p>Google, Azure, IBM Watson all provide Speech-to-text capabilities APIs, with a very good accuracy, and one can refer their APIs. This GitHub package “<a href="https://github.com/Uberi/speech_recognition"><strong>speech_recognition</strong></a>” provides a very good wrapper to connect the aforementioned speech-to-text enterprise-class APIs</p><p>One open source speech-to-text package that i used and personally liked well was Carnegie Mellon University’s Sphinx package (<a href="https://cmusphinx.github.io/"><strong>CMU Sphinx</strong></a>). It provides a beautiful framework for speech-to-text for any languages. It also provides a package called PocketSphinx for porting on to mobile applications (light weight).</p><p>Acoustic and Speech models for major languages (US English, British English, German, Hindi, Spanish, Italian) and are available <a href="https://sourceforge.net/projects/cmusphinx/files/Acoustic%20and%20Language%20Models/"><strong>here</strong></a>.</p><p>The only fallback of this CMU Sphinx package, was the accuracy of the models weren’t great, for us — it was at around 60% or so. However if your project has appetite to create acoustic/language models, CMU provides toolkits to build your own models for your language.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3766c78bc5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demonetization — Impact on Debit Card — POS usage — A Statistical Study]]></title>
            <link>https://medium.com/@satish.vt/demonetization-impact-on-debit-card-pos-usage-a-statistical-study-c1aaf5a4fcf0?source=rss-c460585cbaec------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1aaf5a4fcf0</guid>
            <category><![CDATA[demonetization]]></category>
            <category><![CDATA[payments]]></category>
            <dc:creator><![CDATA[Satish VT]]></dc:creator>
            <pubDate>Tue, 28 Aug 2018 10:14:23 GMT</pubDate>
            <atom:updated>2018-08-30T06:19:44.545Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>Abstract:</em> </strong>There was a spike in the usage of Debit Cards post demonetization over POS, than the normal (natural) growth it could have achieved. In my study, i wanted to understand how much impact did it really have?</blockquote><ul><li>Used ARIMA forecasting technique to forecast the POS growth, had demonetization not happened</li><li>Compared with the actual growth</li><li>There was an average increase of about <strong>Rs. 16,000 crore per month </strong>in the usage of Debit Card over POS, due to demonetization. Detailed approach is given below</li></ul><blockquote><strong>Data used:</strong> ATM &amp; POS Statistics from RBI website, from Apr’11 — Mar’18.</blockquote><blockquote><strong>Detailed Approach:</strong></blockquote><p>To understand how much impact (post demonetization), the approach i have used is to find out the difference amount between usage with &amp; without demonetization. Now comes the tricky part, how do we find out the amount had demonetization not happened.</p><p>To achieve this, i have used Time Series Forecasting technique ARIMA technique to find out the natural growth trend without demonetization.</p><p>I used the data between Apr’11 till Oct’16 to find the ARIMA model.</p><p><strong>Step 1:</strong> Decompose the trend of pre-demonetization period</p><ul><li>From the below trend-decomposition, it could be observed that there is a linear growth, with a proper seasonality</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*Vg9hmw0WjMaCAIZOY6PuUw.png" /></figure><p><strong>Step 2: Finding the right ARIMA model:</strong></p><ul><li>Split the data from Apr’11 — Oct’16 data into two parts — Training &amp; Validation</li><li>Training period — Apr’11 till Oct’15</li><li>Validation period — Nov’15 till Oct’16</li><li>RMSE was used as the measure to select the right ARIMA model</li><li>(p,d,q) = (1,0,1), seasonal (p,d,q,seasonality) = (1,0,1,12) was selected as the right model using the training</li><li>The ARIMA model selected was validated with the training data</li><li>It is shown in the below graph (blue line is actual, orange is forecasted using ARIMA model)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*9up8Nr68AVH30OlSRBtY0Q.png" /></figure><p><strong>Step 3: Forecast the Debit Card — POS Usage from Nov’16 till Mar’2018</strong></p><ul><li>Using the aforementioned ARIMA model, forecasted the POS Usage for the post-demonetization period</li><li>It was observed that approximately from Feb’17 till Mar’18 there was an average increase of about 16,000 crore per month in the usage of Debit Card over POS. (I’ve excluded Nov’16 — Jan’17 from calculating the average as cash was not available in the market)</li><li>Below graph shows the actual growth (orange line) vs forecasted growth (blue line)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*OMMa-78yGPgCMWLQ_kzJbA.png" /></figure><blockquote><strong>Key points:</strong></blockquote><ul><li>This doesn’t take into consideration the other digital payment modes available for citizens to transact</li><li>The above study was done as an academic exercise</li></ul><blockquote><strong>Technologies used:</strong></blockquote><ul><li>All the data collection, data cleansing, statistical forecasting, graphs were done in Python</li><li>Python packages: BeautifulSoup, urllib, wget, statsmodels</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1aaf5a4fcf0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>