<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashwini Kumar Maurya on Medium]]></title>
        <description><![CDATA[Stories by Ashwini Kumar Maurya on Medium]]></description>
        <link>https://medium.com/@ashwinikumarmaurya?source=rss-44fa4ce31baa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*aFzZZKyIjqfM52WZ7l6f5w.jpeg</url>
            <title>Stories by Ashwini Kumar Maurya on Medium</title>
            <link>https://medium.com/@ashwinikumarmaurya?source=rss-44fa4ce31baa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:54:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashwinikumarmaurya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Entity Resolution at Scale: Mapping 100+ million customer’s account to customer’s identity]]></title>
            <link>https://ashwinikumarmaurya.medium.com/copyright-2023-ashwini-maurya-albertsons-companies-52ff8fa98011?source=rss-44fa4ce31baa------2</link>
            <guid isPermaLink="false">https://medium.com/p/52ff8fa98011</guid>
            <category><![CDATA[identity-resolution]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[customer-growth]]></category>
            <category><![CDATA[customer-360]]></category>
            <category><![CDATA[innovation]]></category>
            <dc:creator><![CDATA[Ashwini Kumar Maurya]]></dc:creator>
            <pubDate>Sun, 13 Aug 2023 23:10:55 GMT</pubDate>
            <atom:updated>2023-08-14T06:37:25.115Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>By Ashwini Maurya, Albertsons Companies</strong></blockquote><h4>Abstract</h4><blockquote><em>Customer Identity is at core of many businesses. This becomes very complex in presence of over 100 million customers, especially when most of Personal Identifiable Information (PII) are missing or have invalid values. At Albertsons, we use PII info along with engagement data, item level transactions, payments, and third party data to map customer’s account at both customer and household levels. We develop various methodologies1 and pipelines for data processing , feature engineering, predictive modeling, id elevation and stamping. These algorithms are scalable and quite robust in the sense that the generated shopper id and household ids are invariant to change when new cards are added to the their account. Model is packaged as python package and deployed in production. The analysis of about 100+ million card numbers shows about reduction of 9% card numbers to customer id, and reduction of about 16% from card numbers to household id.</em></blockquote><blockquote><strong><em>The development entity resolution framework, its methodology, technical details and algorithmic implementations are Albertsons propriety material and unfortunately not available to public.</em></strong></blockquote><blockquote><em>Corresponding author(s): Ashwini Maurya, Albertsons Companies, ashwini.maurya@albertsons.com</em></blockquote><h3>1 Introduction and Scope</h3><p>In north America, an individual usually engages with 4 devices on average (a significant increase from 2 devices in 2018). Among digitally engaged users, over 50% have 3 or more active emails. These statistics are much higher at household level. When digitally engaging or even when registering at store, these customers often don’t disclose the correct personally identifiable information (PII). At Albertsons we found that very low coverage on PIIs such as first and last names, phones, emails, addresses. The statistics on missing and invalid pii varies over time, more recent registrations have better coverage than previous years. Also the missing values across PIIs is highly skewed for store only customers than ecom customers. Moreover, the ecom users leave a rich trail of information such as cookies, browsing history, device info, ip address (location), clicks, likes of content or product they are engaging with. In addition, use of CRM software, subscription info, payments systems used in point of sale transactions (such as credit card info), first and third party delivery info (such as doordash, uber eats etc), social media data, customer surveys,and non PII data (such as transactions) adds to the complexity that makes it quite challenging for businesses when it comes to mapping these data points to customers data silos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uyb9vlM75Jggc2Ehhm5V-w.png" /></figure><p>Entity Resolution refers to process of stitching various customer info into single entity. The entity here can be an individual or a set of individual sharing certain PIIs or core attributes of the underlying set. For example in personalization such as music recommendations the entity could be an individual. An example of household level entity is where a group of individuals want to be put together in one entity such as to avail aggregated reward or loyalty points at household level. In a house, husband and wife are two different shoppers but they are part of same household. Consequently both husband and wife will have separate shopper ids but will have common household (unless they want to be put in different household).</p><p>Figure 1 gives an illustration of complexity in the entity resolution process. In this example, we know that first two records are same individual and third record is a different individual. Although this is an example of clean and complete data on all PII fields, it is often not the case in real data sets. In this example, all records share same first name. First and third record share same address whereas second and third record share same email address. The goal of entity resolution is to map first two records to same entity, and third record to a different entity</p><h4><strong>1.1 Properties of a good Entity ID</strong></h4><p>The following properties are applicable to broader context of Entity IDs ( outside the scope of this paper).</p><p><strong>1.</strong> Each Entity ID should uniquely map to an individual customer (or set of individual when entity is group of individuals).</p><p>2. Entity ID should be time invariant, i.e. it does not change over time.</p><p>3. Entity ID should be invariant to customer getting new club cards. In this case new card should be assigned the same entity id as the original one.</p><p>4. Entity ID should be invariant to customer’s updated PII at later point of time.</p><p>At Albertsons we developed the entity IDs that satisfy all of the properties outlined above.</p><h3><strong>2 Entity Resolution as building block of customer 360.</strong></h3><p>Entity Resolution is key element in building customer 360. Customer 360 is single and comprehensive view of customer’s info and their journey (that can happen through various channels), their engagement through various products and services. Customer 360 can also be thought of providing the following views of customers at one place:</p><p><strong>Who are they?</strong></p><p>The customer PII such as name, phone, email, physical address etc may be collected during sign up process. Having knowledge of name, one can deduce gender of customer to have better understanding of their products needs. Using their purchase history business can also learn if they are ecom or store only customers thus enriching info about customer’s identity.</p><p><strong>What are they doing?</strong></p><p>Knowledge of customers engagement such as their browsing and purchasing histories, use of services such as delivery enables business to understand customer and their needs at deeper level. This can be used to enhance recommendations, send better promotional offers such as sending coupons of items they might need in a near future. Business can also use these insights and the third party ( engagement with other social media apps such as twitter, facebook etc) to target potential customers.</p><p><strong>How are they doing?</strong></p><p>Knowledge of how a customer engages with products and services is very useful information. For example knowing a customer uses third party app (e.g. instacart) to order groceries, business can uses this info to improve in house delivery services. Knowledge of customers engagement with different devices can be very helpful too. For instance if in a given household, different shoppers use different devices to order groceries, it may be good idea to generate device specific recommendations using device customer’s engagement than aggregated household level. Having 360 view of customers can enable business quickly learn about customers personal preferences, having a better understanding of their products and services, offer new products (such as Albertsons Fresh Pass) and lower churn rates. For more details refer to section on use cases of customer 360.</p><h3>5 Customer 360 Use Cases</h3><p><strong>5.1 Creating Amazing Customer Experience</strong></p><p>A recent research survey4 by McKinsey &amp; Company found that ‘many organizations fail to see how improving the customer experience can create value. And more importantly despite their efforts, many struggle to understand what really drives the customer experience. The research revealed that improving the customer experience increased sales revenues by 2 to 7 percent, profitability by 1 to 2 percent, and the overall shareholder return by 7 to 10 percent.’ Customers experience is backbone of business especially the ecommerce one. Ecom business can have many challenges from incorrect product description, slow delivery, poor accessibility, and suffer from long wait for customer service. This makes customer online shopping an unpleasant one. And in fact, for many customers their frustration can lead to bad reviews among their friends and on social media channels reaching hundreds of users in no time.</p><p>Among others one of the reason Albertsons customer reach out to call center is to update their household assignment. An example of this is to merge a group of card numbers to one household to avail accumulated reward points. With over 100+ million customers, this requires significant customer service personals and thus significant cost to company. While it is desired to have a convenient and time efficient process ( such as automatic algorithmic process that can run in backend) to handle shopper and household id assignment, there is none at this time. Also the call center may not have access to other source of information such as third party data, payments data extracts PII, or even other first party data at Albertsons. Customer 360 is big win in such cases.</p><p><strong>5.2 Accelerating Businesses Digital Growth</strong></p><p>Online shopping have been on the rise for all sort of businesses in past few decades and Amazon has been a leading example of this. Online businesses are successful for many reasons such as low operating costs, easier scalability, flexible returns, quality customer support services, easier marketing and better security among many. Albertsons has made strategic investments such as building and enhancing micro fullfilment centers, fresh assortments, automation technologies, building own picking app among others. The company also made strong emphasis to online deliveries viz : ‘lovable product’, ‘fast shipping, and ‘differentiated experiences’. Explosion of social media applications (for instance reddit.com) offers better interactions among customers where they can learn about other customer’s opinion and product reviews. Such customer interactions offers challenges as well as opportunities for digital businesses. For instance by leveraging customer 360, business can quickly learn about decline in sales and address the underlying product issues. In case of boost in sales, businesses can focus on product supply, and logistics to meet product’s demand.</p><p><strong>3. Marketing</strong></p><p>The core of a great marketing strategy is to promote the brand/product through a meaningful story that connects people to its core values. With increased used of digital tools, consumers generate rich engagement history. Businesses can use this rich trail of information to create personalized marketing campaigns5 for its customers. For instance given recent purchase history of a customer, business can choose to show specific ads to the customer that has maximum opportunity of a cross sell or use to engage them with favorite influencers video/post for brand building and increased product awareness. Businesses can design better A/B testing experiments such as analyze how different segments of population would respond to new business initiatives and such.</p><p>The Albertsons companies spend $498 million in advertising cost being lowest among some top grocery retailers. The ratio of revenue to ads expenditure for Albertsons, Kroger, and Walmart are quite similar. These stores are quite similar in the sense that they have strong store as well as growing ecom presence. Since Amazon does not have as many physical store as other retailers, the company spends large sum of money on ads expenditure for brand building and for to remain a competitive business. Note that all of the sales are not driven by ads and therefore this requires careful interpretation. With customer 360, businesses can optimize the ads spending to generate higher returns and create brand value for their customers.</p><p><strong>5.4 Personalized Recommendation</strong></p><p>Imagine Netflix without its recommendation system, it is just a database of movies, not visible to naked eye. To watch a movie, you would need to precisely know its details such as title and manually search from a large database and then play. Quite inconvenient, right? Same with ecom grocery, without having a good recommendation system, the users would have to search every product they want from the pool of over half a million products. Needless to say, recommendation systems are backbone of successful ecom businesses. Top retailers like amazon attribute over 35% of revenue to successful recommendation system and uses it in every step: from product discovery to checkout. In fact for streaming businesses such as Netflix6 (over 75%), Youtube, Spotify, the majority of these company’s revenue can be tied to a very successful recommendation systems.</p><p>Recommendation systems (recsys) are also a powerful tool for product discovery: showing customers something they would like and something they won’t be able to find themselves. At the core, these recommendation algorithms are statistical machine learning models trained on customer’s past purchases, product views, likes, comments, PIIs (such as gender) and many other derived features. Armed with comprehensive view of customer experiences, customer 360 can enable business to deliver better personalized recommendations.</p><p><strong>5.5 Early Identification of churning customers</strong></p><p>For business of any size, higher churn rates can have devastating effect on its growth. With help of customer 360, businesses can consolidate customer data from various touch points including past interactions, support tickets, chat history, social media engagements and apply predictive models to infer likelihood of churn, well ahead they actually churn. This provides an early opportunity for business to make an intervention such as address any issues, offer promotional deals and incentives thus increasing customer satisfaction and increasing their lifetime value.</p><p>Above I have highlighted few of important use cases focusing on customer experience, digital growth, brand building, personalization and customer retention. Some of other use cases that business can benefit from customer 360 are aiding in regularity compliance and data security, leveraging unified data to build better products, making easier for many business and data science professionals within company to enhance their productively, among others.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52ff8fa98011" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generalized Similarity (Distance) Functions for Comparing Strings]]></title>
            <link>https://ashwinikumarmaurya.medium.com/generalized-similarity-distance-functions-for-comparing-strings-d84dac64b90a?source=rss-44fa4ce31baa------2</link>
            <guid isPermaLink="false">https://medium.com/p/d84dac64b90a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[text-similarity]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[string-similarity]]></category>
            <category><![CDATA[distance-metric]]></category>
            <dc:creator><![CDATA[Ashwini Kumar Maurya]]></dc:creator>
            <pubDate>Wed, 12 Jul 2023 16:37:57 GMT</pubDate>
            <atom:updated>2023-07-13T05:59:31.723Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is Edit Distance between two strings?</h3><p>Edit distance functions are used in many applications. For instance, in words completion, search engine (such as google) auto correction, mis-spelling detection etc are some uses cases we encounter almost every day.</p><p>Edit distance between two string is typically defined as number of insertion, deletion, substitutions, transpose etc of characters. There are different variants of the algorithms such as hamming distance (number of character positions two strings different from each other), Levenshtein distance (accounts for minimum number of insertion, deletion, and substitutions), Damerau-Levenshtien (accounts for minimum number of insertion, deletion, substitutions, and transpose), Jaro-Winkler (not a metric, is an improvement of Jaro distance/similarity as it gives more weight to strings that have larger common prefixes, based on matching count: a matching is defined if a character that appears in both string is at most max(string_1, string_2)/2) distance away). Among these distance functions, Jaro Winkler works better than others but again this depends upon application.</p><p>Below I discuss a variation of these edit distance functions which I would call as generalized distance (similarity) functions.</p><h3>Generalized Similarity (distance) Function</h3><p>Intuition: Typically different parts of speech of same words differ towards the end of the string than at the beginning. Also misspelled words often are misspelled towards at middle or end of string. Therefore given two strings, more they match from the beginning, more likely they are similar words than if they match from end. The generalized distance/similarity functions computes the distance between each substring from left, and then mean of all these distances. One can also use different weights for substrings, but again if that would be useful for a particular uses cases.</p><h3>Here is Python implementation:</h3><pre>def get_generalized_DamerauLevenshtein_similarity(s1,s2):<br>     &quot;&quot;&quot;<br>     Compute the generalized DamerauLevenshtein similarity between two strings<br>     &quot;&quot;&quot;<br>     <br>     from itertools import zip_longest<br>     import rapidfuzz<br>     if s1==&#39;&#39; or s2==&#39;&#39;:<br>         return 0<br>     <br>     s1=s1.lower()<br>     s2=s2.lower()<br>     t1=&#39;&#39;<br>     t2=&#39;&#39;<br>     <br>     simi=0; cnt=0<br>     for a,b in zip_longest(s1,s2):<br>         if a:<br>             t1=t1+a<br>         if b:<br>             t2=t2+b<br>         cnt+=1<br>         simi+=rapidfuzz.distance.DamerauLevenshtein.normalized_similarity(t1,t2)<br>         <br>     return simi/cnt<br><br><br>def get_generalized_jaro_winkler_similarity(s1,s2):<br>    <br>     &quot;&quot;&quot;<br>     Compute the generalized Jaro Winkler similarity between two strings<br>     &quot;&quot;&quot;<br><br>    from itertools import zip_longest<br>    import numpy as np<br>    if s1==&#39;&#39; or s2==&#39;&#39;:<br>        return 0<br>    s1=s1.lower()<br>    s2=s2.lower()<br>    t1=&#39;&#39;<br>    t2=&#39;&#39;<br>    <br>    simi=[]; cnt=0<br>    for a,b in zip_longest(s1,s2):<br>        if a:<br>            t1=t1+a<br>        if b:<br>            t2=t2+b<br>        cnt+=1<br>        simi.append(rapidfuzz.distance.JaroWinkler.similarity(t1,t2))<br>        <br>    return sum(simi)/cnt</pre><h3>Example 1:</h3><p>### strings with similar in meaning that differ at end of string</p><pre>import rapidfuzz, numpy as np<br><br>s1=&#39;dummy&#39;<br>s2=&#39;dummies&#39;<br><br>print(f&#39;Hamming similarity is :                           {np.round(rapidfuzz.distance.Hamming.normalized_similarity(s1,s2),3)}&#39;)<br>print(f&#39;Levenshtein similarity is :                       {np.round(rapidfuzz.distance.Levenshtein.normalized_similarity(s1,s2),3)}&#39;)<br>print(f&#39;Damerau-Levenshtein similairty is :               {np.round(rapidfuzz.distance.DamerauLevenshtein.normalized_similarity(s1,s2),3)}&#39;)<br>print(f&#39;Generalized Damerau-Levenshtein similairty is :   {np.round(get_generalized_DamerauLevenshtein_similarity(s1, s2),3)}\n&#39;)<br><br>print(f&#39;Jaro Winkler Similairty is :                      {np.round(rapidfuzz.distance.JaroWinkler.normalized_similarity(s1,s2),3)}&#39;)<br>print(f&#39;Generalized Jaro Winkler Similairty is :          {np.round(get_generalized_jaro_winkler_similarity(s1,s2),3)}&#39;)<br><br>Hamming similarity is :                           0.571<br>Levenshtein similarity is :                       0.571<br>Damerau-Levenshtein similairty is :               0.571<br>Generalized Damerau-Levenshtein similairty is :   0.863<br><br>Jaro Winkler Similairty is :                      0.874<br>Generalized Jaro Winkler Similairty is :          0.955<br></pre><p>for similar meaning strings, generalized similarity works quite well in this example. Generalized Jaro Winkler similarity seems to better than standard Jaro Winkler similarity (JW is skewed towards strings having larger common prefix).</p><h3>Example 2:</h3><p>### homonym strings (spelled similar but different in meaning)</p><pre>  s1=&#39;faithful&#39;<br>  s2=&#39;fruitful&#39;<br>  <br>  print(f&#39;Hamming similarity is :                           {np.round(rapidfuzz.distance.Hamming.normalized_similarity(s1,s2),3)}&#39;)<br>  print(f&#39;Levenshtein similarity is :                       {np.round(rapidfuzz.distance.Levenshtein.normalized_similarity(s1,s2),3)}&#39;)<br>  print(f&#39;Damerau-Levenshtein similairty is :               {np.round(rapidfuzz.distance.DamerauLevenshtein.normalized_similarity(s1,s2),3)}&#39;)<br>  print(f&#39;Generalized Damerau-Levenshtein similairty is :   {np.round(get_generalized_DamerauLevenshtein_similarity(s1, s2),3)}\n&#39;)<br><br>  print(f&#39;Jaro Winkler Similairty is :                      {np.round(rapidfuzz.distance.JaroWinkler.normalized_similarity(s1,s2),3)}&#39;)<br>  print(f&#39;Generalized Jaro Winkler Similairty is :          {np.round(get_generalized_jaro_winkler_similarity(s1,s2),3)}&#39;)<br>  <br>  Hamming similarity is :                           0.5<br>  Levenshtein similarity is :                       0.625<br>  Damerau-Levenshtein similairty is :               0.625<br>  Generalized Damerau-Levenshtein similairty is :   0.522<br>  <br>  Jaro Winkler Similairty is :                      0.85<br>  Generalized Jaro Winkler Similairty is :          0.766</pre><p>for similar spelled words with different meaning, generalized similarity works quite well in this example.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d84dac64b90a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scrapping Linkedin]]></title>
            <link>https://ashwinikumarmaurya.medium.com/scrapping-linkedin-872ab795b4df?source=rss-44fa4ce31baa------2</link>
            <guid isPermaLink="false">https://medium.com/p/872ab795b4df</guid>
            <category><![CDATA[python-web-scraper]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[web-scrapping]]></category>
            <dc:creator><![CDATA[Ashwini Kumar Maurya]]></dc:creator>
            <pubDate>Sun, 03 Jan 2021 23:53:24 GMT</pubDate>
            <atom:updated>2021-01-03T23:53:24.662Z</atom:updated>
            <content:encoded><![CDATA[<p>Scrapping linkedin could be very beneficial for professional looking to find new opportunities, for employers to identify the potential employee churn or head hunters to find the talent they are looking for. Following the lawsuit between <a href="https://mlexmarketinsight.com/insights-center/editors-picks/antitrust/north-america/linkedin,-hiq-spar-at-ninth-circuit-in-data-scraping-case">HiQ Labs v LinkedIn</a> which favoured <a href="https://mlexmarketinsight.com/insights-center/editors-picks/antitrust/north-america/linkedin,-hiq-spar-at-ninth-circuit-in-data-scraping-case">HiQ Labs</a>, Linkedin allows scrapping the data from its website.</p><p>For this story, I use Selenium to scrape the linkedin data as its easy to automate the process when extracting data for multiple profiles.</p><p><strong>Necessary Software</strong></p><p>To use selenium, you must have installed the WebDriver. <a href="https://chromedriver.chromium.org/downloads">Chrome Webdriver</a> is good choice.</p><pre>import string, pandas as pd, os, sys, json, time, re, itertools<br>from parsel import Selector<br>from collections import deque</pre><pre>from selenium.webdriver.common.keys import Keys<br>from selenium import webdriver<br>from selenium.webdriver.chrome.options import Options<br>from selenium.webdriver.common.by import By<br>from selenium.webdriver.support.ui import WebDriverWait<br>from selenium.webdriver.support import expected_conditions as EC<br>from selenium.common.exceptions import TimeoutException, WebDriverException</pre><pre>options = Options()<br>options.add_argument(&quot;start-maximized&quot;)<br>options.add_argument(&quot;disable-infobars&quot;)<br>options.add_argument(&quot;--disable-extensions&quot;)</pre><pre>pd.set_option(&#39;display.max.rows&#39;,1000)<br>pd.set_option(&#39;display.max.columns&#39;,1000)</pre><p><strong>Setting up driver</strong></p><p>I wrote the following function to get the driver logged into to linked account. Login in necessary to browse the profile details.</p><pre>def get_driver_details():<br> driver = webdriver.Chrome(executable_path=’/Users/amaur24/Desktop/scrape/chromedriver’)<br> driver.get(‘<a href="https://www.linkedin.com/uas/login?session_redirect=%2Fvoyager%2FloginRedirect%2Ehtml&amp;amp;fromSignIn=true&amp;amp;trk=uno-reg-join-sign-in&#39;">https://www.linkedin.com/uas/login?session_redirect=%2Fvoyager%2FloginRedirect%2Ehtml&amp;amp;fromSignIn=true&amp;amp;trk=uno-reg-join-sign-in</a>&#39;)<br> try:<br> <a href="/service/mailto:userid=&#39;akmaurya07@gmail.com">userid=&#39;u</a>ser_name&#39;<br> pwd=&#39;password&#39;<br> WebDriverWait(driver, 10).until(<br> EC.presence_of_element_located((By.XPATH, &#39;//*[<a href="http://twitter.com/id">@id</a>=&quot;username&quot;]&#39;))).send_keys(userid)<br> WebDriverWait(driver, 10).until(<br> EC.presence_of_element_located((By.XPATH, &#39;//*[<a href="http://twitter.com/id">@id</a>=&quot;password&quot;]&#39;))).send_keys(pwd)<br> log_in_button = WebDriverWait(driver, 10).until(<br> EC.presence_of_element_located((By.XPATH, &#39;//*[<a href="http://twitter.com/id">@id</a>=&quot;app__container”]/main/div[2]/form/div[3]/button&#39;)))<br> log_in_button.click()<br> <br> except Exception:<br> driver.quit()<br> <br> return driver</pre><p>driver=get_driver_details()</p><p><strong>Google Search of Profiles</strong></p><p>One can search linkedin profiles using google search given name of skill/title , location etc. For instance the following google search query is returns result for linkedin data scientist users from San Francisco.</p><pre>search_query=’site:linkedin.com/in/ AND ‘+&#39;Data Scientist&#39;+’ AND ‘+&#39;San Francisco&#39;</pre><p>The function below automates this search and returns the linkedin web link of the searched profiles:</p><pre>def get_profile_links(driver,skill,location,search_pages=None):<br> driver.get(‘<a href="https://www.google.com&/#39;">https://www.google.com&#39;</a>)<br> query=driver.find_element_by_name(‘q’)<br> search_query=’site:linkedin.com/in/ AND ‘+skill+’ AND ‘+location<br> query.send_keys(search_query)<br> query.send_keys(Keys.RETURN)<br> <br> counter = 1;<br> all_links=[]<br> <br> if search_pages is None:<br> search_pages=1<br> while counter &lt;= search_pages:<br> try:<br> driver.find_element_by_xpath(“//*[contains(local-name(), ‘span’) and contains(text(), ‘Next’)]”).click()<br> time.sleep(1)<br> linkedin_urls = driver.find_elements_by_class_name(‘yuRUbf’)<br> ps=driver.page_source<br> search_results=ps.split(“Search Result”)[1]<br> profile_links=search_results.split(‘yuRUbf’)[1:]<br> all_links=all_links+ profile_links<br> counter = counter+1<br> except Exception:<br> pass<br> return all_links</pre><pre>def get_profile_lists(profile_links):<br>    <br>    profile_lists=[]<br>    <br>    for i in range(len(profile_links)):<br>        text=profile_links[i]<br>        text=text[text.find(&quot;https:&quot;):]<br>        for c in url_permissible_chars:<br>            if c in text:<br>                text=text.split(c)[0]<br>        profile_name=text.split(&quot; &quot;)[0].strip().replace(&#39;&quot;&#39;,&quot;&quot;).replace(&quot;&#39;&quot;,&quot;&quot;)<br>        profile_lists.append(profile_name)<br>    return profile_lists</pre><p><strong>Fetch Links: </strong><br>skill=’Data Scientist’; location=San Francisco’<br>profiles=get_profile_lists(get_profile_links(driver=driver,skill=skill,location=location,search_pages=10))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*j6e_7r8nfqaeac6QvGS4Tg.png" /></figure><p><strong>Fetching Profile Data:</strong></p><p>Next we can define a customized functions to fetch section of profile. For instance the following function extracts names of skills listed on Linkedin profile.</p><pre>def get_data_from_FullProfileSkill(D):<br> <br> result=dict()<br> try:<br> if ‘name’ in D:<br> skill_name=D[‘name’]<br> else:<br> skill_name=’’<br> except Exception:<br> skill_name=’’<br> <br> result.update({‘skill_name’:skill_name})<br> return result</pre><p><strong>Function to get the JSON text and relevant Elements</strong></p><pre>## define a stack to get only json string from the page_source</pre><pre>def get_json_text(text):<br> S=[]<br> D=deque()<br> D.append(‘{‘) ## first { character<br> S.append(‘{‘)<br> for c in text[1:]:<br> if c==’{‘:<br> D.append(‘{‘)<br> if c==’}’:<br> D.pop()<br> S.append(c)<br> if len(D)==0:<br> S1=””.join([x for x in S])<br> return json.loads(S1)<br> return print(“not json string”)</pre><pre>### get relevant profile data dicts: there are many elements which are not relevant to profile. We can ignore these.</pre><pre>def get_relevant_list_of_dict(L):<br> L_new=[]<br> for i in range(len(L)):<br> if (‘paging’ in L[i]) or (‘urn:li:fsd_company’ in L[i][‘entityUrn’]):<br> pass<br> else:<br> L_new.append(L[i])<br> return L_new</pre><p><strong>Function to parse the skills data from the profile page and return the dataframe.</strong></p><pre>def get_parsed_data(D):</pre><pre>“””Given the dictionary of data, parses and returns the relevant details”””<br> <br> result=dict()<br> if isinstance(D[‘$recipeTypes’],list):<br> recipeTypes=D[‘$recipeTypes’][0]<br> else:<br> recipeTypes=D[‘$recipeTypes’]</pre><pre>if recipeTypes==’com.linkedin.voyager.dash.deco.identity.profile.FullProfileSkill’:<br> result=get_data_from_FullProfileSkill(D)<br> <br> return result</pre><pre>def get_profile_df(profile_data):<br>    <br>    &quot;&quot;&quot;Returns a pandas dataframe of profile details&quot;&quot;&quot;</pre><pre>c_names=[&#39;first_name&#39;,&#39;last_name&#39;,&#39;job_profile_headline&#39;,&#39;skills&#39;,&#39;industry_experiences&#39;,&#39;certificates&#39;,&#39;profile_summary&#39;,&#39;current_location_city&#39;,&#39;current_country_name&#39;,&#39;fields_of_study&#39;,&#39;degree_names&#39;,&#39;school_names&#39;]<br>    final_df=pd.DataFrame(columns=c_names)</pre><pre>result=pd.DataFrame()</pre><pre>for info in profile_data:<br>    parsed_info=get_parsed_data(info)<br>    result=result.append(parsed_info,ignore_index=True)</pre><pre>if &#39;skill_name&#39; in result.columns:<br>    skill_names=result.loc[~(result.skill_name.isna())].skill_name<br>    if any(skill_names):<br>        skills=[x for x in skill_names if (isinstance(x,str)) and len(x)&gt;0]<br>       if len(skills)&gt;1:<br>           skills=&quot;,&quot;.join(skills)<br>       elif len(skills)==1:<br>           skills=skills[0]<br>       else:<br>           skills=&#39;&#39;<br>    else:<br>        skills=&#39;&#39;<br>else:<br>    skills=&#39;&#39;</pre><pre>result[&#39;skills&#39;]=skills<br>if result.shape[0]&gt;0:<br>        final_df=result[c_names].drop_duplicates()<br>    <br>    return final_df</pre><p><strong>Finally get the result</strong></p><pre>prof_page=driver.get(profiles[0])<br>profile_page_source=driver.page_source<br>kwd_data=’{“data”:{“entityUrn”’;<br>data_strings=[kwd_data+x for x in profile_page_source.split(kwd_data)]</pre><pre>pos=-1<br>for i in range(len(data_strings)):<br>if data_strings[i].find(‘com.linkedin.voyager.dash.deco.identity.profile.FullProfileWithEntities’)&gt;0:<br> pos=i<br> if pos&gt;-1:<br> profile_info=get_json_text(data_strings[pos])<br> profile_data_all=profile_info[‘included’]<br> profile_data=get_relevant_list_of_dict(profile_data_all)<br> df=get_profile_df(profile_data)<br> </pre><p><strong>Here is the profile page:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*CSz10diMyS8SjP4gdcOeLQ.png" /></figure><p><strong>Skills listed on profile:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*6k9BYTfv8ntEk-fuYUj_4w.png" /></figure><p><strong>Here is skill names fetched using code:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*0BV_Gu8K4KpHYOMy7KUn3Q.png" /></figure><p><strong>Or just skills:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEY3KwthARhE2RL88vAQpg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=872ab795b4df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python script to automatically download trading data from binance]]></title>
            <link>https://ashwinikumarmaurya.medium.com/python-script-to-automatically-download-trading-data-from-binance-c85ba25b60af?source=rss-44fa4ce31baa------2</link>
            <guid isPermaLink="false">https://medium.com/p/c85ba25b60af</guid>
            <category><![CDATA[binance]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[cryptocurrency-investment]]></category>
            <category><![CDATA[trading]]></category>
            <dc:creator><![CDATA[Ashwini Kumar Maurya]]></dc:creator>
            <pubDate>Sun, 16 Aug 2020 06:35:31 GMT</pubDate>
            <atom:updated>2021-03-07T07:01:54.266Z</atom:updated>
            <content:encoded><![CDATA[<p>If you are into cryptocurrency trading, you must be familiar with high volatility of cryptocurrency coins. To place an order, one needs to quickly derive insights by analyzing the realtime trading data. While making an intelligent trading bot is definitely lots of work and involves number of steps. I will talk about some of my insights in a series of posts. Before we get into that, first I will talk about how we can automatically download realtime trading data using Binance API.</p><p>While OHLC data can be download for past many time points at various intervals, the trading data i.e. buy/sell price and quantity can be downloaded for maximum of 500 data points. Therefore if one wants to download trading data for some time, it is good idea to automate the code and keep your code running on a machine. One good place is AWS EC2 and S3 for cloud computing and storage. Usually an instance with 8 vCPUs like c4.2xlarge is quite sufficient. I do not recommend the free tier t2.micro as it often runs into issues. I have tried for quite few times and every time job gets automatically killed with varying exceptions.</p><p>Below is the code with some descriptions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa9417948be96a33bca34deee05deb20/href">https://medium.com/media/aa9417948be96a33bca34deee05deb20/href</a></iframe><p>To automate the downloading of trading data, just run this code as .py file from terminal.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c85ba25b60af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Downloading CoinMarketCap Cryptocurrency Info]]></title>
            <link>https://ashwinikumarmaurya.medium.com/downloading-coinmarketcap-cryptocurrency-info-64020e0e063d?source=rss-44fa4ce31baa------2</link>
            <guid isPermaLink="false">https://medium.com/p/64020e0e063d</guid>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[coinmarketcap]]></category>
            <category><![CDATA[bitcoin]]></category>
            <dc:creator><![CDATA[Ashwini Kumar Maurya]]></dc:creator>
            <pubDate>Wed, 05 Aug 2020 04:06:14 GMT</pubDate>
            <atom:updated>2020-08-05T04:19:43.393Z</atom:updated>
            <content:encoded><![CDATA[<h3>Downloading CoinMarketCap Cryptocurrency Info Using Python</h3><p>Founded in 2013 by Brandon Chez, <a href="https://coinmarketcap.com/">coinmarketcap</a> has grown into one of the most popular source of information about price movement of cryptocurrencies. The website has listing of over 2000 cryptocurrency has most of the data is up to date. Since they list cryptocurrency info aggregated from various markets and many other sources, it is often a good source to get most updated data.</p><p>One need to register and obtain <a href="https://coinmarketcap.com/api/">API key</a> to queries against their database.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e51881b651078214fdacccb07ab55d1/href">https://medium.com/media/0e51881b651078214fdacccb07ab55d1/href</a></iframe><p>Here is the sample of output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_86EU4n1qOBiIJBHUIZsTg.png" /><figcaption>Fig: Sample of cryptocurrencies market data (source coinmarketcap.com)</figcaption></figure><p>Next I will share some plots which can be used to gain insights about price movement of cryptocurrencies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64020e0e063d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Downloading Binance Data: OHLC data]]></title>
            <link>https://ashwinikumarmaurya.medium.com/downloading-binance-data-ohlc-data-6d3cb42f1695?source=rss-44fa4ce31baa------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d3cb42f1695</guid>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[binance]]></category>
            <dc:creator><![CDATA[Ashwini Kumar Maurya]]></dc:creator>
            <pubDate>Mon, 03 Aug 2020 07:24:38 GMT</pubDate>
            <atom:updated>2020-08-16T20:07:43.497Z</atom:updated>
            <content:encoded><![CDATA[<h3>Downloading Cryptocurrency OHLC Data Using Binance’s Python API</h3><p>Binance API makes it easy to download the trading data. While you can download most of historical data like OHLC (open, high, low, close etc), recent trades, order books without providing your API keys, for orders related to your account such as deposit, withdrawal etc you need to provide your API secret key.</p><p>As you may be aware, Bitcoin and other alt coins change price trajectory quite frequently and that is 24 hours and 7 days a week. It is not uncommon to be over excited and start contemplating ideas with profitable strategy. While it is true that the cryptocurrency trading offers unique opportunity to make good money in short term but same time if not traded carefully, it can drive your entire portfolio to zero.</p><p>While these cryptocurrencies exchanges are not regulated, there is good amount of manipulation. The price is often driven by speculation and that makes it quite dangerous to trade. However I do believe that one can always look into the data to derive better trading strategies.</p><p>Over the past few weeks, I have invested some time to automate some of data download and generating plots to gain insights. I would like to share some in series of posts.</p><p>In a first post, we start with downloading the historical klines using <a href="https://python-binance.readthedocs.io/en/latest/binance.html">binance api </a>. The below function reshapes the data as pandas data frame object .</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/246bb040709b6bb7ad8e1c017abb3d6c/href">https://medium.com/media/246bb040709b6bb7ad8e1c017abb3d6c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zVXSDQMR_fy8mNhK0yPu6w.png" /><figcaption>Fig: Sample output of downloaded historical klines.</figcaption></figure><p>In next post, we will discuss about how to cryptocurrencies info on price, circulating supply, market cap from Coinmarketcap.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d3cb42f1695" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>