<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vikash Singh on Medium]]></title>
        <description><![CDATA[Stories by Vikash Singh on Medium]]></description>
        <link>https://medium.com/@vicky301186?source=rss-ea2f87aaedec------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5OF8wCwJiGfnUOrv2zYOjA.jpeg</url>
            <title>Stories by Vikash Singh on Medium</title>
            <link>https://medium.com/@vicky301186?source=rss-ea2f87aaedec------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:32:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vicky301186/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stream Metrics]]></title>
            <link>https://medium.com/@vicky301186/stream-metrics-4feeffd4857f?source=rss-ea2f87aaedec------2</link>
            <guid isPermaLink="false">https://medium.com/p/4feeffd4857f</guid>
            <category><![CDATA[real-time-streaming-data]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <dc:creator><![CDATA[Vikash Singh]]></dc:creator>
            <pubDate>Sun, 27 Aug 2023 08:41:36 GMT</pubDate>
            <atom:updated>2023-08-27T08:41:36.796Z</atom:updated>
            <content:encoded><![CDATA[<p>Stream metrics enable to define metrics on stream(Kafka) using Sql.<br>It has three components:</p><ul><li>stream: represents a continuous unbounded flow of data into a Kafka topic</li><li>sql: flink sql query to define metrics</li><li>sinks: where processed data or metrics are sent for further processing, storage, visualization, or analysis</li></ul><h4>Stream</h4><blockquote>Let’s say we have a kafka topic user_events with events looks like</blockquote><blockquote>{&quot;user_id&quot;: &quot;952483&quot;, &quot;item_id&quot;:&quot;310884&quot;, &quot;category_id&quot;: &quot;4580532&quot;, &quot;behavior&quot;: &quot;pv&quot;, &quot;ts&quot;: 1693122742}<br>{&quot;user_id&quot;: &quot;794777&quot;, &quot;item_id&quot;:&quot;5119439&quot;, &quot;category_id&quot;: &quot;982926&quot;, &quot;behavior&quot;: &quot;pv&quot;, &quot;ts&quot;: 1693122842}</blockquote><blockquote>Register stream as a table</blockquote><blockquote>We need following configuration to register topic:</blockquote><blockquote>SOURCE_KAFKA_TOPIC_NAMES : kafka topic name, in this case it’s user_events</blockquote><blockquote>INPUT_SCHEMA_TABLE : table name to be used for sql, in this case it’s user_behavior</blockquote><blockquote>INPUT_DATATYPE : datatype of records in kafka topic, in this it’s JSON</blockquote><blockquote>INPUT_SCHEMA_JSON_SCHEMA: json schema of records</blockquote><pre>{ &quot;$schema&quot;: &quot;http://json-schema.org/draft-04/schema#&quot;, &quot;type&quot;: &quot;object&quot;, &quot;properties&quot;: { &quot;user_id&quot;: { &quot;type&quot;: &quot;string&quot; }, &quot;item_id&quot;: { &quot;type&quot;: &quot;string&quot; }, &quot;category_id&quot;: { &quot;type&quot;: &quot;string&quot; }, &quot;behavior&quot;: { &quot;type&quot;: &quot;string&quot; }, &quot;ts&quot;: { &quot;type&quot;: &quot;integer&quot; } }, &quot;required&quot;: [ &quot;user_id&quot;, &quot;item_id&quot;, &quot;category_id&quot;, &quot;behavior&quot;, &quot;ts&quot; ] }</pre><blockquote>INPUT_SCHEMA_JSON_EVENT_TIMESTAMP_FIELD_NAME : field in record capture event time as epoch in second, in this case it’s ts</blockquote><blockquote>INPUT_SCHEMA_EVENT_TIMESTAMP_FIELD_INDEX : index of event time column in record, in this case it’s 4</blockquote><blockquote>SOURCE_KAFKA_CONSUMER_CONFIG_BOOTSTRAP_SERVERS : kafka bootstrap server</blockquote><blockquote>SOURCE_KAFKA_CONSUMER_CONFIG_AUTO_COMMIT_ENABLE : enable auto offset commit</blockquote><blockquote>SOURCE_KAFKA_CONSUMER_CONFIG_AUTO_OFFSET_RESET : configuration parameter to decide where to start consuming messages from</blockquote><blockquote>SOURCE_KAFKA_CONSUMER_CONFIG_GROUP_ID : kafka consumer group id</blockquote><h4>Sql query to define metric</h4><blockquote>The hourly trading volume is the number of “buy” behaviors completed each hour. Therefore, we can use a TUMBLE window function to assign data into hourly windows. Then, we count the number of “buy” records in each window. To implement this, we can filter out the “buy” data first and then apply COUNT(*).</blockquote><pre>SELECT <br>        HOUR(TUMBLE_START(rowtime, INTERVAL &#39;1&#39; HOUR)),<br>        COUNT(*)<br>        from user_behavior<br>        WHERE behavior = &#39;buy&#39;  <br>        GROUP BY TUMBLE (rowtime, &#39;1&#39; HOUR)</pre><h4>Register sink</h4><blockquote>here we are registering influx to capture hourly buy trend and need following configuration:</blockquote><blockquote>SINK_TYPE=influx</blockquote><blockquote>SINK_INFLUX_URL=<a href="http://localhost:8086/">http://localhost:8086</a></blockquote><blockquote>SINK_INFLUX_ORG=home</blockquote><blockquote>SINK_INFLUX_BUCKET=temp</blockquote><blockquote>SINK_INFLUX_TOKEN= **********</blockquote><blockquote>SINK_INFLUX_BATCH_SIZE=100</blockquote><blockquote>SINK_INFLUX_FLUSH_DURATION_MS=5000</blockquote><blockquote>SINK_INFLUX_MEASUREMENT_NAME=orders_metrics</blockquote><blockquote>SINK_INFLUX_DB_NAME=metrics</blockquote><h3>All pieces together</h3><pre># == Query ==<br>FLINK_SQL_QUERY=SELECT HOUR(TUMBLE_START(rowtime, INTERVAL &#39;1&#39; HOUR)), COUNT(*) from user_behavior WHERE behavior = &#39;buy&#39; GROUP BY TUMBLE (rowtime, &#39;1&#39; HOUR)<br>FLINK_WATERMARK_INTERVAL_MS=60000<br>FLINK_WATERMARK_DELAY_MS=10000<br># == Input Stream ==<br>STREAMS=[{&quot;SOURCE_KAFKA_TOPIC_NAMES&quot;:&quot;user_events&quot;,&quot;INPUT_SCHEMA_TABLE&quot;:&quot;user_behavior&quot;,&quot;INPUT_DATATYPE&quot;:&quot;JSON&quot;,&quot;INPUT_SCHEMA_JSON_SCHEMA&quot;:&quot;{\\&quot;$schema\\&quot;:\\&quot;http://json-schema.org/draft-04/schema#\\&quot;,\\&quot;type\\&quot;:\\&quot;object\\&quot;,\\&quot;properties\\&quot;:{\\&quot;user_id\\&quot;:{\\&quot;type\\&quot;:\\&quot;string\\&quot;},\\&quot;item_id\\&quot;:{\\&quot;type\\&quot;:\\&quot;string\\&quot;},\\&quot;category_id\\&quot;:{\\&quot;type\\&quot;:\\&quot;string\\&quot;},\\&quot;behavior\\&quot;:{\\&quot;type\\&quot;:\\&quot;string\\&quot;},\\&quot;ts\\&quot;:{\\&quot;type\\&quot;:\\&quot;integer\\&quot;}},\\&quot;required\\&quot;:[\\&quot;user_id\\&quot;,\\&quot;item_id\\&quot;,\\&quot;category_id\\&quot;,\\&quot;behavior\\&quot;,\\&quot;ts\\&quot;]}&quot;, &quot;INPUT_SCHEMA_JSON_EVENT_TIMESTAMP_FIELD_NAME&quot;: &quot;ts&quot;, &quot;INPUT_SCHEMA_EVENT_TIMESTAMP_FIELD_INDEX&quot;:&quot;4&quot;, &quot;SOURCE_KAFKA_CONSUMER_CONFIG_BOOTSTRAP_SERVERS&quot;:&quot;localhost:9092&quot;,&quot;SOURCE_KAFKA_CONSUMER_CONFIG_AUTO_COMMIT_ENABLE&quot;:&quot;false&quot;,&quot;SOURCE_KAFKA_CONSUMER_CONFIG_AUTO_OFFSET_RESET&quot;:&quot;earliest&quot;,&quot;SOURCE_KAFKA_CONSUMER_CONFIG_GROUP_ID&quot;:&quot;group1&quot;,&quot;SOURCE_KAFKA_NAME&quot;:&quot;local-kafka-stream&quot;,&quot;SOURCE_DETAILS&quot;:[{&quot;SOURCE_TYPE&quot;:&quot;UNBOUNDED&quot;,&quot;SOURCE_NAME&quot;:&quot;KAFKA_CONSUMER&quot;}]}]<br># == Sink ==<br>SINK_TYPE=influx<br>SINK_INFLUX_URL=http://localhost:8086<br>SINK_INFLUX_ORG=home<br>SINK_INFLUX_BUCKET=temp<br>SINK_INFLUX_TOKEN=*****<br>SINK_INFLUX_BATCH_SIZE=100<br>SINK_INFLUX_FLUSH_DURATION_MS=5000<br>SINK_INFLUX_MEASUREMENT_NAME=orders_metrics<br>SINK_INFLUX_DB_NAME=metrics</pre><p><a href="https://github.com/singhvikash11/stream-metrics">https://github.com/singhvikash11/stream-metrics</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4feeffd4857f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RapideX-Experiments in Rapido at a scale]]></title>
            <link>https://medium.com/rapido-labs/rapidex-experiments-in-rapido-at-a-scale-e5f92bdc203b?source=rss-ea2f87aaedec------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5f92bdc203b</guid>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Vikash Singh]]></dc:creator>
            <pubDate>Thu, 08 Jun 2023 07:40:40 GMT</pubDate>
            <atom:updated>2023-06-08T07:46:51.434Z</atom:updated>
            <content:encoded><![CDATA[<h4>What’s an A/B experiment and its significance:</h4><p>A/B experiments, also known as A/B testing or split testing, are common technique used in software development to compare two or more variations of a product or feature. In an A/B experiment, users are randomly divided into groups, and each group is exposed to a different version of the product or feature. By comparing the outcomes and user behaviors of each group, developers can evaluate which variation performs better and make data-driven decisions about the product’s design, functionality, or user experience.</p><p>The importance of A/B experiments in software development lies in their ability to provide objective insights into the effectiveness and impact of changes. Here are some key reasons why A/B experiments are valuable:</p><ol><li><strong>Data-driven decision-making:</strong> A/B experiments help eliminate subjective opinions and rely on concrete data to make decisions.</li><li><strong>Iterative improvement:</strong> A/B experiments facilitate a continuous improvement process.</li><li><strong>Risk reduction:</strong> A/B experiments provide a controlled environment for testing new ideas or features before rolling them out to a wider audience.</li><li><strong>Personalization and customization:</strong> A/B experiments can help tailor experiences to different user segments.</li><li><strong>Cost-effectiveness:</strong> A/B experiments allow developers to test and validate hypotheses with relatively low cost and effort compared to fully implementing and deploying a new feature.</li></ol><p>Overall, A/B experiments provide a systematic approach to improving software products by using empirical evidence. They enable developers to make informed decisions, reduce risks, and deliver better user experiences.</p><h4>How we model experiments into Rapido:</h4><p>We model our experiment platform into the following unit:</p><blockquote>Experimental Lever/Design variation:</blockquote><p>These are different experimental levers or variations we want to test in our A/B experiment. These can be variations of features, designs, pricing models, etc.<br>Modify our codebase or system to expose the experimental levers to our users or target audiences.</p><blockquote>Experiment/Hypothesis:</blockquote><p>An experiment refers to the process of comparing two or more variations of a product or feature to determine which performs better. Each variation represents a different hypothesis that developers aim to test and evaluate based on user data.<br>We model variations into kinds of rules like belonging to an experimental lever, a target population, a split users strategy, day and time condition, data collection, and configs as part of the variation.</p><blockquote>Execution/Experiment run:</blockquote><p>In an A/B experiment, a “run” refers to the execution or implementation of the experiment with the specified variations and test conditions. Here are the key steps involved in running an A/B experiment:</p><ol><li><strong>Design variations:</strong> The experiment starts with designing the different variations of the product or feature that will be tested.</li><li><strong>Randomization and assignment:</strong> Users participating in the experiment are randomly divided into groups, with each group assigned to a specific variation.</li><li><strong>Implementation:</strong> The variations are implemented in the software or system, either through code changes, configuration updates, or any other necessary means.</li><li><strong>Data collection:</strong> During the run of the experiment, data is collected on user interactions, behaviors, and desired metrics.</li></ol><p>We have modeled <strong>design variation</strong>, r<strong>andomization and assignment, implementation, and data collection </strong>in the creation of experiments. <br>However, we have introduced geo-temporal factors, overridden time-condition, and populated configs at the execution layer.</p><blockquote>Execution lifecycle:</blockquote><p>schedule -&gt; run -&gt; pause/resume -&gt; terminated</p><blockquote>Geo variation modelling:</blockquote><p>We modeled geo variation as a route from a source to a destination</p><h4>RapideX experiment playbook:</h4><p>Here we showcase a simple experiment using RapideX.</p><p><strong>Experiment lever:</strong> Banner shows on the App screen and we can identify this as <strong>visitor-webpage-banner. </strong>On the app screen, we would integrate to find configured experiment:</p><pre>bannerLever = RapideX.getExperiment(&quot;visitor-webpage-banner&quot;)<br>bannerText = bannerLevel<br>.payload({&quot;user_id&quot;, &quot;123&quot;})<br>.apply({&quot;source&quot;:&quot;bangalore&quot;, &quot;destination&quot;:&quot;bangalore&quot;})<br>.config(&quot;banner&quot;)</pre><p><strong>Experiment: </strong>A curl request to configure an experiment with a combination of the following rules:</p><ul><li>Days: Monday, Tuesday, and Wednesday</li><li>Dates: 1st February 2023 to 1st March 2023</li><li>Time: 8.00 AM to 7.00 PM</li><li>Population Condition: Access Device should Mobile</li><li>Locations(Test): Bangalore</li></ul><pre>{<br>    &quot;name&quot;: &quot;greeter_experiment&quot;,<br>    &quot;actor&quot;: &quot;visitor&quot;,<br>    &quot;domain&quot;: &quot;webpage&quot;,<br>    &quot;feature&quot;: &quot;banner&quot;,<br>    &quot;time_condition&quot;: &quot;$weekday in [&#39;Monday&#39;, &#39;Tuesday&#39;, &#39;Wednesday&#39;] and $hour in 8..19&quot;,<br>    &quot;population_condition&quot;: &quot;$payload.device_type == &#39;Mobile&#39;&quot;,<br>    &quot;split_strategy&quot;: &quot;EvenAttributeAsTest&quot;,<br>    &quot;split_attribute&quot;: &quot;$payload.user_id&quot;,<br>    &quot;test_control&quot;: true,<br>    &quot;config&quot;: {<br>        &quot;params&quot;: [<br>            {<br>                &quot;name&quot;: &quot;banner&quot;,<br>                &quot;type&quot;: &quot;string&quot;,<br>                &quot;struct&quot;: null<br>            }<br>        ],<br>        &quot;schema&quot;: {<br>            &quot;id&quot;: &quot;user_id&quot;,<br>            &quot;device_type&quot;: &quot;device_type&quot;<br>        }<br>    }<br>}</pre><p><strong>Execution:</strong> A curl request to schedule execution on Bangalore from 1st May to 15th May</p><pre>{<br>  &quot;name&quot;: &quot;greeter_execution&quot;,<br>  &quot;locations&quot;: [<br>    {<br>      {<br>       &quot;source&quot;:&quot;bangalore&quot;,<br>       &quot;destination&quot;: &quot;bangalore&quot;<br>      }<br>      &quot;type&quot;: &quot;city&quot;<br>    }<br>  ],<br>  &quot;config&quot;: {<br>    &quot;params&quot;: [<br>      {<br>        &quot;name&quot;: &quot;banner&quot;,<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;value&quot;: &quot;Welcome User&quot;<br>      }<br>    ]<br>  },<br>  &quot;start_date&quot;: &quot;2023-05-01&quot;,<br>  &quot;end_date&quot;: &quot;2023-05-15&quot;<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5f92bdc203b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rapido-labs/rapidex-experiments-in-rapido-at-a-scale-e5f92bdc203b">RapideX-Experiments in Rapido at a scale</a> was originally published in <a href="https://medium.com/rapido-labs">Rapido Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thinking in Big Data at Rapido: Cohort KPI]]></title>
            <link>https://medium.com/rapido-labs/thinking-in-big-data-cohort-kpi-851a902f4f?source=rss-ea2f87aaedec------2</link>
            <guid isPermaLink="false">https://medium.com/p/851a902f4f</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[mobility]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Vikash Singh]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 09:57:28 GMT</pubDate>
            <atom:updated>2020-08-18T06:50:23.263Z</atom:updated>
            <content:encoded><![CDATA[<h3>Thinking In Big Data at Rapido — Cohort KPI</h3><p>This post is a continuation of my<strong> </strong><a href="https://medium.com/@vicky301186/thinking-in-bigdata-cohort-6809beea1bd2"><strong>earlier post</strong></a>, in which we have built a basic understanding of the cohort and how it can be used for analyzing customer engagement with the product. In this post, we are going to see how we extract the KPIs for our customer cohort.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s9Ig90SZ80bNd-d2FKHUFw.png" /></figure><blockquote>In the transportation and mobility sector, data and independent research are at the core of solving potential problems. Rapido aims to find core solutions to these problems.</blockquote><p>Customer or rider cohort KPI is an important dataset to analyze customer/rider engagement with the product. It gives us visibility on the usage pattern as well as on the parameters like acquisition, retention, loyal users, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xsRHChhgL0xLXvQ0wCfDxw.png" /></figure><p>The above dataset represents customer cohort KPIs and it has variations of time level(weekly, monthly), geo_level(city, country).</p><p>Let me deep-dive into how we implemented the Cohorts KPefficiently across a year of raw data using Apache Spark.</p><p>We used YAML to define KPIs of customer cohort:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q5NKE4s3REm_lsiC99h3vw.png" /><figcaption>A sample YAML code</figcaption></figure><p>We implemented a YAML parser to translate the KPIs defined in the above file to the following:</p><pre>selectColumns: _id,customer,createdon,status,service,subtotal,discount,rider,serviceobj_city,createdts,time_level,time_value,geo_level,geo_value</pre><pre>groupColumns:<br>customer,time_level,geo_level,geo_value,time_value</pre><pre>columns:<br>count(CASE WHEN ((service IN (Link, Lite)) AND (status = dropped)) THEN _id END) AS `net_rides`<br>bround(sum(CASE WHEN ((service IN (Link, Lite)) AND (status = dropped)) THEN amount ELSE 0 END), 2) AS `net_gmv`<br>bround(sum(CASE WHEN ((service IN (Link, Lite)) AND (status = dropped)) THEN discount ELSE 0 END), 2) AS `discount`</pre><p>Now, let’s take a step-by-step look at how we transformed raw data into the Cohort KPIs</p><pre><strong>val </strong>dataFrame = <em>Seq</em>(<br>  (<strong>&quot;_id1&quot;</strong>, <strong>&quot;cus1&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-03-01&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06),<br>  (<strong>&quot;_id2&quot;</strong>, <strong>&quot;cus1&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-02-04&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06),<br>  (<strong>&quot;_id3&quot;</strong>, <strong>&quot;cus1&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-01-05&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06),<br>  (<strong>&quot;_id4&quot;</strong>, <strong>&quot;cus1&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-01-10&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06),<br>  (<strong>&quot;_id6&quot;</strong>, <strong>&quot;cus3&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-03-10&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06),<br>  (<strong>&quot;_id7&quot;</strong>, <strong>&quot;cus1&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-02-05&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06),<br>  (<strong>&quot;_id8&quot;</strong>, <strong>&quot;cus1&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-02-08&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06),<br>  (<strong>&quot;_id11&quot;</strong>, <strong>&quot;cus7&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-01-15&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06),<br>  (<strong>&quot;_id12&quot;</strong>, <strong>&quot;cus7&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-03-12&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06),<br>  (<strong>&quot;_id13&quot;</strong>, <strong>&quot;cus8&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;2020-01-12&quot;</strong>, <strong>&quot;dropped&quot;</strong>, <strong>&quot;Bangalore&quot;</strong>, 4.06)<br>).toDF(<strong>&quot;_id&quot;</strong>, <strong>&quot;customer&quot;</strong>, <strong>&quot;rider&quot;</strong>, <strong>&quot;orderDate&quot;</strong>, <strong>&quot;status&quot;</strong>, <strong>&quot;serviceobj_city&quot;</strong>, <strong>&quot;discount&quot;</strong>)<br>  .withColumn(<strong>&quot;createdOn&quot;</strong>, epochUdf(<strong>$&quot;orderDate&quot;</strong>))<br>  .withColumn(<strong>&quot;serviceObj_service&quot;</strong>, <em>lit</em>(<strong>&quot;Link&quot;</strong>))<br>  .withColumn(<strong>&quot;subtotal&quot;</strong>, <em>lit</em>(1.5))<br>  .withColumn(<strong>&quot;createdts&quot;</strong>, <em>from_unixtime</em>(<em>col</em>(<strong>&quot;createdon&quot;</strong>) / 1000))<br>  .withColumn(<strong>&quot;created_ist&quot;</strong>, <em>from_utc_timestamp</em>(<em>col</em>(<strong>&quot;createdts&quot;</strong>), <strong>&quot;IST&quot;</strong>))<br>  .withColumn(<strong>&quot;RideDate&quot;</strong>, <em>date_format</em>(<em>col</em>(<strong>&quot;created_ist&quot;</strong>), <strong>&quot;yyyy-MM-dd&quot;</strong>))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83L2oRPL_46unqFINeyxaQ.png" /><figcaption>Sample Order Data</figcaption></figure><p>In the first transformation, we exploded the data and added columns for time_level, time_value, geo_level, and geo_value.</p><p>Here we have created a user-defined function to extract time_level, time_value, geo_level, and geo_value.</p><pre><strong>case class </strong>GroupCohort(time_value: String, time_level: String, geo_value: String, geo_level: String)</pre><pre><strong>val </strong><em>zone </em>= DateTimeZone.<em>forID</em>(<strong>&quot;Asia/Kolkata&quot;</strong>)</pre><pre><strong>def </strong>extractGroupCohort(createdOn: Long, city: String): Seq[GroupCohort] = {<br>  <strong>val </strong>time = DateTime.<em>now</em>().withZone(<em>zone</em>).withMillis(createdOn)<br><br>  <strong>var </strong>groupCohorts: Seq[GroupCohort] = <em>Seq</em>.empty<br><br>  <strong>val </strong>timeFmt = DateTimeFormat.<em>forPattern</em>(<strong>&quot;yyyyMMdd&quot;</strong>)<br>  <strong>val </strong>cityInLowerCase = city.replaceAll(<strong>&quot; &quot;</strong>, <strong>&quot;_&quot;</strong>).toLowerCase()<br>  groupCohorts = groupCohorts :+ <em>GroupCohort</em>(timeFmt.print(time.withDayOfMonth(1).withHourOfDay(0).withMinuteOfHour(0)), <strong>&quot;monthly&quot;</strong>, cityInLowerCase, <strong>&quot;city&quot;</strong>)<br>  groupCohorts = groupCohorts :+ <em>GroupCohort</em>(timeFmt.print(time.withDayOfMonth(1).withHourOfDay(0).withMinuteOfHour(0)), <strong>&quot;monthly&quot;</strong>, <strong>&quot;india&quot;</strong>, <strong>&quot;panindia&quot;</strong>)<br><br>  groupCohorts = groupCohorts :+ <em>GroupCohort</em>(timeFmt.print(time.withDayOfWeek(1).withHourOfDay(0).withMinuteOfHour(0)), <strong>&quot;weekly&quot;</strong>, cityInLowerCase, <strong>&quot;city&quot;</strong>)<br>  groupCohorts = groupCohorts :+ <em>GroupCohort</em>(timeFmt.print(time.withDayOfWeek(1).withHourOfDay(0).withMinuteOfHour(0)), <strong>&quot;weekly&quot;</strong>, <strong>&quot;india&quot;</strong>, <strong>&quot;panindia&quot;</strong>)<br><br>  groupCohorts<br>}</pre><p>Post that our data frame transformation looked as follows:</p><pre><strong>val </strong>extractGroupCohortUdf = <em>udf</em>(<em>extractGroupCohort </em>_)</pre><pre><strong>val </strong>orderWithTimeGeo = ordersBaseDf<br>  .withColumn(<strong>&quot;group_cohort&quot;</strong>, <em>explode</em>(extractGroupCohortUdf(<strong>$&quot;createdOn&quot;</strong>, <strong>$&quot;serviceobj_city&quot;</strong>)))<br>  .withColumn(<strong>&quot;time_level&quot;</strong>, <strong>$&quot;group_cohort.time_level&quot;</strong>)<br>  .withColumn(<strong>&quot;time_value&quot;</strong>, <strong>$&quot;group_cohort.time_value&quot;</strong>)<br>  .withColumn(<strong>&quot;geo_level&quot;</strong>,  <strong>$&quot;group_cohort.geo_level&quot;</strong>)<br>  .withColumn(<strong>&quot;geo_value&quot;</strong>,  <strong>$&quot;group_cohort.geo_value&quot;</strong>)<br>  .drop(<strong>&quot;group_cohort&quot;</strong>)</pre><p>orderWithTimeGeo.show(false)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83L2oRPL_46unqFINeyxaQ.png" /><figcaption>Add time_level, time_value, geo_level and geo_value</figcaption></figure><p>In the next transformation, we computed customer KPIs aggregation based on time_level, geo_value, and geo_level and associated it with time_value.</p><pre><strong>val </strong>customerKpiMonthlyAggregation = orderWithTimeGeo<br>  .select(selectColumns: _*)<br>  .groupBy(groupColumns: _*)<br>  .agg(<em>array</em>(<strong>$&quot;time_value&quot;</strong>, <em>concat_ws</em>(<strong>&quot;@@&quot;</strong>, columns: _*).as(<strong>&quot;payload&quot;</strong>)).as(<strong>&quot;values&quot;</strong>))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83L2oRPL_46unqFINeyxaQ.png" /><figcaption>Customer KPIs aggregation</figcaption></figure><p>Then, we have created the cohort dataset with source_time_value as previous- appeared- week and compare_time_value as next- appeared- week.</p><p>The following user-defined function helped us create pairs of time as previous and next time.</p><pre><strong>case class </strong>CohortKpiRecord(time_value: String, payload: String)<strong>case class </strong>TimeValueWithKpi(source: String, compare: String, payload: String)</pre><pre><strong>def </strong>createKpiCohort(values: Seq[Seq[String]], level: String): List[TimeValueWithKpi] = {<br>  <strong>val </strong>cohortKpiRecords = values.map({<br>    <strong>case </strong>Seq(time_value, payload) =&gt; <em>CohortKpiRecord</em>(time_value, payload)<br>  })<br>    .sortBy(_.time_value)<br><br>  cohortKpiRecords.toList.combinations(2)<br>    .map({<br>      <strong>case </strong>List(current, next) =&gt;<br>        <em>TimeValueWithKpi</em>(current.time_value, next.time_value, next.payload)<br>    })<br>    .toList ++ cohortKpiRecords.map({<br>    <strong>case </strong><em>CohortKpiRecord</em>(time_value, payload) =&gt;<br>      <em>TimeValueWithKpi</em>(time_value, <strong>&quot;&quot;</strong>, payload)<br>  })<br>}</pre><p>The data transformation looked as follows:</p><pre><strong>val </strong>dataTypeMap = <em>Map</em>(<br>  <strong>&quot;discount&quot; </strong>-&gt; LongType,<br>  <strong>&quot;net_rides&quot; </strong>-&gt; LongType,<br>  <strong>&quot;net_gmv&quot; </strong>-&gt; LongType<br>)</pre><pre><strong>val </strong>payloadColumnsWihType = columns.zipWithIndex<br>  .map({<br>    <strong>case </strong>(field, index) =&gt;<br>      <strong>val </strong>alias = field.expr.asInstanceOf[Alias]<br>      <em>col</em>(<strong>&quot;data&quot;</strong>)(index).as(alias.name)<br>        .cast(dataTypeMap(alias.name))<br>  }).toList</pre><pre><strong>val </strong>kpiUdf = <em>udf</em>(<em>createKpiCohort </em>_)</pre><pre><strong>val </strong>cohortBaseDf = customerKpiMonthlyAggregation<br>.groupBy(<em>groupColumns</em><br>  .filterNot(_.equals(<strong>&quot;time_value&quot;</strong>)).map(<em>col</em>): _*)<br>  .agg(<em>collect_set</em>(<strong>$&quot;values&quot;</strong>).as(<strong>&quot;values&quot;</strong>))<br>  .withColumn(<strong>&quot;time_frame&quot;</strong>,<em>explode</em>(kpiUdf(<strong>$&quot;values&quot;</strong>,<strong>$&quot;time_level&quot;</strong>)))<br>  .withColumn(<strong>&quot;source_time_value&quot;</strong>, <strong>$&quot;time_frame.source&quot;</strong>)<br>  .withColumn(<strong>&quot;compare_time_value&quot;</strong>, <strong>$&quot;time_frame.compare&quot;</strong>)<br>  .withColumn(<strong>&quot;payload&quot;</strong>, <strong>$&quot;time_frame.payload&quot;</strong>)<br>  .withColumn(<strong>&quot;data&quot;</strong>, <em>split</em>(<strong>$&quot;payload&quot;</strong>, <strong>&quot;@@&quot;</strong>))<br>  .drop(<strong>&quot;values&quot;</strong>, <strong>&quot;time_frame&quot;</strong>, <strong>&quot;payload&quot;</strong>)<br>  .select(<strong>$&quot;customer&quot; </strong>:: <strong>$&quot;time_level&quot; </strong>:: <strong>$&quot;geo_level&quot; </strong>:: <strong>$&quot;geo_value&quot; </strong>:: <strong>$&quot;source_time_value&quot; </strong>:: <strong>$&quot;compare_time_value&quot; </strong>:: payloadColumnsWihType: _*)<br>  .withColumn(<strong>&quot;compare_time_value&quot;</strong>, <em>when</em>(<strong>$&quot;compare_time_value&quot; </strong>=== <strong>&quot;&quot;</strong>, <strong>$&quot;source_time_value&quot; </strong>).otherwise(<strong>$&quot;compare_time_value&quot;</strong>))</pre><p>cohortBaseDf.show(false)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83L2oRPL_46unqFINeyxaQ.png" /><figcaption>Customer dataset with source and compare time value</figcaption></figure><p>In the final step transformation, where we computed customer cohort KPIs based on source_time_value, compare_time_value, time_level, geo_value, and geo_level.</p><pre><strong>val </strong>result = cohortBaseDf.groupBy(<strong>&quot;source_time_value&quot;</strong>, <strong>&quot;compare_time_value&quot;</strong>, <strong>&quot;time_level&quot;</strong>, <strong>&quot;geo_value&quot;</strong>, <strong>&quot;geo_level&quot;</strong>)<br>  .agg(<em>count</em>(<strong>&quot;customer&quot;</strong>).as(<strong>&quot;compare_kpi_value&quot;</strong>),<br>    <em>sum</em>(<strong>&quot;net_gmv&quot;</strong>).as(<strong>&quot;compare_net_gmv&quot;</strong>),<br>    <em>sum</em>(<strong>&quot;net_rides&quot;</strong>).as(<strong>&quot;compare_net_rides&quot;</strong>),<br>    <em>sum</em>(<strong>&quot;discount&quot;</strong>).as(<strong>&quot;compare_discount&quot;</strong>))<br>  .withColumn(<strong>&quot;source_kpi_value&quot;</strong>, <em>max</em>(<strong>&quot;compare_kpi_value&quot;</strong>) over Window.<em>partitionBy</em>(<strong>&quot;source_time_value&quot;</strong>, <strong>&quot;geo_value&quot;</strong>))<br>  .withColumn(<strong>&quot;source_net_gmv&quot;</strong>, <em>max</em>(<strong>&quot;compare_net_gmv&quot;</strong>) over Window.<em>partitionBy</em>(<strong>&quot;source_time_value&quot;</strong>, <strong>&quot;geo_value&quot;</strong>))<br>  .withColumn(<strong>&quot;source_net_rides&quot;</strong>, <em>max</em>(<strong>&quot;compare_net_rides&quot;</strong>) over Window.<em>partitionBy</em>(<strong>&quot;source_time_value&quot;</strong>, <strong>&quot;geo_value&quot;</strong>))<br>  .withColumn(<strong>&quot;source_discount&quot;</strong>, <em>max</em>(<strong>&quot;compare_discount&quot;</strong>) over Window.<em>partitionBy</em>(<strong>&quot;source_time_value&quot;</strong>, <strong>&quot;geo_value&quot;</strong>))<br>  .select(<strong>&quot;source_time_value&quot;</strong>, <strong>&quot;compare_time_value&quot;</strong>, <strong>&quot;time_level&quot;</strong>, <strong>&quot;geo_level&quot;</strong>, <strong>&quot;geo_value&quot;</strong>, <strong>&quot;source_kpi_value&quot;</strong>, <strong>&quot;compare_kpi_value&quot;</strong>, <br> <strong>&quot;source_net_gmv&quot;</strong>, <strong>&quot;compare_net_gmv&quot;</strong>, <strong>&quot;source_net_rides&quot;</strong>, <strong>&quot;compare_net_rides&quot;</strong>, <strong>&quot;source_discount&quot;</strong>, <strong>&quot;compare_discount&quot;</strong>)</pre><p>result.show(false)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83L2oRPL_46unqFINeyxaQ.png" /><figcaption>Final Customer Cohort KPI</figcaption></figure><h3>Conclusion:</h3><p>The above approach is not a methodically approach like joining of previous- appeared- week customers with and next- appeared- week customers and it uses the pivot to extract KPIs.</p><p>I would like to thanks my teammates <a href="https://medium.com/u/11bb42d93f98">Pramod N</a>, <a href="https://medium.com/u/72cef4b2490a">Pubali Bhaduri</a>, <a href="https://medium.com/u/bd01bbcfbad1">Jaganreddyguda</a>, and Rapido data platform team for their valuable contributions and insight.</p><p>You can also reach out to me via <a href="https://www.linkedin.com/in/vikash-kumar-singh-02463624/"><strong>LinkedIn</strong></a> or on <a href="http://vikash.singh@rapido.bike/">vikash.singh@rapido.bike</a> if in case you are exploring roles in our Data team.</p><p>Happy coding and learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=851a902f4f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rapido-labs/thinking-in-big-data-cohort-kpi-851a902f4f">Thinking in Big Data at Rapido: Cohort KPI</a> was originally published in <a href="https://medium.com/rapido-labs">Rapido Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[THINKING IN BIGDATA-COHORT]]></title>
            <link>https://medium.com/@vicky301186/thinking-in-bigdata-cohort-6809beea1bd2?source=rss-ea2f87aaedec------2</link>
            <guid isPermaLink="false">https://medium.com/p/6809beea1bd2</guid>
            <category><![CDATA[cohort-3]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Vikash Singh]]></dc:creator>
            <pubDate>Sun, 31 May 2020 11:49:16 GMT</pubDate>
            <atom:updated>2020-05-31T13:04:26.626Z</atom:updated>
            <content:encoded><![CDATA[<h4>What is a cohort?</h4><p>In a nutshell, a cohort is simply a subset of users grouped by shared characteristics. In the context of business analytics, a cohort usually refers to a subset of users specifically segmented by acquisition date (i.e. the first time a user visits your website).</p><p>A “cohort analysis,” then, simply allows you to compare the behaviour and metrics of different cohorts over time. In reality, the lack of activity of the old users is being hidden by the impressive growth numbers of new users, which results in concealing the lack of engagement from a small number of people.</p><h4>Cohort Analysis Example</h4><p>Let’s understand using cohort analysis with an example — the daily cohort of users who have launched an app first time and revisited the app in the next 10 days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Qk3QE9oLzQW7ftE.png" /></figure><p>From the above retention table — Triangular chart, we can infer the following</p><ul><li>1358 users launched an app on Jan 26. Day 1 retention was 31.1%, day 7 retention was 12.9%, and day 9 retention was 11.3%. So on the 7th day after using the app, 1 in 8 users who launched an app on Jan 26 were still active users on the app.</li><li>Out of all of the new users during this time range (13,487 users), 27% of users are retained on day 1, 12.5% on day 7, and 12.1% on day 10.</li></ul><p>We can read more about the usage and benefits of cohorts across the web.</p><h4>Retention cohort implementation:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*DD3h5n8Nu-I4LzoNkT3_Yw.png" /></figure><p>Let’s assume we have the above dataset of the delivery app customer and schema is:</p><p>ts: timestamp of a customer event</p><p>number: number/identification of a customer</p><p>pick_lat: pickup latitude of order</p><p>pick_lng: pickup longitude of order</p><p>drop_lat: drop latitude of order</p><p>drop_lng: drop longitude of order</p><h4>Example 1:</h4><pre><strong>val </strong>visitedWeeks = data<br>  .withColumn(<strong>&quot;week&quot;</strong>, <em>date_trunc</em>(<strong>&quot;week&quot;</strong>, <em>col</em>(<strong>&quot;ts&quot;</strong>)))<br>  .selectExpr(<strong>&quot;number&quot;</strong>, <strong>&quot;week&quot;</strong>)<br>  .distinct()<br><br><strong>val </strong>leadWeeks = visitedWeeks.as(<strong>&quot;visited&quot;</strong>)<br>  .join(visitedWeeks.as(<strong>&quot;leader&quot;</strong>),<br>    <em>col</em>(<strong>&quot;visited.number&quot;</strong>) === <em>col</em>(<strong>&quot;leader.number&quot;</strong>) and <em>col</em>(<strong>&quot;visited.week&quot;</strong>) &lt;= <em>col</em>(<strong>&quot;leader.week&quot;</strong>),<br>    <strong>&quot;left&quot;</strong>)<br>  .selectExpr(<strong>&quot;leader.week as lead_week&quot;</strong>, <strong>&quot;visited.week as visit_week&quot;</strong>)<br><br>leadWeeks.groupBy(<strong>&quot;visit_week&quot;</strong>, <strong>&quot;lead_week&quot;</strong>)<br>  .count()<br>  .groupBy(<strong>&quot;visit_week&quot;</strong>)<br>  .pivot(<strong>&quot;lead_week&quot;</strong>)<br>  .agg(<em>first</em>(<em>col</em>(<strong>&quot;count&quot;</strong>)))</pre><p>With a record of 1111568 and size of data is 543M, the metric of execution time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BH8Gotelk1LWW8sphzdoVg.png" /><figcaption>Execution Time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zmbjWwCXvtaw0_WMmXcJvw.png" /><figcaption>Pivot DAG</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EkWeNu2kPW2CuTVGTWIgSA.png" /><figcaption>Count DAG</figcaption></figure><p>There could be other ways to solve it.</p><p>Now I gonna discuss how I solved it using with common sense, the idea is very simple to collect all weeks where a given customer ordered delivery and create all possible combination of weeks in an ascending way.</p><p>For example, let’s assume customer C1 ordered in weeks w1, w3, w5 so transform data will look like:</p><p>C1 , (w1, w3) | C1 , (w1, w5) | C1 , (w3, w5)</p><p>Now we just need to simply count the number of customers based on combination of weeks.</p><pre><strong>case class </strong>TimeValue(source: String, compare: String)<br><br><strong>def </strong>createCohortPair(dates: Seq[String]): List[TimeValue] = {<br>  <strong>val </strong>sortedDates = dates.sortBy((date: String) =&gt; date)<br>  <strong>val </strong>temps = sortedDates.toList.combinations(2).map({ <strong>case </strong>List(source, compare) =&gt; <em>TimeValue</em>(source, compare) }).toList ++ sortedDates.map(<em>TimeValue</em>(_, <strong>&quot;&quot;</strong>))<br>  temps<br>}<br><br><strong>val </strong>createCohortUdf = <em>udf</em>(createCohortPair _)<br><br><strong>val </strong>cohortBaseDf = dataFrame<br>  .withColumn(<strong>&quot;week&quot;</strong>, <em>date_trunc</em>(<strong>&quot;week&quot;</strong>, <em>col</em>(<strong>&quot;ts&quot;</strong>)))<br>  .groupBy(<strong>&quot;number&quot;</strong>)<br>  .agg(<em>collect_set</em>(<strong>&quot;week&quot;</strong>).as(<strong>&quot;values&quot;</strong>))<br>  .withColumn(<strong>&quot;time_frame&quot;</strong>, <em>explode</em>(createCohortUdf(<strong>$&quot;values&quot;</strong>)))<br>  .withColumn(<strong>&quot;source_time_value&quot;</strong>, <strong>$&quot;time_frame.source&quot;</strong>)<br>  .withColumn(<strong>&quot;compare_time_value&quot;</strong>, <strong>$&quot;time_frame.compare&quot;</strong>)<br>  .withColumn(<strong>&quot;compare_time_value&quot;</strong>, <em>when</em>(<strong>$&quot;compare_time_value&quot; </strong>=== <strong>&quot;&quot;</strong>, <strong>$&quot;source_time_value&quot;</strong>).otherwise(<strong>$&quot;compare_time_value&quot;</strong>))<br>  .drop(<strong>&quot;time_frame&quot;</strong>, <strong>&quot;values&quot;</strong>)<br><br>cohortBaseDf.groupBy(<strong>&quot;source_time_value&quot;</strong>, <strong>&quot;compare_time_value&quot;</strong>)<br>  .agg(<em>count</em>(<strong>&quot;number&quot;</strong>).as(<strong>&quot;compare_kpi_value&quot;</strong>))<br>  .withColumn(<strong>&quot;source_kpi_value&quot;</strong>, <em>max</em>(<strong>&quot;compare_kpi_value&quot;</strong>) over Window.<em>partitionBy</em>(<strong>&quot;source_time_value&quot;</strong>))<br>  .select(<strong>&quot;source_time_value&quot;</strong>, <strong>&quot;compare_time_value&quot;</strong>, <strong>&quot;source_kpi_value&quot;</strong>, <strong>&quot;compare_kpi_value&quot;</strong>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yxVFFAbRtodyYStdI56HKg.png" /><figcaption>Execution Time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*79lwYwMjoTVMF8hLMarGxg.png" /><figcaption>Count DAG</figcaption></figure><p>We can see the improvement of execution time in the later one, this above solution only restricted to count of retaining customers. Hope you folks like the read and try to find other KPI on customer cohort. I’ll tell about for these KPI on part-2 of it.</p><p>Happy coding and learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6809beea1bd2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[THINKING IN BIG DATA]]></title>
            <link>https://medium.com/@vicky301186/thinking-in-big-data-838d09e9d0c3?source=rss-ea2f87aaedec------2</link>
            <guid isPermaLink="false">https://medium.com/p/838d09e9d0c3</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Vikash Singh]]></dc:creator>
            <pubDate>Mon, 23 Sep 2019 06:48:59 GMT</pubDate>
            <atom:updated>2019-09-23T06:48:59.042Z</atom:updated>
            <content:encoded><![CDATA[<p>From last two and half years, working on Big Data using a few tools from the Hadoop ecosystem and spark. For me, Big Data comprises of three v’s (volume, velocity, and variety). When I about to start the journey of Data Engineering struggling with many buzzwords like CAP theorem, Data skewness, shuffling, etc. But what I discovered is some common sense with the understanding of the above-mentioned tools. Here I am taking you on my journey with common sense as my oar.</p><h3>Scenario 1:</h3><p>We are working on a BI project for one of credit card provider company. The client wants to generate a report of the performance metric for a given merchant says Wallmart for a given period. Here performance metrics include total spend of transactions and a total count of customers.</p><p>It is a simple metrics to generate, we can use sum() and distinct() function of Spark over the columns amount and customer_id and combine the result. A great job is done but waits it needs to process millions of transactions and response need to be in seconds.</p><p>While solving the problem, thinking of why not apply the sum and distinct count evaluation into a single scan of the dataset. Then come up with a solution of we can sum the spend of customers transaction and take one for count, so the transformed tuple is (customer_id,sum_spend,1 as count, some constant says <strong>ABC</strong>) and in next iteration, we can sum the individual customer spend and sum the count and group by it on <strong>ABC</strong>. This solution works and performative over the initial implementation.</p><p>We can argue that a customer data can be distributed across partitions, so we need to do partition by on customer_id that means shuffling but we can use the features of hive bucketing while we create transaction table to avoid the initial partitioning.</p><h3>Scenario 2:</h3><p>Part of same BI project, a new requirement of analyzing intersection and subtraction(A∩B, A-B, B-A) of transactions of different merchants for a given period.</p><p>Not an easy metric, few more lines of codes compare than Scenario 1. Create two different datasets based on merchants criteria, do a left outer join of datasets and evaluate A∩B, A-B, B-A.</p><p>Ohh no, it’s a joint operation between millions of transactions and we are facing a performance issue. While solving the problem, thinking of both the datasets has the same underlying dataset and come up with a solution for tagging the data with A or B or both based on different merchant criteria. Now it’s become a simple single scan to check for A∩B, A-B, B-A. Just apply a common sense and it solves the performance bottleneck.</p><h3>Scenario 3:</h3><p>This metric is a little complicated, here for a given merchant wants to know about their customer’s interaction with competitors and segments of industries like food, shopping automobile, etc for a given period. This metric includes the percentage of customer counts, percentage of customer spends in different industry segments of competitors.</p><p>This is really a complicated metric, initial implementations have the following steps:</p><ul><li>create a dataset for given merchant say A.</li><li>create another dataset for competitors merchant say B.</li><li>create customer dataset from dataset A say Customer.</li><li>join Customer dataset with B and group them on the industry segment and evaluate the required metrics.</li></ul><p>This looks do-able but we are not able to execute this metric for a period of one, two, three years even on in-house yarn cluster. I took this performance issue until this time I am more confident due to the fixing of the earlier performance bottleneck. For the issue deep-diving into GC’s log, strategies and stumble on the mark-sweep term. It gives me a solution and it has the following steps:</p><ul><li>underlying data same for given merchant and competitor merchants, so use the tagging approach for given merchant and competitor’s customers.</li><li>now filter out all those customers with a given merchant(A-B) (mark-sweep).</li><li>now we have all the transactions details of given merchant’s customer with competitors.</li><li>group them on industry segment and evaluate the required metrics.</li></ul><p>This solution works well and becomes more performative.</p><h3>Scenario 4:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/0*bnWVVntBNvRHcoCc" /></figure><p>The <a href="https://en.wikipedia.org/wiki/Star_schema">star schema</a> consists of one or more fact tables referencing any number of dimension tables. For Client requirement, we are building a star schema and the initial implementation is just joined all the tables. This approach has serious problems because it deals with many joins(in some cases outer-join) and the number of dimensions can be range from 5 to 10 dimensions. We can also say it aggregated structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*rB1bfuqlIvilZatr" /></figure><p>Here my assumption is that we can use a NoSQL key-value database and add insert column families for different dimension against same row key. Not able to implement due to client restriction.</p><p>Come up with another approach of a star schema, transform each dimension into the same schema and dump each dimension into the same folder as parquet. With spark read all the parquet files and partition by on join key.</p><h3>Scenario 5:</h3><p>Working on items similarity for a product catalogs and ends with the cross-join of items to generate every possible tuple for comparison.</p><p>Ohh no! cross joins for 1,00,000 items will create a very large number of records and also not very scalable in terms of prediction of resources required. While solving the problem thinking of why generate all possible tuples once then come up with a solution and it’s something like:</p><ul><li>cross-join of 1,00,000 items with first n values say 10000</li><li>then compute the cross-join of 1,00,000 items with next n values</li><li>repeat until process all possible tuples.</li></ul><p>So here common sense helps me to solve the problems and of course, knowledge of tools helps me to understand what’s happening on a system level.</p><p>Happy coding and learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=838d09e9d0c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BLOCKCHAIN FUNDAMENTAL AND IT’S BASICS:]]></title>
            <link>https://medium.com/@vicky301186/blockchain-fundamental-and-its-basics-ce6d87a58383?source=rss-ea2f87aaedec------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce6d87a58383</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[distributed-ledgers]]></category>
            <dc:creator><![CDATA[Vikash Singh]]></dc:creator>
            <pubDate>Mon, 23 Sep 2019 06:44:13 GMT</pubDate>
            <atom:updated>2019-09-23T06:44:13.227Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s connected and integrated world, the economic activity takes place in business networks. Business networks typically come together at marketplaces where producers, consumers, suppliers, partners, market makers/enablers, and other stakeholders own, control, and exercise their rights, privileges, and entitlements on objects of value known as assets.</p><p>Assets can be tangible and physical, such as cars and homes, or intangible and virtual, such as stock certificates and patents. Asset ownership and transfer create value in a business network and are known as transactions.</p><p>A business typically uses multiple ledgers to keep track of asset ownership and asset transfers between participants in its various lines of businesses. Ledgers are the systems of record (SORs) for a business’s economic activities and interests.</p><p>A typical business ledger looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*x7BIw2hiRRhwdFog" /></figure><p>We collectively referred a collection of these entries or SOR as a BLOCK at a certain interval of time. These collections of BLOCK’s recorded as a different instance of the timestamp can be linked together to form a chain of BLOCK (Blockchain).</p><p><strong>Ways of maintaining those business ledgers(SOR)?</strong></p><p>Business entities use centralized, trust-based, third-party systems, such as financial institutions, clearinghouses, and other mediators of existing institutional arrangements.</p><p>Centralized, trust-based ledger systems lead to bottlenecks and slowdowns of transaction settlements. Lack of transparency, as well as susceptibility to corruption and fraud, lead to disputes. Having to resolve disputes and possibly reverse transactions or provide insurance for transactions is costly. These risks and uncertainties contribute to missed business opportunities.</p><p>Furthermore, out-of-sync copies of business ledgers on each network participant’s own systems lead to faulty business decisions made on temporary, incorrect data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*M8umSCZxEutc-w65" /></figure><p>A centralized ledger network controls the flow of information and operational control from a single central point.</p><p>A distributed ledger network spreads the computational workload across multiple nodes in a network.</p><p>A decentralized ledger network allows nodes to make independent processing and computational decisions irrespective of what other peer nodes may decide.</p><p><strong>Blockchain’s attributes:</strong></p><ol><li>It’s a peer to peer network structure where nodes can be ‘Blockchain’, ‘Routing Node’, ‘Miner’ and client(‘Wallet’).</li><li>It maintains a distributed shared ledger where all node is communicated/agreed on entries of a system of record based on consensus.</li><li>It’s use proof of work, or another consensus mechanism, as a basis of trust, accountability, and transparency, instead of relying on a third-party mediator financial institution or actor.</li><li>The proof of work required high intensive computational resource so until halves of computation power of the network is honest, it’s secure.</li><li>Solving proof of work is very tough but verifying it is very easy. It’s like thousands of rows and columns Sudoku problem, tough to solve but verification is easy.</li><li>Entries in shared ledger are irreversible.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*264B_YHGjhZaN3S6" /></figure><p><strong>How Does Blockchain work?</strong></p><ul><li>New records enter into the network in an unconfirmed manner and are broadcast to all nodes.</li><li>Each processing node independently assembles the records into a block.</li><li>Each processing node of miner type finds proof of work for its assembled block independently and in competition with other miner nodes for incentives.</li><li>Each miner’s proof of work involves computing a problem for an assembled block that meets the published difficulty level target for the current computational cycle.</li><li>When a processing node of miner type finds proof of work, it broadcasts the block to all nodes.</li><li>Processing nodes of miner type accept the block only if all records in it are valid.</li><li>Processing nodes of miner type express their acceptance of the block by creating the next block in the chain, using the hash of the accepted block as the previous hash. The blockchain continues to lengthen over time.</li><li>By protocol design, processing nodes of miner type agree on the longest chain as the correct chain to work on to extend the chain. All exception processing events — such as nodes joining the network or departing from the network, or transaction or block broadcasts reaching destinations or getting dropped, or nodes simultaneously broadcasting different versions of a block — get resolved by the P2P network protocol in a robust, tolerant, and decentralized consensus manner.</li></ul><p><strong>Bitcoin</strong></p><p>Bitcoin is a cryptographic currency(virtual) used to trade between B2B and B2C, first published in Nakatamo Satoshi paper. It’s trust/security backend by Blockchain architecture.</p><p>Here we demonstrate a simple transaction “Alice pays Bob 15 bitcoin for a cup of coffee”:</p><p><strong>Transaction:</strong></p><p>In Bitcoin transactions means a transfer of ownership. New bitcoins(mint) are generated only by miners. Each transaction is broadcast to its near node and so on, this way transaction is broadcast to each node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*0ok4iA1eBzaFPHDS" /></figure><p>A sample Bitcoin transaction:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*rLCRrSh2Np3lQEsu" /></figure><p><strong>Authorization:</strong></p><p>The transaction block also includes section <strong>vout</strong> and <strong>vin</strong>.</p><p><strong>vout:</strong></p><p>Bitcoin user uses a pair of a private and public key to authorize the transfer of ownership.</p><p>The private key is generated by double hashing of a text of length of a random number less than 10⁸⁰.</p><p>The public key is generated from this private key using mathematical function like elliptical curve(one way) function. A derived hash(bitcoin address) for this public key used by other users to transfer ownership to a user associated with this public key.</p><p>vout consists of a digital signature of a user’s private key.</p><p><strong>vin:</strong></p><p>This contains a list of user’s bitcoin address to transfer ownership. Only those users who are associated with this public key able to spend that spent amount.</p><p><strong>Block:</strong></p><p>Every node in the blockchain network assemble a set of the transaction into a block and it’s linked by the previous block known to a node.</p><p>Structure of Block:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*eJXe-BSeXWjtVpVF" /></figure><p>Block Header is metadata of the Block. Structure of Block Header:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*-QeEiUFkWMBti6b2" /></figure><p><strong>Merkle Trees:</strong></p><p>Merkle trees are used in bitcoin to summarize all the transactions in a block, producing an overall digital fingerprint of the entire set of transactions, providing a very efficient process to verify if a transaction is included in a block. A Merkle tree is constructed by recursively hashing pairs of nodes until there is only one hash, called the root, or Merkle root. The cryptographic hash algorithm used in bitcoin’s Merkle trees is SHA256 applied twice, also known as double-SHA256.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*eLIsNmg-3Co77Cbr" /></figure><p><strong>Mining and Consensus:</strong></p><p>Here we use the Blockchain architecture to solve a proof of work for broadcast a Block after verification of transactions.</p><p>Proof of work statement:</p><p>Hash(SHA-256) of generated block header(merkel tree) suffixed with a counter which generate a hash value trail with nth zeroes.</p><p># example of iterating a nonce in a hashing algorithm’s input</p><p>import hashlib</p><p>text = “I am Satoshi Nakamoto”</p><p># iterate nonce from 0 to 19</p><p>for nonce in xrange(20):</p><p># add the nonce to the end of the text</p><p>input = text + str(nonce)</p><p># calculate the SHA-256 hash of the input (text+nonce)</p><p>hash = hashlib.sha256(input).hexdigest()</p><p># show the input and hash result</p><p>print input, ‘=&gt;’, hash</p><p>$ python hash_example.py</p><p>…</p><p>I am Satoshi Nakamoto12 =&gt; 60f01db30c1a0d4cbce2b4b22e88b9b…</p><p>I am Satoshi Nakamoto13 =&gt; 0ebc56d59a34f5082aaef3d66b37a66…</p><p>I am Satoshi Nakamoto14 =&gt; 27ead1ca85da66981fd9da01a8c6816…</p><p>…</p><p>After successful mining this block is added to Blockchain</p><p>and height of blockchain increases.</p><p><strong>IBM Blockchain:</strong></p><p>The Hyperledger Project defined a protocol specification known as Open Blockchain Protocol Specification to create a blockchain fabric for use in a variety of industry use-cases involving B2B and B2C transactions.</p><p>The primary goals of this effort are:</p><ul><li>Support a wide variety of industrial use cases with different requirements</li><li>Comply with regulatory regimes that exist today</li><li>Support verified identities, private and confidential transactions</li><li>Support permission shared ledger</li><li>Support performance, scaling, audit ability, identity, security, and privacy</li><li>Support reduction of costly computations involved in the proof of work</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*YIzyRKRB5g61YZAx" /></figure><p>Hyperledger’s Blockchain fabric implementations use the following concepts as its underpinnings:</p><ul><li>Smart contracts</li><li>Digital assets</li><li>System of record repositories/stores</li><li>Decentralized consensus-based network</li><li>Pluggable consensus algorithms/models</li><li>Cryptographic security</li></ul><p>IBM provides a pluggable method to implement our business logic as a chain code(Smart Contract).</p><p>Chaincode is a piece of code that lets you interact with a network’s shared ledger. Whenever you ‘invoke’ a transaction on the network, you are effectively calling a function in a piece of chaincode that reads and writes values to the ledger.</p><p>The three main functions are Init, Invoke, and Query in chaincode shim interface.</p><p><strong>Init:</strong></p><p>Init is called when we first deploy chaincode. This function used to do for chaincode configurations.</p><p>func (t *SimpleChaincode) Init(stub *shim.ChaincodeStub, function string, args []string) ([]byte, error) {</p><p>if len(args) != 1 {</p><p>return nil, errors.New(“Incorrect number of arguments. Expecting 1”)</p><p>}</p><p>err := stub.PutState(“hello_world”, []byte(args[0]))</p><p>if err != nil {</p><p>return nil, err</p><p>}</p><p>return nil, nil</p><p>}</p><p><strong>Invoke:</strong></p><p>Invoke is called when we to call chaincode functions to do real work. The structure of Invoke is simple. It receives a function argument and based on this argument calls Go functions in the chaincode.</p><p>func (t *SimpleChaincode) Invoke(stub *shim.ChaincodeStub, function string, args []string) ([]byte, error) {</p><p>fmt.Println(“invoke is running ” + function)</p><p>// Handle different functions</p><p>if function == “init” {</p><p>return t.Init(stub, “init”, args)</p><p>} else if function == “write” {</p><p>return t.write(stub, args)</p><p>}</p><p>fmt.Println(“invoke did not find func: ” + function)</p><p>return nil, errors.New(“Received unknown function invocation”)}</p><p><strong>Query:</strong></p><p>Query used to get chaincode state. It not add any block to chain.</p><p>func (t *SimpleChaincode) Query(stub *shim.ChaincodeStub, function string, args []string) ([]byte, error) {</p><p>fmt.Println(“query is running ” + function)</p><p>// Handle different functions</p><p>if function == “read” {</p><p>return t.read(stub, args)</p><p>}</p><p>fmt.Println(“query did not find func: ” + function)</p><p>return nil, errors.New(“Received unknown function query”)</p><p>}</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce6d87a58383" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Call Bridge game implemented in Python]]></title>
            <link>https://medium.com/@vicky301186/call-bridge-game-implemented-in-python-8d788aa4374e?source=rss-ea2f87aaedec------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d788aa4374e</guid>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Vikash Singh]]></dc:creator>
            <pubDate>Fri, 03 Jun 2016 18:17:35 GMT</pubDate>
            <atom:updated>2019-09-23T09:36:30.995Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/200/0*BNCcCo2b4UsHAQ4D.jpg" /></figure><h3>Introduction</h3><p>Call bridge is a game of tricks, trumps, and bidding.</p><h3>Players and Cards</h3><p>This game is normally played by 4 people using a standard international 52-card pack.</p><p>The cards of each suit rank from high to low A-K-Q-J-10–9–8–7–6–5–4–3–2.</p><p>Deal and play are counter-clockwise.</p><h3>Deal</h3><p>Any player may deal first: subsequently, the turn to deal passes to the right.</p><p>The dealer deals out all the cards, one at a time, face down, so that each player has 13 cards. The players pick up their cards and look at them.</p><h3>Bidding</h3><p>Starting with the player to dealer’s right, and continuing counter-clockwise around the table, ending with the dealer, each player calls a number, which must be at least 2. (The maximum sensible call is 12.) This call represents the number of tricks which the player undertakes to win. In this game, the tricks bid are known as “calls”.</p><h3>Play</h3><p>The player to dealer’s right leads to the first trick, and subsequently, the winner of each trick leads to the next.</p><p>Any card may be led, and the other three players must follow suit if they can. A player who cannot follow suit must trump with a spade, provided that this spade is high enough to beat any spades already in the trick. A player who has no cards of the suit led and no spades high enough to head the trick may play any card.</p><p>The trick is won by the player of the highest spade in it, or if it contains no spade, by the player of the highest card of the suit that was led.</p><p>Note that:</p><ol><li>A player who is able to play a card of the suit that was led is not obliged to head the trick. This also applies when spades are led: players may play higher or lower spades as they wish.</li><li>A player who has no cards of a suit is said to be “off” that suit. It is off the suit that was led, and there are no spades in the trick yet, the player must play a spade if possible. If there is already a spade in the trick, the player who is “off” the led suit must play a higher spade if possible. If the player only has lower spades, he or she can “waste” one of these spades to avoid taking an unwanted trick later or can throw a card of another suit.</li></ol><p><strong>Python Card Object:</strong></p><pre>class Card(object): def __init__(self, suite, face=&quot;jocker&quot;): self.suite = suite self.face = face self.point = Suite.points.get(face)</pre><p>Here the class Card represents a card with the following attributes:</p><p><strong>Python Suite Object:</strong></p><pre>class Suite(object): faces = [&quot;2&quot;, &quot;3&quot;, &quot;4&quot;, &quot;5&quot;, &quot;6&quot;, &quot;7&quot;, &quot;8&quot;, &quot;9&quot;, &quot;10&quot;, &quot;jack&quot;, &quot;queen&quot;, &quot;king&quot;, &quot;ace&quot;] points = {&quot;2&quot;: 1, &quot;3&quot;: 2, &quot;4&quot;: 3, &quot;5&quot;: 4, &quot;6&quot;: 5, &quot;7&quot;: 6, &quot;8&quot;: 7, &quot;9&quot;: 8, &quot;10&quot;: 9, &quot;jack&quot;: 10, &quot;queen&quot;: 11, &quot;king&quot;: 12, &quot;ace&quot;: 13, &quot;jocker&quot;: 0} def __init__(self, color, type): self.color = color self.type = type self.faces = list(Suite.faces)</pre><p>Here the class Suite represents a suite with the following attributes:</p><ul><li>Color(Red and Black)</li><li>Type(Spade,Ace,Heart,Die)</li><li>Faces in a suite</li></ul><p><strong>Python Hand Object:</strong></p><pre>class Hand(object): def __init__(self, name): self.name = name self.suites = {} def __add__(self, other): if isinstance(other, Card): suite = self.suites.get(other.suite) if suite is None: suite = [] self.suites[other.suite] = suite suite.insert(0, other)</pre><p>Here the class Hand represents a hand of a player with a function add(+) to add cards.</p><p><strong>Python Round Object:</strong></p><pre>class Round(object): def __init__(self, suite): self.cards = [] self.suite = suite self.maxPoint = 0 self.stand = {&quot;card&quot;: Card(suite)} def placeCard(self, other, hand): if isinstance(other, Card): card = other card.hand = hand self.cards.insert(0, card) if card.suite.type == self.suite.type: if int(card.point) &gt; self.maxPoint: self.maxPoint = card.point self.stand[&quot;hand&quot;] = hand self.stand[&quot;card&quot;] = card def stand(self): maxPoint = 0 for card in self.cards: if card.suite.type == self.suite.type: if int(card.point) &gt; maxPoint: maxPoint = card.point hand = card.hand stand = {&quot;hand&quot;: hand, &quot;card&quot;: card} return stand</pre><p>Here the class Round represents all the cards in a round of play, has the following methods:</p><ul><li>placeCard: a method used to place a card from the hand of a user</li><li>stand: method used to find winning stand-in around</li></ul><p><strong>Python CallBridge Object:</strong></p><pre>class CallBridge(object): cardSet = [Suite(&quot;black&quot;, &quot;spade&quot;), Suite(&quot;red&quot;, &quot;diamond&quot;), Suite(&quot;red&quot;, &quot;heart&quot;), Suite(&quot;black&quot;, &quot;club&quot;)] def __init__(self): self.rounds = [] def shuffle(self, hands): self.hands = list(hands) while len(CallBridge.cardSet) &gt; 0: suite = random.choice(CallBridge.cardSet) hand = random.choice(hands) if len(suite.faces) &gt; 0: random_choice = random.choice(suite.faces) card = Card(suite, random_choice) hand + card suite.faces.remove(random_choice) if len(suite.faces) == 0: CallBridge.cardSet.remove(suite) count = 0 for suite in hand.suites.values(): count += len(suite) if count == 13: hands.remove(hand) for hand in self.hands: print hand.name for cards in hand.suites.values(): for card in cards: print card.suite.type + &#39;:&#39; + str(card.face), print &quot;\n&quot; def play(self, winnerHand=None): if winnerHand is None: winnerHand = random.choice(self.hands) maxPoint = 0 suite = None placeCard = None for cards in winnerHand.suites.values(): for card in cards: if int(card.point) &gt;= maxPoint: maxPoint = card.point suite = card.suite placeCard = card if placeCard is not None: aRound = Round(suite) self.rounds.insert(0, aRound) aRound.placeCard(placeCard, winnerHand) winnerHand.suites.get(placeCard.suite).remove(placeCard) stand = aRound.stand for hand in self.hands: if hand != winnerHand: maxPoint = 0 placeCard = None if suite in hand.suites and len(hand.suites.get(suite)) &gt; 0: for card in hand.suites.get(suite): if int(card.point) &gt;= maxPoint: maxPoint = card.point placeCard = card if stand.get(&quot;card&quot;).point &gt; placeCard.point: minPoint = 13 placeCard = None for card in hand.suites.get(suite): if int(card.point) &lt;= minPoint: minPoint = card.point placeCard = card else: minPoint = 13 placeCard = None for cards in hand.suites.values(): for card in cards: if int(card.point) &lt;= minPoint: minPoint = card.point placeCard = card aRound.placeCard(placeCard, hand) hand.suites.get(placeCard.suite).remove(placeCard) print stand[&quot;hand&quot;].name + &#39;:&#39; + aRound.suite.type + &#39;==&gt;&#39;, for card in aRound.cards: print card.suite.type + &#39;:&#39; + str(card.face) +\ &#39;:&#39; + str(card.hand.name), print &quot;\n&quot; winnerHand = stand.get(&quot;hand&quot;) self.play(winnerHand) else: for aRound in self.rounds: card = aRound.stand.get(&quot;card&quot;) print &quot;winner: &quot; + aRound.stand.get(&quot;hand&quot;).name + \ &quot; card: &quot; + card.suite.color + &quot;:&quot; + card.face</pre><p>Here the class CallBridge represent the game playing strategy with the following methods:</p><ul><li>shuffle: this method is used to distribute cards among hands</li><li>play: this method is actually playing the game</li></ul><p>Here I missed the function of a bid.</p><p><strong>Python CardGame Object:</strong></p><pre>__author__ = &#39;Vicky&#39; from CallBridge import * vicky = Hand(&#39;Vicky&#39;) vikash = Hand(&#39;Vikash&#39;) ravi = Hand(&#39;Ravi&#39;) harris = Hand(&#39;Harris&#39;) hands = [vicky, vikash, ravi, harris] callBridge = CallBridge() callBridge.shuffle(hands) callBridge.play()</pre><p>Here the class CardGame where we set the players and play the game :).</p><pre>__author__ = &#39;Vicky&#39; class Suite(object): faces = [&quot;2&quot;, &quot;3&quot;, &quot;4&quot;, &quot;5&quot;, &quot;6&quot;, &quot;7&quot;, &quot;8&quot;, &quot;9&quot;, &quot;10&quot;, &quot;jack&quot;, &quot;queen&quot;, &quot;king&quot;, &quot;ace&quot;] points = {&quot;2&quot;: 1, &quot;3&quot;: 2, &quot;4&quot;: 3, &quot;5&quot;: 4, &quot;6&quot;: 5, &quot;7&quot;: 6, &quot;8&quot;: 7, &quot;9&quot;: 8, &quot;10&quot;: 9, &quot;jack&quot;: 10, &quot;queen&quot;: 11, &quot;king&quot;: 12, &quot;ace&quot;: 13, &quot;jocker&quot;: 0} def __init__(self, color, type): self.color = color self.type = type self.faces = list(Suite.faces) class Card(object): def __init__(self, suite, face=&quot;jocker&quot;): self.suite = suite self.face = face self.point = Suite.points.get(face) class Hand(object): def __init__(self, name): self.name = name self.suites = {} def __add__(self, other): if isinstance(other, Card): suite = self.suites.get(other.suite) if suite is None: suite = [] self.suites[other.suite] = suite suite.insert(0, other) class Round(object): def __init__(self, suite): self.cards = [] self.suite = suite self.maxPoint = 0 self.stand = {&quot;card&quot;: Card(suite)} def placeCard(self, other, hand): if isinstance(other, Card): card = other card.hand = hand self.cards.insert(0, card) if card.suite.type == self.suite.type: if int(card.point) &gt; self.maxPoint: self.maxPoint = card.point self.stand[&quot;hand&quot;] = hand self.stand[&quot;card&quot;] = card def stand(self): maxPoint = 0 for card in self.cards: if card.suite.type == self.suite.type: if int(card.point) &gt; maxPoint: maxPoint = card.point hand = card.hand stand = {&quot;hand&quot;: hand, &quot;card&quot;: card} return stand</pre><pre>__author__ = &#39;Vicky&#39; import random from Suite import * class CallBridge(object): cardSet = [Suite(&quot;black&quot;, &quot;spade&quot;), Suite(&quot;red&quot;, &quot;diamond&quot;), Suite(&quot;red&quot;, &quot;heart&quot;), Suite(&quot;black&quot;, &quot;club&quot;)] def __init__(self): self.rounds = [] def shuffle(self, hands): self.hands = list(hands) while len(CallBridge.cardSet) &gt; 0: suite = random.choice(CallBridge.cardSet) hand = random.choice(hands) if len(suite.faces) &gt; 0: random_choice = random.choice(suite.faces) card = Card(suite, random_choice) hand + card suite.faces.remove(random_choice) if len(suite.faces) == 0: CallBridge.cardSet.remove(suite) count = 0 for suite in hand.suites.values(): count += len(suite) if count == 13: hands.remove(hand) for hand in self.hands: print hand.name for cards in hand.suites.values(): for card in cards: print card.suite.type + &#39;:&#39; + str(card.face), print &quot;\n&quot; def play(self, winnerHand=None): if winnerHand is None: winnerHand = random.choice(self.hands) maxPoint = 0 suite = None placeCard = None for cards in winnerHand.suites.values(): for card in cards: if int(card.point) &gt;= maxPoint: maxPoint = card.point suite = card.suite placeCard = card if placeCard is not None: aRound = Round(suite) self.rounds.insert(0, aRound) aRound.placeCard(placeCard, winnerHand) winnerHand.suites.get(placeCard.suite).remove(placeCard) stand = aRound.stand for hand in self.hands: if hand != winnerHand: maxPoint = 0 placeCard = None if suite in hand.suites and len(hand.suites.get(suite)) &gt; 0: for card in hand.suites.get(suite): if int(card.point) &gt;= maxPoint: maxPoint = card.point placeCard = card if stand.get(&quot;card&quot;).point &gt; placeCard.point: minPoint = 13 placeCard = None for card in hand.suites.get(suite): if int(card.point) &lt;= minPoint: minPoint = card.point placeCard = card else: minPoint = 13 placeCard = None for cards in hand.suites.values(): for card in cards: if int(card.point) &lt;= minPoint: minPoint = card.point placeCard = card aRound.placeCard(placeCard, hand) hand.suites.get(placeCard.suite).remove(placeCard) print stand[&quot;hand&quot;].name + &#39;:&#39; + aRound.suite.type + &#39;==&gt;&#39;, for card in aRound.cards: print card.suite.type + &#39;:&#39; + str(card.face) +\ &#39;:&#39; + str(card.hand.name), print &quot;\n&quot; winnerHand = stand.get(&quot;hand&quot;) self.play(winnerHand) else: for aRound in self.rounds: card = aRound.stand.get(&quot;card&quot;) print &quot;winner: &quot; + aRound.stand.get(&quot;hand&quot;).name +\ &quot; card: &quot; + card.suite.color + &quot;:&quot; + card.face</pre><pre>__author__ = &#39;Vicky&#39; from CallBridge import * vicky = Hand(&#39;Vicky&#39;) vikash = Hand(&#39;Vikash&#39;) ravi = Hand(&#39;Ravi&#39;) harris = Hand(&#39;Harris&#39;) hands = [vicky, vikash, ravi, harris] callBridge = CallBridge() callBridge.shuffle(hands) callBridge.play()</pre><p><strong>Result of this game:</strong></p><p><strong>Vicky</strong><br> club:5 club:6 club:8 diamond:5 diamond:jack spade:5 spade:10 spade:ace spade:king spade:3 heart:5 heart:king heart:10</p><p><strong>Vikash</strong><br> club:4 club:jack club:9 diamond:4 diamond:10 diamond:2 diamond:king diamond:3 diamond:7 diamond:9 diamond:ace spade:8 heart:3</p><p><strong>Ravi</strong><br> club:10 club:king diamond:queen diamond:6 spade:4 spade:7 spade:6 heart:jack heart:7 heart:6 heart:4 heart:queen heart:2</p><p><strong>Harris</strong><br> club:7 club:2 club:ace club:3 club:queen diamond:8 spade:jack spade:2 spade:queen spade:9 heart:ace heart:8 heart:9</p><p>Player:Card =&gt; Cards in a round</p><p>Harris:heart==&gt; heart:2:Ravi heart:3:Vikash heart:5:Vicky heart:ace:Harris</p><p>Harris:club==&gt; club:10:Ravi club:4:Vikash club:5:Vicky club:ace:Harris</p><p>Vicky:spade==&gt; spade:4:Ravi spade:8:Vikash spade:ace:Vicky spade:queen:Harris</p><p>Vicky:heart==&gt; heart:8:Harris heart:4:Ravi diamond:2:Vikash heart:king:Vicky</p><p>Vicky:spade==&gt; spade:2:Harris spade:6:Ravi diamond:3:Vikash spade:king:Vicky</p><p>Vikash:diamond==&gt; diamond:8:Harris diamond:6:Ravi diamond:ace:Vikash diamond:jack:Vicky</p><p>Vikash:diamond==&gt; club:2:Harris diamond:queen:Ravi diamond:5:Vicky diamond:king:Vikash</p><p>Ravi:club==&gt; club:3:Harris club:king:Ravi club:6:Vicky club:jack:Vikash</p><p>Ravi:heart==&gt; heart:9:Harris diamond:4:Vikash heart:10:Vicky heart:queen:Ravi</p><p>Ravi:heart==&gt; club:7:Harris diamond:7:Vikash spade:3:Vicky heart:jack:Ravi</p><p>Ravi:heart==&gt; spade:9:Harris diamond:9:Vikash spade:5:Vicky heart:7:Ravi</p><p>Harris:spade==&gt; spade:jack:Harris club:9:Vikash spade:10:Vicky spade:7:Ravi</p><p>Harris:club==&gt; heart:6:Ravi diamond:10:Vikash club:8:Vicky club:queen:Harris</p><p>winner: Harris card: black:queen<br> winner: Harris card: black:jack<br> winner: Ravi card: red:7<br> winner: Ravi card: red:jack<br> winner: Ravi card: red:queen<br> winner: Ravi card: black:king<br> winner: Vikash card: red:king<br> winner: Vikash card: red:ace<br> winner: Vicky card: black:king<br> winner: Vicky card: red:king<br> winner: Vicky card: black:ace<br> winner: Harris card: black:ace<br> winner: Harris card: red:ace</p><p>Happy coding 🙂</p><p><em>Originally published at </em><a href="https://vickyblog2016.wordpress.com/2016/06/03/call-bridge-game-implemented-in-python/"><em>http://vickyblog2016.wordpress.com</em></a><em> on June 3, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d788aa4374e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ethereum As a BlockChain Platform]]></title>
            <link>https://medium.com/@vicky301186/ethereum-as-a-blockchain-platform-826ab4cd93e5?source=rss-ea2f87aaedec------2</link>
            <guid isPermaLink="false">https://medium.com/p/826ab4cd93e5</guid>
            <category><![CDATA[ethereum]]></category>
            <dc:creator><![CDATA[Vikash Singh]]></dc:creator>
            <pubDate>Tue, 31 May 2016 21:36:16 GMT</pubDate>
            <atom:updated>2019-09-23T09:32:06.487Z</atom:updated>
            <content:encoded><![CDATA[<p>Ethereum differs from BitCoin in following ways as implementation or usage perceptive:</p><p>Ethereum introduced the Proof of Stake model instead of the Proof of Work model were all miners put some stake to network before mining. This help network to punish those nodes which are trying to forgery the network with invalid block.</p><p>For solving the proof of work in Bitcoin needs high computation power(billions of hashing in a second) which is not possible for commodity hardware and so this may lead to centralization also due to high profitable consensus reward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/0*kmmIa4NKwiRp2Muu" /></figure><p>The mining ecosystem has solidified into a relatively small number of pools, and each one has a substantial portion of the network — and, of course, last week one of those pools, GHash, reached 51%.</p><p>Ethereum uses a <a href="https://en.wikipedia.org/wiki/Memory_bound_function">memory hardness</a> problem for consensus which is bound by Memory cycle and so nullify the advantage of having high computation power.</p><p><a href="https://github.com/ethereum/wiki/wiki/Ethash">Ethash</a> is the planned PoW algorithm for Ethereum 1.0. It is the latest version of Dagger-Hashimoto.</p><p>The general route that the algorithm takes is as follows:</p><ul><li>There exists a seed which can be computed for each block by scanning through the block headers up until that point(where the dataset is last updated).</li><li>From the seed, one can compute a 16 MB pseudorandom cache.</li><li>From the cache, we can generate a 1 GB dataset, with the property that each item in the dataset depends on only a small number of items from the cache.</li><li>This dataset is stored in a form of directed acyclic graphs to simultaneously achieve <a href="https://github.com/ethereum/wiki/blob/master/Dagger-Hashimoto.md">memory-hard computation but memory-easy validation</a>.</li><li>Mining involves grabbing random slices of the dataset and hashing them together. Verification can be done with low memory by using the cache to regenerate the specific pieces of the dataset that you need, so you only need to store the cache.</li></ul><p>The large dataset is updated once every 30000 blocks, so the vast majority of a miner’s effort will be reading the dataset, not making changes to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*r56ckSUdJIybvQ8n" /></figure><p>Example of this is currently out of syllabus for me, hopefully soon I able to give an example.</p><p><strong>Not just Crypto Currency:</strong></p><p>Bitcoin is just only about crypto-currency where person A transferring ownership for service or product. Ethereum is a platform that is intended to allow people to easily write decentralized applications (Dapps) using blockchain technology. The Ethereum blockchain has a built-in programming language, or as a consensus-based globally executed virtual machine. The part of the protocol that actually handles internal state and computation is referred to as the Ethereum Virtual Machine (EVM).</p><ol><li><strong>Externally owned account (EOAs)</strong>: an account controlled by a private key, and if you own the private key associated with the EOA you have the ability to send ether and messages from it.</li><li><strong>Contract</strong>: an account that has its own code, and is controlled by code.</li></ol><p><strong>The Fist Block — Genesis Block:</strong></p><p>The Genesis Block is the start block of the Blockchain. The consensus algorithm will ensure that no other node will agree with your version of the blockchain unless they have the same genesis block (and some other crucial parameters, discussed later).<br> Sample Genesis Block:<br> {<br> “nonce”: “0x0000000000000042”,<br> “mixhash”: “0x0000000000000000000000000000000000000000000000000000000000000000”,<br> “difficulty”: “0x4000”,<br> “alloc”: {},<br> “coinbase”: “0x0000000000000000000000000000000000000000”,<br> “timestamp”: “0x00”,<br> “parentHash”: “0x0000000000000000000000000000000000000000000000000000000000000000”,<br> “extraData”: “Custem Ethereum Genesis Block”,<br> “gasLimit”: “0xffffffff”<br> }<br> Save the JSON into a file lets call it genesis.json and save in your home directoory<br> geth -genesis &lt;genesis json file path&gt; -datadir &lt;some path to an empty folder&gt;</p><p><strong>Creata a EOA account:</strong> <br> In the geth console enter<br> personal.newAccount(“mypassword”); 0xcc8C048426978c5877212281b8a75F1B4E71a862<br> The last command above generated a new account with address ‘0xcc8C048426978c5877212281b8a75F1B4E71a862’<br> Once you’ve generated your account, quit geth with &lt;ctrl-c&gt; and remove every folder except keystore/ from your datadir:<br> $ cd &lt;your datadir&gt;<br> $ rm -rf `ls | grep -v keystore`<br> update your genesis block json, adding the following to the alloc key:</p><p>“alloc”: {<br> “&lt;your account address e.g. 0xcc8C048426978c5877212281b8a75F1B4E71a862&gt;”: {<br> “balance”: “10000000000000000000”<br> }<br> }<br> Now re-run the geth command using the newly updated genesis json file and the same datadir, when you check your account balance you will find you now have 10 ether:<br> &gt; primary = eth.accounts[0];<br> ‘0xcc8C048426978c5877212281b8a75F1B4E71a862′<br> &gt; balance = web3.fromWei(eth.getBalance(primary), “ether”);<br> ‘10’</p><p><a href="https://github.com/ethereum/go-ethereum/wiki/Setting-up-private-network-or-local-cluster">How to set a private local network</a>?</p><p>Ethereum also exposed those functions as JSON-RPC(localhost:8545) or Web3(Web Socket) which helps to build a Dapp around Ethereum network.</p><p>Get Account: POST <a href="http://localhost:8545/">http://localhost:8545</a></p><p>Get Account: POST <a href="http://localhost:8545/">http://localhost:8545</a></p><p>{<br> “jsonrpc”: “2.0”,<br> “method”: “eth_blockNumber”,<br> “params”: [],<br> “id”: 1<br> }</p><p><strong>Etherem Management API</strong></p><p>Geth provides the following management API namespaces:</p><ul><li>admin: Geth node management</li><li>debug: Geth node debugging</li><li>miner: Miner and <a href="https://github.com/ethereum/wiki/wiki/Ethash-DAG">DAG</a> management</li><li>personal: Account management</li><li>txpool: Transaction pool inspection</li></ul><p><em>Originally published at </em><a href="https://vickyblog2016.wordpress.com/2016/05/31/ethereum-as-a-block-chain-platform/"><em>http://vickyblog2016.wordpress.com</em></a><em> on May 31, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=826ab4cd93e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>