<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eswara Prasad on Medium]]></title>
        <description><![CDATA[Stories by Eswara Prasad on Medium]]></description>
        <link>https://medium.com/@eswaraprasad?source=rss-907fb3b469c0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DwzaXi9ZqFlLucDlliB_Hw.png</url>
            <title>Stories by Eswara Prasad on Medium</title>
            <link>https://medium.com/@eswaraprasad?source=rss-907fb3b469c0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:27:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eswaraprasad/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Glue Custom visual transform]]></title>
            <link>https://eswaraprasad.medium.com/glue-custom-visual-transform-3905a0dc4429?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3905a0dc4429</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-glue]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Wed, 25 Jun 2025 01:41:27 GMT</pubDate>
            <atom:updated>2025-06-25T01:41:27.368Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s AI-driven world, the focus is increasingly on eliminating manual processes and reducing heavy lifting in data engineering. The shift towards low-code or no-code solutions is gaining momentum, and visual ETL (Extract, Transform, Load) is at the forefront of this trend. AWS Glue Studio offers a powerful platform to perform visual ETL, making data transformation tasks simpler and more efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*GvI1ox5jsOaBk9fG.png" /></figure><h4>Glue Visual Studio</h4><p>AWS Glue Studio provides a user-friendly graphical interface that simplifies the process of creating, running, and monitoring data integration jobs. With its visual composition capabilities, you can design data transformation workflows with ease and seamlessly execute them on AWS Glue’s serverless, Apache Spark–based ETL engine.</p><p>In AWS Glue Studio, each node within the visual interface represents a specific function in an ETL pipeline. There are three main types of nodes: Source, Action, and Target.</p><ul><li><strong>Source:</strong> This node establishes connections to data sources such as RDS, S3, SharePoint, Salesforce, and more.</li><li><strong>Action:</strong> This node allows you to apply business logic and write custom code to meet data transformation and analytics requirements.</li><li><strong>Target:</strong> This node loads the transformed data into the desired destination, such as S3, Redshift, Snowflake, or other supported storage solutions.</li></ul><p>AWS Glue Studio offers a wide range of predefined transformations that can be easily applied using a drag-and-drop interface. However, if the required transformation isn’t available, you can use the “Custom Transform” node to write your own PySpark code to implement the desired logic. The challenge is that custom transformations are created at the job level and are not designed for reusability, making it difficult to share or reuse them across different jobs or with other developers.</p><p>This is where the <strong>Custom Visual Transform</strong> comes in. It allows you to create a custom action node that performs the transformation and can be reused across multiple jobs and by different developers — without writing any code each time. With this feature, you can build a reusable library of action nodes that can be applied to various jobs, solving the reusability challenge and streamlining the development process.</p><p>Creating a Custom Visual Transform involves following specific steps to convert your custom code into a reusable visual component:</p><ol><li><strong>Create a Configuration File:</strong> This file defines the structure and behavior of your custom visual transform, including input and output schema details.</li><li><strong>Write Custom Code:</strong> Implement the logic for your custom transformation using PySpark. This code will be the core of your custom action node.</li><li><strong>Upload to S3 Bucket:</strong> Store both the configuration file and custom code in an S3 bucket, which AWS Glue Studio will reference to make your custom transform available in the visual interface.</li></ol><p>We will build a custom action node that masks personally identifiable information (PII) in our dataset. This is a common requirement when handling sensitive data, and creating a reusable custom visual transform will allow us to apply PII masking easily across multiple jobs.</p><h3><strong>Create a Configuration File</strong></h3><pre>{<br>    &quot;name&quot;: &quot;custom_filter_state&quot;,<br>    &quot;displayName&quot;: &quot;Apply Masking&quot;,<br>    &quot;description&quot;: &quot;This state will perform masking for email and Phone columns&quot;,<br>    &quot;functionName&quot;: &quot;custom_filter_state&quot;,<br>    &quot;parameters&quot;: [<br>       {<br>        &quot;name&quot;: &quot;colName&quot;,<br>        &quot;displayName&quot;: &quot;Name of the Column with Email&quot;,<br>        &quot;type&quot;: &quot;list&quot;,<br>        &quot;listOptions&quot;:&quot;column&quot;,<br>        &quot;listType&quot;:&quot;str&quot;,<br>        &quot;description&quot;: &quot;Name of the column in the data that holds the state postal code&quot;<br>       }<br>      ]<br>}</pre><p>The configuration file for a Custom Visual Transform is a JSON file that defines the UI elements and functionality for the masking node. Below are the key sections of the JSON file:</p><ul><li><strong>name:</strong> The system identifier for the transform.</li><li><strong>displayName:</strong> The name displayed in AWS Glue Studio’s visual editor.</li><li><strong>description:</strong> A brief, searchable description in AWS Glue Studio.</li><li><strong>functionName:</strong> The Python function to be invoked in the script.</li><li><strong>parameters:</strong> A list of input fields used for configuring the transform.</li></ul><p>Within <strong>parameters</strong>, you have:</p><ul><li><strong>name:</strong> The parameter name, passed as a named argument to the Python function. Follows Python variable naming rules.</li><li><strong>displayName:</strong> The label shown in the AWS Glue Studio editor for the parameter.</li><li><strong>type:</strong> The data type of the parameter (e.g., string, list). In our example, it’s a list for selecting values.</li><li><strong>listOptions:</strong> A list of options to display in a Select or Multiselect UI, auto-populated with column names from the parent node’s schema.</li><li><strong>listType:</strong> Specifies the type of list (for type = &#39;list&#39;).</li></ul><p>This configuration creates a node that retrieves all the column names from the parent schema and displays them in a multiselect dropdown. The user can then select the PII fields that need to be masked. The resulting UI will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4dTKe35qXNc_2eRG61oGrQ.jpeg" /></figure><h3>Create the Custom Transform Code</h3><pre>from awsglue import DynamicFrame<br><br>def myTransform(self, email, phone, age=None, gender=&quot;&quot;, <br>                      country=&quot;&quot;, promotion=False):<br>   resulting_dynf = # do some transformation on self <br>   return resulting_dynf<br>   <br>DynamicFrame.myTransform = myTransform</pre><p>The custom code for the Custom Visual Transform follows the structure where self refers to the DynamicFrame that needs to be transformed.</p><p><strong>Parameter Naming:</strong></p><ul><li>The parameter names in the code must match those defined in the config file.</li><li>If a parameter is optional, provide a default value in the function.</li></ul><p><strong>Function Naming:</strong></p><ul><li>Ensure the function name in the code matches the one defined in the config file.</li><li>It’s a good practice to keep both the function name and the name field in the config file identical. This avoids confusion between the two.</li></ul><p><strong>Assigning Results:</strong></p><ul><li>In the final line, assign the function’s results to the output in the config file.</li><li>Always return the transformed result as a DynamicFrame.</li><li>You can convert the DynamicFrame into a DataFrame if needed for further transformations using libraries like pandas, numpy, etc.</li></ul><p>Below is the code that masks the columns selected by the user in the UI.</p><pre>from awsglue import DynamicFrame<br>from pyspark.sql.functions import udf<br>from pyspark.sql.types import StringType<br>import re<br><br># Define the masking function for email<br>def mask_email(email):<br>    if email and isinstance(email, str):<br>        return re.sub(r&#39;(\w{2})(.*)(@.*)&#39;, r&#39;\1***\3&#39;, email)<br>    return email<br><br># Define the custom transformation<br>def custom_filter_state(self, colName):<br>    # Create UDF for Spark SQL<br>    mask_udf = udf(mask_email, StringType())<br>    <br>    # Convert DynamicFrame to DataFrame for transformation<br>    df = self.toDF()<br>    <br>    # Apply masking to each column in colName<br>    for col in colName:<br>        df = df.withColumn(col, mask_udf(df[col]))<br>    <br>    # Convert DataFrame back to DynamicFrame<br>    return DynamicFrame.fromDF(df, self.glue_ctx, self.name)<br><br># Assign custom transformation to DynamicFrame class<br>DynamicFrame.custom_filter_state = custom_filter_state</pre><h3>Deploying Custom Visual Transforms</h3><p><strong>Save Configuration and Code Files:</strong></p><ul><li>Save the configuration file (mytransform.json) and the transformation code (mytransform.py).</li><li>Ensure both files share the same name to simplify the integration between the configuration and the code.</li></ul><p><strong>Upload to S3 Bucket:</strong></p><ul><li>Upload both files to the specified S3 bucket location:<br>s3://aws-glue-assets-&lt;accountid&gt;-&lt;region&gt;/transforms/.</li><li>This S3 path makes the transform accessible for your AWS Glue jobs in that region and account.</li></ul><p><strong>Cross-Region and Cross-Account Reusability:</strong></p><p>To reuse the transform across multiple AWS regions or accounts, you can:</p><ul><li><strong>Manually replicate</strong> the configuration and code files across your accounts and regions.</li><li><strong>Leverage S3 automatic replication</strong>, allowing your transforms to be synchronized automatically for reuse across different environments.</li></ul><h3><strong>End To End Pipeline Implementation</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P75h-_LhAAUg5idJTdhRlQ.png" /></figure><p>After uploading the required configuration and code files to the designated S3 bucket, your custom node will appear in the “Transforms” tab within AWS Glue Studio shortly. To demonstrate an end-to-end pipeline, consider a dataset containing two PII fields: personal_email_address and work_email_address. By selecting these fields in your custom masking node, AWS Glue will automatically apply the transformations. In the &quot;Data Preview&quot; tab, you can verify that the masking has been successfully applied. This process showcases how custom visual transforms enable seamless data transformations within the AWS Glue Studio environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OFbh5VyHpVH5rUgVzo-QzQ.png" /></figure><h3>Conclusion</h3><p>By leveraging custom visual transforms in AWS Glue Studio, you can build specialized ETL transformations that fit your exact needs. From setting up configuration files to deploying the solution across regions, the entire process is designed to enhance and simplify data management tasks. AWS Glue’s flexibility and scalability make it an excellent choice for building robust, automated data pipelines in the cloud.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3905a0dc4429" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas vs Dask : The Power of Parallel Computing !]]></title>
            <link>https://medium.com/featurepreneur/pandas-vs-dask-the-power-of-parallel-computing-994a202a74bd?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/994a202a74bd</guid>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[featurepreneur]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[dask]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Thu, 24 Jun 2021 09:57:28 GMT</pubDate>
            <atom:updated>2021-06-24T09:57:28.730Z</atom:updated>
            <content:encoded><![CDATA[<h3>Pandas vs Dask : The Power of Parallel Computing!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fg8gM37Q1qQWF3KT.jpg" /></figure><p>Assuming you are running code on the personal laptop, for example, with 32GB of RAM, you have been handling large datasets for your machine learning projects. But, due to your PC configuration, It takes much time to process the data. So, you have decided to upgrade my pc to a better configuration. But, still, you faced the issue. After some research, I found the problem is with pandas and not with my PC. After some research, I found many solutions like PySpark, Dask, Modin, etc. so I’ve decided to benchmark the most popular ones and allow myself to know in which case use what. Here, we are gonna benchmark Pandas and Dask.</p><h3>Dask:</h3><p>It scales NumPy, pandas, and sci-kit-learn. Just use Dask instead of those libraries, and you’re good to go. What makes it fast that it allows us to use multiple cores and disk for spilling data if it doesn’t fit in memory. The most significant advantage here I’d say that it allows more parallelism on your local machine. Faster results with less effort. The below image explains parallel computing in dask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gOL8-xbODVm5JzEp" /></figure><p>Let&#39;s get started with Benchmarking!</p><h3>1.Simple Function</h3><p>Let&#39;s write a simple function in python as well using dask framework. It&#39;s just an additional function, where I have included the sleep function to easily interpret the results. Python takes around 3s.</p><h4>Pandas:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*yjeIPf1qA5lnYngN9nnQYA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*8f9koCG4b-EM_-XNM_PbiA.png" /><figcaption>Python Performance</figcaption></figure><h4>Dask:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*-Oc8yR63HSyKKEzrq1LMXg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*GT8wWTIAxqA0VoS8fKZrNw.png" /><figcaption>Dask Performance</figcaption></figure><h3>2. Reading a DataFrame</h3><p>Reading a data frame is the most common thing while getting started with machine learning. Here, Pandas uses the traditional procedure of reading data frames, but dask uses parallel computing. Where the data frame is split into parts and then it is processed. I have used a million-row data frame here. You can see the drastic difference below!</p><h4>Pandas:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*EGnpWBFMpeoFta48FDEb0w.png" /></figure><h4>Dask:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*QZ2JhCSfIXA4P0N2QgpJNg.png" /></figure><h3>3. Exporting to CSV</h3><p>Processing our data frame to CSV is a CPU-intensive process. In the case of large datasets, but dask lags in this part. As I have mentioned earlier, dask splits the data into small chunks according to the size. Whenever you export a data frame using dask. It will be exported as 6 equally split CSVs(the number of splits depends on the size of data or upon your mention in the code). But, Pandas exports the dataframe as a single CSV. So, Dask takes more time compared to Pandas.</p><h4>Pandas</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*2lx6dLayYJK1ETMF-0Cx8A.png" /></figure><h4>Dask</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*5OoiWDjoqZIoCRsRp4NbFA.png" /></figure><h3>4.Merging Dataframes</h3><p>Merging data frames is intensive in the part of data manipulation. So, we added this process in our benchmarking process. So, we can get a clear understanding of the Pandas and Dask.</p><h4>Pandas:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*I92g4R6s-oucKuiDuVdNOQ.png" /></figure><h4>Dask:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*p5gitYZDCpvZU8_Uzgu0MA.png" /></figure><h3>Conclusion</h3><p>So far, We saw the Pros and Cons of both pandas and dask. So, due to parallel computing in dask a single task is done by four workers unlike the single worker in pandas. Learning dask is also not a quite tough job. It uses the Pandas API that overlays with this technology, That’s It for this article. Enjoy Learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=994a202a74bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/pandas-vs-dask-the-power-of-parallel-computing-994a202a74bd">Pandas vs Dask : The Power of Parallel Computing !</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to install Mac Theme on Ubuntu]]></title>
            <link>https://medium.com/featurepreneur/how-to-install-mac-theme-on-ubuntu-a0aea7a9136?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0aea7a9136</guid>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[themes]]></category>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[featurepreneur]]></category>
            <category><![CDATA[terminal]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Thu, 24 Jun 2021 09:54:30 GMT</pubDate>
            <atom:updated>2021-06-24T09:54:30.962Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*zyVn9ca2QIWmI9rZ" /></figure><p>You will need this basic requirement to get the work done: Ubuntu, Gnome Tweaks, Plank, Privileged access to your Linux system as root or via the sudo command.</p><p><strong>Step 1</strong>: First of all, we have to install gnome tweaks. Open the terminal and follow the commands.</p><pre>$ sudo add-apt-repository universe<br>$ sudo apt install gnome-tweak-tool<br>$ gnome-tweaks   #to check for installation</pre><p><strong>Step 2 </strong>: Next, we are going to install the Mojave macOS theme. Now go to this <a href="https://www.gnome-look.org/p/1275087/">link</a> and install as per your wish, Locate it in your downloads folder. I am going for a Dark theme.</p><pre>$ cd Downloads<br>$ tar xf Mojave-dark.tar.xz<br>$ mkdir ~/.themes<br>$ mv Mojave-dark ~/.themes/</pre><p><strong>Step 3</strong> : Once the theme is installed you can apply the theme to your current desktop. Open up the Gnome Tweaks tool, Go to Appearance and change to Application’s theme appearance to Mojave-dark.</p><p><strong>Step 4</strong> : Next step is to install Mojave CT icons. Now go to this <a href="https://www.gnome-look.org/p/1210856/">link</a> and install as per your wish, Locate it in your downloads folder.</p><pre>$ cd Downloads<br>$ tar xf Mojave-CT-Night-Mode.tar.xz<br>$ mkdir ~/.icons<br>$ mv Mojave-CT-Night-Mode ~/.icons/</pre><p><strong>Step 5</strong> : Once the icon is installed you can apply the icon to your current desktop. Reload the Gnome tweak, Go to Appearance and set the icon appearance to Mojave.</p><p><strong>Step 6 </strong>: Next, set up the macOS panel. To do so first install Plank.</p><pre>$ sudo apt install plank</pre><p>Once the Plank application is installed launch it from your Activities menu. At this point, you should see the macOS panel at the bottom of your desktop. Hold the left CTRL and right-click to the bottom macOS panel to open up Preferences. Customise the macOS panel to fit your desired look and feel.</p><p>That’s It. you are all set to enjoy your mac-theme experience in your Ubuntu :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0aea7a9136" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/how-to-install-mac-theme-on-ubuntu-a0aea7a9136">How to install Mac Theme on Ubuntu</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Statistics: Measures of Central Tendency]]></title>
            <link>https://medium.com/featurepreneur/statistics-measures-of-central-tendency-32ffd17c5293?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/32ffd17c5293</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[central-tendency]]></category>
            <category><![CDATA[featurepreneur]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Wed, 23 Jun 2021 18:27:27 GMT</pubDate>
            <atom:updated>2021-06-23T18:27:27.710Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*5vg0cFpodEPCTVi4" /></figure><p>Whenever any metric of interest (e.g. share price of a stock, the height of an individual, score of a student, delay in the arrival of a flight, etc.) is measured, a fairly large number of observations in the sample tend to centre around a single value. This value can be considered as a single representative value of the desired metric for the sample. As these values give the ‘location’ of a ‘central’ value of the sample, they are called more commonly “Measures of Central Tendency” of the sample.</p><p>There are different statistics to measure the central tendency of any data. In this module, we will cover the most important measures of Central Tendency — Mean, Median, Mode, Percentile and Quartile</p><h3>Mean</h3><p>Mean is the most common measure of central tendency. Mean can be defined for all ratio-scale and interval-scale data. To calculate the mean, simply add all of your numbers together. Next, divide the sum by however many numbers you added. The result is your <em>mean </em>or average score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/0*Si6ntF8dlgrQK_Md" /></figure><h3>Median</h3><p>Median is the middle number of any data series which has been sorted in ascending (i.e. lowest to highest) or descending (i.e. highest to lowest) order. If the series is odd, then the median is exactly the middle number. However, if the series is even, then it is the average of the middle two numbers. Median is defined for ordinal data too, along with interval-scale or ratio-scale data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y5jE0clhcBM7f1V3" /></figure><h3>Mode</h3><p>In statistics, the mode in a list of numbers refers to the integers that occur most frequently. Unlike the median and mean, the mode is about the frequency of occurrence. There can be more than one mode or no mode at all; it all depends on the data set itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*wRnhZnB7x5GpJVCE" /></figure><h3>Percentiles</h3><p>In statistics, percentiles are used to understand and interpret data. The <em>n</em>th percentile of a set of data is the value at which <em>n</em> per cent of the data is below it. In everyday life, percentiles are used to understand values such as test scores, health indicators, and other measurements. An 18-year-old male who is only five and a half feet tall, on the other hand, is in the 16th percentile for his height, meaning only 16 per cent of males his age are the same height or shorter.</p><p>percentiles should not be confused with <a href="https://www.thoughtco.com/how-to-calculate-percent-608321">percentages</a>. The latter is used to express fractions of a whole, while percentiles are the values below which a certain percentage of the data in a data set is found. In practical terms, there is a significant difference between the two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*c4rvPY701_ZweqGfYWmrxw.png" /></figure><h3>Quartiles</h3><p>A quartile is a statistical term that describes a division of observations into four defined intervals based on the values of the data and how they compare to the entire set of observations. Quartiles are the values that divide a list of numerical data into three quarters. The middle part of the three quarters measures the central point of distribution and shows the data which are near to the central point.</p><p>The lower part of the quarters indicates just half the information set which comes under the median and the upper part shows the remaining half, which falls over the median. In all, the quartiles depict the distribution or dispersion of the data set</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*in4nSe1BBlxDw5bR" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32ffd17c5293" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/statistics-measures-of-central-tendency-32ffd17c5293">Statistics: Measures of Central Tendency</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Statistics: Classification of Data]]></title>
            <link>https://medium.com/featurepreneur/statistics-classification-of-data-e8c8f8666e6f?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8c8f8666e6f</guid>
            <category><![CDATA[featurepreneur]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Wed, 23 Jun 2021 18:27:03 GMT</pubDate>
            <atom:updated>2021-06-23T18:27:03.869Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*fgLSMbqiwCDdaMMC" /></figure><p>Statistics is a field of mathematics that is universally agreed to be a prerequisite for a deeper understanding of machine learning. In almost every endeavor of human activity, the scientific method has proven effective for solving problems and improving performance. Statistical methods are also commonly used in business practice, e.g. to forecast demand for goods and services or to determine the most efficient method of operation. Actuaries use statistical methods to assess risk levels and set premium rates for the insurance and pension industries. There are various types of data in statistics</p><p>Data can be described as a collection of values either qualitative or quantitative. Data can be numbers or measurement or scale or rank or it can even be purely qualitative. Data points can be classified into five broad categories — Nominal, Ordinal, Interval, Ratio</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/0*igBwpV3Cz75EW1Hs" /></figure><h3>Nominal Data:</h3><p>Nominal data represent qualitative information without order. It gives a label to the data. whereas the classification describes the data. For example, the Type of gender could be either Men, Women, Transgender. . It is important to note the absence of order. Here, Order doesn’t determine the priority. In an exam, Gender doesn’t matter. The person, who puts more effort will ace the examination. Nominal data is also called categorical data</p><h3>Ordinal Data:</h3><p>Ordinal data represent qualitative information without order, indicates that the measurement classifications are different and can be ranked. For example The grading system of A, B, C, D. When Nominal data has some measure of rank built within, it is referred to as Ordinal data. A letter grade of A in the exam is ranked higher than a grade of B. Ordinal data is also known as ordered categorical data.</p><h3>Interval data:</h3><p>Interval data is measured along a numerical scale that has equal distances between adjacent values. These distances are called “intervals”.there is no complete zero in interval data, which makes it different from the ratio. For example: In an Examination, the Difference between 90 and 100 is similar to the difference between 80 and 90.In other words, if the difference between two values is meaningful, then the data is classified as interval data. Quasi-Interval data is a special case of Interval Data, that falls between ordinal and interval. For example, An opinion poll with options from Strongly Disagree to Strongly Agree.</p><h3>Ratio:</h3><p>Ratio data measures have equal intervals and a ‘true’ zero point. It has all the properties of interval data with a clear definition of true zero points. Unlike on an interval scale, a zero on a ratio scale means there is a total absence of the variable you are measuring. For example, Weight, height, price are all ratio variables. A package of 100 grams is twice in weight of a package of 50 grams. Similarly, a height of 5 ft. is 5 times the height of 1 ft.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*3FtSRszxGCHDwCAw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8c8f8666e6f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/statistics-classification-of-data-e8c8f8666e6f">Statistics: Classification of Data</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HyperParameter Tuning and Its Types]]></title>
            <link>https://medium.com/featurepreneur/hyperparameter-tuning-and-its-types-20184cd0ee0a?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/20184cd0ee0a</guid>
            <category><![CDATA[featurepreneur]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[differential-evolution]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Mon, 21 Jun 2021 05:16:13 GMT</pubDate>
            <atom:updated>2021-06-21T05:16:13.790Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*P71yZGHEKJ83WFIH" /></figure><p>Machine learning is the way you predict future values based on historical data. There is a list of different machine learning models. They all are different in some way or the other, but what makes them different is nothing but input parameters for the model. These input parameters are named Hyperparameters<strong>.</strong></p><p>Hyperparameter tuning is the process of tuning the parameters present as the tuples while we build machine learning models. These parameters are defined by us which can be manipulated according to the programmer&#39;s wish. Models can have many hyperparameters and finding the best combination of parameters can be treated as a search problem. There are various methods of tuning</p><ul><li><strong>Grid SearchCV</strong></li><li><strong>Random SearchCV</strong></li><li><strong>Bayesian Search</strong></li><li><strong>Evolution Search</strong></li></ul><h4><strong>Grid SearchCV</strong></h4><p>Grid Search is an effective method for adjusting the parameters in supervised learning and is used to improve the generalization performance of a model. With Grid Search, we try all possible combinations of the parameters of interest and find the best ones. It is great for spot-checking combinations that are known to perform well generally. If you increase the number of combinations then time complexity will increase. So, It’s not much suited for large combinations of parameters. You can get the code implementation <a href="https://www.kaggle.com/eswarpd001/grid-search-algorithmn">here</a>.</p><h4><strong>Random SearchCV:</strong></h4><p>Random Search defines a search space as a bounded domain of hyperparameter values and randomly sample points in that domain. It is great for discovery and getting hyperparameter combinations that you would not have guessed intuitively, although it often requires more time to execute. However, if the number of parameters to consider is particularly high and the magnitudes of influence are imbalanced, the better choice is to use the Random Search. You can get the code implementation <a href="https://www.kaggle.com/eswarpd001/randomsearch-algorithmn">here</a>.</p><p><strong>Bayesian Search</strong></p><p>Bayesian optimization builds a probabilistic model of the function mapping hyperparameter values to the objective function evaluated on a validation set. It computes the conditional probability of the objective function value for a validation set given a set of values of hyperparameters used to train the machine learning model which is the probability of objective function value and hyperparameter validation set. It&#39;s a very advanced search algorithm, that&#39;s best suited for large parameter combinations. Bayesian Optimization provides a probabilistically principled method for global optimization. You can get the code implementation <a href="https://www.kaggle.com/eswarpd001/bayesian-optimization">here</a>.</p><h4><strong>Evolutionary Search:</strong></h4><p>Genetic algorithms provide a powerful technique for hyperparameter tuning, but they are quite often overlooked genetic algorithms that make use of basic concepts from evolution by natural selection to optimize arbitrary functions.It basically works on Charles Darwins Theory. So, Parameters are equivalent to genes in biological systems. Its works on three major steps Mutation, Recombination, and Replacement. These steps are recursively done until the convergence of the parameter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g2iVUjf5JQJQyA41" /></figure><h4>Conclusion:</h4><p>These are types of hyperparameter tuning so far, Every algorithm has its own pros and cons. Whereas choosing wisely for your requirement would be great. The below image gives you a clear understanding of the performance benchmarking of these algorithms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20184cd0ee0a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/hyperparameter-tuning-and-its-types-20184cd0ee0a">HyperParameter Tuning and Its Types</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Auto-ARIMA?]]></title>
            <link>https://medium.com/featurepreneur/what-is-auto-arima-b8025c6d732d?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8025c6d732d</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[time-series-model]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[arima]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Thu, 20 May 2021 07:22:13 GMT</pubDate>
            <atom:updated>2021-05-20T07:22:13.592Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/839/0*CMSrRGJHPAB6HeOH" /></figure><p>Auto_arima, a routine from IMSL, applies automated configuration tasks to the autoregressive integrated moving average (ARIMA) model. Before getting into it, we have to get a fair understanding of ARIMA Models.</p><h4>What is ARIMA:</h4><p>An autoregressive integrated moving average, or ARIMA, is a statistical analysis model that uses time-series data to better understand the data set or predict future trends. A statistical model is autoregressive if it predicts future values based on past values. For example, an ARIMA model might seek to predict a stock’s future prices based on its past performance or forecast a company’s earnings based on past periods.</p><p>We can split the Arima term into three terms, AR, I, MA:</p><p>AR(p) stands for the autoregressive model, the p parameter is an integer that confirms how many lagged series are going to be used to forecast periods ahead.</p><p>I(d) is the differencing part, the d parameter tells how many differencing orders are going to be used to make the series stationary.</p><p>MA(q) stands for moving average model, the q is the number of lagged forecast error terms in the prediction equation. SARIMA is a seasonal ARIMA and it is used with time series with seasonality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/0*5eoQ-yE91Gce8lI0" /></figure><h4>Auto ARIMA:</h4><p>For a while, we have been going through the process of manually fitting different models and deciding which one is best. So, we are going to automate the process. Basically, it takes the data and fits many models in a different order before comparing the characteristics. However, the processing time increases substantially, when we try to fit complex models.</p><h4><strong>How does it work:</strong></h4><p>In Time Series Analysis, a correct model should yield the highest Log-Likelihood and requires the lowest AIC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*S60i35hXTWUuQe5Q" /></figure><p>The AIC and BIC of a model depend on the log-likelihood, to be more precise AIC and BIC uses Log-Likelihood in their formula. This would indirectly imply that the log-likelihood is high therefore this method cycles through the model with varying orders and specifications and out of those returns the one with the lowest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/0*NtlWnaIFezjmILLz" /><figcaption>Formula of Log-Likelihood</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*On1JdhwJy0-q3hVb" /><figcaption>Formula of AIC</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/0*ErzZHVf1gIk7omnF" /><figcaption>Formula of BIC</figcaption></figure><h4>Pros:</h4><ul><li>It saves an enormous amount of time</li><li>eliminate the need to understand the statistics and theory behind the model selection.</li><li>this method will also reduce the risk of human error and the possible mistakes caused by an incorrect interpretation of the results.</li></ul><h4>Cons:</h4><ul><li>Blindly putting our faith into one criterion</li><li>Never really see how well the other models perform</li><li>Topic expertise</li></ul><p>Check out my <a href="https://github.com/eswaraprasad001/Time-Series-Analysis-101">Time-Series-101</a>, which covers all the major concepts in Time Series Analysis. Keep Exploring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8025c6d732d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/what-is-auto-arima-b8025c6d732d">What is Auto-ARIMA?</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Scaling: Normalization vs Standardization]]></title>
            <link>https://medium.com/featurepreneur/feature-scaling-normalization-vs-standardization-7b9cd09d0b4b?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b9cd09d0b4b</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-scaling]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Thu, 20 May 2021 07:11:08 GMT</pubDate>
            <atom:updated>2021-05-20T07:11:08.443Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/992/0*huWycVZbx932S-7I" /></figure><p>Feature scaling is one of the most important data preprocessing steps in machine learning. Algorithms that compute the distance between the features are biased towards numerically larger values if the data is not scaled. Tree-based algorithms are fairly insensitive to the scale of the features. Also, feature scaling helps machine learning, and deep learning algorithms train and converge faster.</p><h3>Why Feature Scaling?</h3><p>If we have a dataset, which has two features namely height and weight. There will be two measures magnitude and Unit. The unit represents the form of the value like kg, gm, mg etc. Magnitude represents the exact value. Basically, machine learning algorithms use euclidean distance to predict values. If the magnitude is very high, then it will take more time to analyze the value. So, we need feature scaling in these situations. There are two major types Normalization and Standardization.</p><h3>Normalization:</h3><p>Normalization (also called, <strong>Min-Max normalization</strong>) is a scaling technique such that when it is applied the features will be rescaled so that the data will fall in the <em>range of [0,1]</em></p><p>The normalized form of each feature can be calculated as follows:</p><pre><strong><em>                X’ = (X — X_min)/(X_max — X_min)</em></strong></pre><h3>Standardization:</h3><p>Standardization (or <strong>Z-score normalization</strong>) is the process of rescaling the features so that they’ll have the properties of a Gaussian distribution with</p><p><em>μ</em>=0 and σ=1</p><p>where <em>μ</em> is the mean and <em>σ</em> is the standard deviation from the mean; standard scores (also called <strong><em>z</em></strong> scores) of the samples are calculated as follows:</p><pre>                     <strong><em> Z = (X - μ)/σ</em></strong></pre><h4>Which one would you choose?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lcc-rm5S4UITW41m" /><figcaption>Photo by <a href="https://unsplash.com/@soymeraki?utm_source=medium&amp;utm_medium=referral">Javier Allegue Barros</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For example, in clustering analyses, standardization may be especially crucial in order to compare similarities between features based on certain distance measures. However, this doesn’t mean that normalization is not useful at all! A popular application is image processing, where pixel intensities have to be normalized to fit within a certain range (i.e., 0 to 255 for the RGB color range). Also, a typical neural network algorithm requires data on a 0–1 scale.</p><p>If you would like to see code snippets of these concepts. Just jump into my notebook <a href="https://www.kaggle.com/eswarpd001/feature-engineering-standardization-normalization">here</a>. Keep Exploring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b9cd09d0b4b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/feature-scaling-normalization-vs-standardization-7b9cd09d0b4b">Feature Scaling: Normalization vs Standardization</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Roadmap for Time-Series Analysis]]></title>
            <link>https://medium.com/featurepreneur/a-roadmap-for-time-series-analysis-3faf49b2126?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3faf49b2126</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[timeseries-forecasting]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[statsmodels]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Mon, 17 May 2021 14:31:29 GMT</pubDate>
            <atom:updated>2021-05-17T14:31:29.643Z</atom:updated>
            <content:encoded><![CDATA[<h3>A RoadMap to Time-Series Analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*EdtFCsUQPiDgMGT0" /></figure><p>Time Series analysis is a technique, Which unites finance and technology. Here we automate every aspect of finance using the tools in programming. Here, I will walk you through every concept in Time Series in this article. If you like to see the python implementation of the topics in the upcoming article. Just jump into my repo <a href="https://github.com/eswaraprasad001/Time-Series-Analysis-101"><strong>here</strong></a>.</p><h3><strong>Introduction:</strong></h3><ul><li>Time-Series data is a series of data points or observations recorded at different or regular time intervals. In general, a time series is a sequence of data points taken at equally spaced time intervals. The frequency of recorded data points may be hourly, daily, weekly, monthly, quarterly or annually.</li><li>Time-Series Forecasting is the process of using a statistical model to predict future values of a time series based on past results.</li></ul><h3><strong>Jargons to Know:</strong></h3><ul><li><strong>Trend</strong> — The linear increasing or decreasing behaviour of the series over time. A trend can be increasing(upward), decreasing(downward), or horizontal(stationary).</li><li><strong><em>Seasonality</em></strong> — The repeating patterns or cycles of behaviour over time. Some examples include an increase in water consumption in summer due to hot weather conditions.</li><li><strong><em>ETS</em></strong> <strong><em>Decomposition</em></strong> — ETS Decomposition is used to separate different components of a time series. The term ETS stands for Error, Trend and Seasonality.</li><li><strong><em>Stationarity</em></strong>- It shows the mean value of the series that remains constant over the time period. If past effects accumulate and the values increase towards infinity then stationarity is not met</li><li><strong><em>Differencing</em></strong>- Differencing is used to make the series stationary and to control the auto-correlations. There may be some cases in time series analyses where we do not require differencing and over-differenced series can produce the wrong estimate.</li><li><strong><em>Dependence</em></strong>- It refers to the association of two observations of the same variable at prior time periods.</li><li><strong><em>Noise</em></strong>- The variability in the observations that cannot be explained by the model.</li></ul><p>After getting a pretty understanding of these topics, you can jump into the core concepts in time series.</p><h3><strong>Smoothing Techniques:</strong></h3><p>The smoothing process is essential to reduce the noise present in our series and point out the true patterns that may present over time. There are three important smoothing methods in time series analysis.</p><ul><li><strong>Single Exponential smoothing:</strong></li></ul><p>Single Exponential Smoothing, SES for short, also called Simple Exponential Smoothing is a time series forecasting method for univariate data without a trend or seasonality.</p><ul><li><strong>Double Exponential Smoothing</strong></li></ul><p>Double Exponential Smoothing is an extension to Exponential Smoothing that explicitly adds support for trends in the univariate time series. Double Exponential Smoothing with an additive trend is classically referred to as Holt’s linear trend model, named for the developer of the method Charles Holt.</p><ul><li><strong>Triple Exponential Smoothing</strong></li></ul><p>Triple Exponential Smoothing is an extension of Exponential Smoothing that explicitly adds support for seasonality to the univariate time series. This method is sometimes called Holt-Winters Exponential Smoothing, named for two contributors to the method: Charles Holt and Peter Winters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/0*4lhXO4j3xVi-8L5w.png" /></figure><h3><strong>Forecasting Models:</strong></h3><p>Basically, the normal machine learning algorithm cannot fit into this process, Hereafter, we are not going to predict the values. But we are going to forecast the values. So, there are algorithms, especially for the Time series.</p><p><strong>Autoregression (AR):</strong></p><p>The autoregression (AR) method models the next step in the sequence as a linear function of the observations at prior time steps. The method is suitable for univariate time series without trend and seasonal components.</p><p><strong>Moving Average(MA):</strong></p><p>A moving average also called a rolling or running average, is used to analyze the time-series data by calculating averages of different subsets of the complete dataset. Since it involves taking the average of the dataset over time, it is also called a moving mean (MM) or rolling mean. It&#39;s simply the rolling window, we explored in the past cells</p><p><strong>Autoregressive Moving Average (ARMA):</strong></p><p>The Autoregressive Moving Average (ARMA) method models the next step in the sequence as a linear function of the observations and residual errors at prior time steps. It combines both Autoregression (AR) and Moving Average (MA) models.</p><p><strong>Autoregressive Integrated Moving Average (ARIMA):</strong></p><p>The Autoregressive Integrated Moving Average (ARIMA) method models the next step in the sequence as a linear function of the differenced observations and residual errors at prior time steps. It combines both Autoregression (AR) and Moving Average (MA) models as well as a differencing pre-processing step of the sequence to make the sequence stationary, called integration (I).</p><p><strong>Seasonal Autoregressive Integrated Moving-Average (SARIMA):</strong></p><p>The Seasonal Autoregressive Integrated Moving Average (SARIMA) method models the next step in the sequence as a linear function of the differenced observations, errors, differenced seasonal observations, and seasonal errors at prior time steps. It combines the ARIMA model with the ability to perform the same autoregression, differencing, and moving average modelling at the seasonal level.</p><p><strong>Seasonal Autoregressive Integrated Moving-Average with Exogenous Regressors (SARIMAX):</strong></p><p>The Seasonal Autoregressive Integrated Moving-Average with Exogenous Regressors (SARIMAX) is an extension of the SARIMA model that also includes the modelling of exogenous variables. The SARIMAX method can also be used to model the subsumed models with exogenous variables, such as ARX, MAX, ARMAX, and ARIMAX. The method is suitable for univariate time series with trend and/or seasonal components and exogenous variables.</p><p><strong>Vector Autoregression Moving-Average (VARMA):</strong></p><p>The Vector Autoregression Moving-Average (VARMA) method models the next step in each time series using an ARMA model. It is the generalization of ARMA to multiple parallel time series, e.g. multivariate time series.The method is suitable for multivariate time series without trend and seasonal components.</p><p><strong>Vector Autoregression Moving-Average with Exogenous Regressors (VARMAX):</strong></p><p>The Vector Autoregression Moving-Average with Exogenous Regressors (VARMAX) is an extension of the VARMA model that also includes the modelling of exogenous variables. It is a multivariate version of the ARMAX method.</p><h3><strong>Conclusion:</strong></h3><p>These are the major topics to be covered in time series analysis. you can also apply deep learning in time-series analysis using LSTM models. We have just given you a glimpse of all topic in the Time Series, Still, there is more to Cover. So, Keep exploring !. If you like to extend your knowledge in time series using python. Just jump into my repo <a href="https://github.com/eswaraprasad001/Time-Series-Analysis-101"><strong><em>here</em></strong></a> and Enjoy learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3faf49b2126" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/a-roadmap-for-time-series-analysis-3faf49b2126">A Roadmap for Time-Series Analysis</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is DVC(Data Version Control) and How to get started?]]></title>
            <link>https://medium.com/featurepreneur/what-is-dvc-data-version-control-and-how-to-get-started-25bf3991e6a4?source=rss-907fb3b469c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/25bf3991e6a4</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[dvc]]></category>
            <dc:creator><![CDATA[Eswara Prasad]]></dc:creator>
            <pubDate>Tue, 26 Jan 2021 17:23:46 GMT</pubDate>
            <atom:updated>2021-02-01T01:06:19.308Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*gN7Xru3A-PTavPI6adpJPQ.png" /></figure><p>Always using a version control system such as Git for source code is a good practice and an industry standard. But tracking large data becomes impossible in Git. Projects that are data-intensive such as deep learning projects, strongly rely on the quality of the dataset to produce results . Then, what’s the solution? Is there any approach such as data versioning?</p><p>Yes , The Data Version Control (<a href="https://dvc.org/">DVC</a>) project aims at bringing Git in projects that use a lot of data. You often find in such projects a link of some sort to download the data, or sometimes <a href="https://git-lfs.github.com/">Git LFS</a> is used. It is a command-line tool written in Python. It mimics Git commands and workflows to ensure that users can quickly incorporate it into their regular Git practice. The workflow of DVC is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*63DTyqd3aXkQaK9LudvTQA.png" /></figure><h3>Getting Started with DVC:</h3><ul><li>Create a virtual env ,install dvc and clone the git repository to be versioned.</li></ul><pre>conda create --name dvc python=3.8.2 -y <br>pip install dvc</pre><ul><li>Then create your branch in Git for each version.</li></ul><pre>git checkout -b &quot;V1.0&quot;</pre><ul><li>Now you can initiate the dvc by using the below command , then you can find a .dvc folder in your repository and by default dvc stores the analytics of the model, so we can set it to false.</li></ul><pre>dvc init<br>dvc config core.analytics false</pre><ul><li>In dvc , you need to have remote storage for storing large file , It can be Googledrive , S3 bucket etc .Here, we use the local storage. PATH is the path of the remote storage folder in your computer.</li></ul><pre>dvc remote add -d remote_storage PATH</pre><ul><li>To add a folder in the tracking list of DVC ( Data folder).</li></ul><pre>dvc add PATH</pre><ul><li>Now your data is copied to the cache and a .dvc file will be created in place of the csv file. eg: Train.csv -&gt; Train.dvc . In each step after add folder to dvc ensure to add in git and commit them to github.</li></ul><pre>git add --all<br>git commit -m&quot; First Commit &quot;</pre><ul><li>Then push the cache to remote in dvc and also in github using new branch as default.</li></ul><pre>dvc push<br>git push --set-upstream origin BRANCH NAME</pre><ul><li>If you want to extract the data in another setup, Use the following commands.</li></ul><pre>dvc checkout<br>dvc fetch</pre><h3>Conclusion:</h3><p>TAADAA! , You have just versioned your data using few lines of commands , DVC is also useful in reproducible pipelines and reusability . For more details , refer <a href="https://dvc.org/">dvc.org</a>. You and your teammates will be happy to save the changes in your data in Git with simple command lines like above while storing your data in your favorite storage service!. Check out my repo on dvc <a href="https://github.com/eswaraprasad001/data-version-control">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25bf3991e6a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/what-is-dvc-data-version-control-and-how-to-get-started-25bf3991e6a4">What is DVC(Data Version Control) and How to get started?</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>