<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by duy ngọc on Medium]]></title>
        <description><![CDATA[Stories by duy ngọc on Medium]]></description>
        <link>https://medium.com/@bluemind1988?source=rss-55170f58a88f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*3NsJ4_v-f5MGdMCU.jpg</url>
            <title>Stories by duy ngọc on Medium</title>
            <link>https://medium.com/@bluemind1988?source=rss-55170f58a88f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:35:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bluemind1988/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Explore Python Libraries for End to End Data Science Project]]></title>
            <link>https://medium.com/geekculture/explore-python-library-for-end-to-end-data-science-project-8068290be097?source=rss-55170f58a88f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8068290be097</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[duy ngọc]]></dc:creator>
            <pubDate>Mon, 03 May 2021 11:16:26 GMT</pubDate>
            <atom:updated>2021-05-07T08:39:15.833Z</atom:updated>
            <content:encoded><![CDATA[<p>[ You can see my another post about Explore R libraries for an end to end data science projects in this <a href="https://bluemind1988.medium.com/explore-r-libraries-for-end-to-end-data-science-projects-b4d0af3a9f5c">link</a>]</p><p><strong>Important note</strong>: this post is just a collection of most popular python libraries for data science I’ve known before, some other important libraries are not in this list therefore please leave me your feedback for improvement. Almost all information related to these libraries I quote directly from their website (<strong>with link/source</strong>) to keep the objectivity. For my personal opinions I also have a specific note. Happy reading!</p><p>Python is a very powerful language for Data science, especially in Machine Learning / Deep Learning. Python<em> can cover full stack (end to end) data science project with some important steps below:</em></p><p>Data preparation→ Data visualization → Feature Engineering→ Build &amp; validate ML model → Explain model → Communicate results → Deployment (web app)</p><p>Today we will explore some useful Python libraries that can be used for full stack data science purpose :</p><ol><li><strong>Data preparation (get and clean data)</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*WvU50h-GeC8AnC5ctyHzXg.png" /></figure><p><strong>For data fit in memory :</strong></p><ul><li>Pandas (<a href="https://pandas.pydata.org/">https://pandas.pydata.org/</a>)</li></ul><p><strong>pandas</strong> is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool (<a href="https://pandas.pydata.org/">link</a>) . It can be used in different file formats: csv, text file, SQL databases, excel, HDF5…</p><pre>Example from this <a href="https://pandas.pydata.org/docs/user_guide/10min.html"><strong>link</strong></a>:</pre><pre>import pandas as pd<br># Object creation:<br>s = pd.Series([1, 3, 5, np.nan, 6, 8])<br>dates = pd.date_range(&quot;20130101&quot;, periods=6)<br># Viewing data:<br>df.head(5) ; df.tail(3); df.index; df.columns; df.describe()<br># Selection:<br>df[&quot;A&quot;], df[0:3] # select by column<br>df.loc[&quot;20130102&quot;, [&quot;A&quot;, &quot;B&quot;]] # select by label<br>df.iloc[3], df.iloc[3:5,0:2] # select by position<br>df[df[&quot;A&quot;] &gt; 0], <strong> </strong>df2[df2[&quot;E&quot;].isin([&quot;two&quot;, &quot;four&quot;])] # Boolean indexing<br># Missing data:<br>df1.dropna(how=&quot;any&quot;) # drop any rows that have missing data<br>df1.fillna(value=5) # filling missing data</pre><p>My opinions :<br>Pros: less coding , fast learning, easy customization<br>Cons: syntax is not easy to understand compared to tidyverse (R), difficult to deal with big data</p><p><strong>For data larger-than-memory or distributed environments (big data) :</strong></p><ul><li>Data table (<a href="https://datatable.readthedocs.io/en/latest/">https://datatable.readthedocs.io/en/latest/</a>)</li></ul><p><strong>Datatable</strong> is a python library for manipulating tabular data. It supports out-of-memory datasets, multi-threaded data processing, and flexible API.</p><pre>Example from this <a href="https://datatable.readthedocs.io/en/latest/start/quick-start.html">link</a></pre><pre><strong>import</strong> datatable <strong>as</strong> dt<br># Loading data:<br>DT4 = dt.fread(&quot;~/Downloads/dataset_01.csv&quot;) <br>DT5 = dt.open(&quot;data.jay&quot;)<br># Basic Frame Properties<br><strong>print</strong>(DT.shape)   <em># (nrows, ncols)</em> <strong>print</strong>(DT.names)   <em># column names</em> <strong>print</strong>(DT.stypes)  <em># column types<br># </em>Select Subsets of Rows/Columns<br>DT[:, &quot;A&quot;]         <em># select 1 column</em> <br>DT[:10, :]         <em># first 10 rows</em> <br>DT[::-1, &quot;A&quot;:&quot;D&quot;]  <em># reverse rows order, columns from A to D</em> <br>DT[27, 3]          <em># single element in row 27, column 3 (0-based)</em></pre><ul><li>Dask (<a href="https://dask.org/">https://dask.org/</a>)</li></ul><p><a href="https://docs.dask.org/en/latest/"><em>Definition</em></a><em> [</em><a href="https://docs.dask.org/en/latest/"><em>Link</em></a><em>]: Dask is a flexible library for parallel computing in Python. </em>Dask is composed of two parts:</p><ol><li><strong>Dynamic task scheduling</strong> optimized for computation. This is similar to <em>Airflow, Luigi, Celery, or Make</em>, but optimized for interactive computational workloads.</li><li><strong>“Big Data” collections</strong> like parallel arrays, dataframes, and lists that extend common interfaces like <em>NumPy, Pandas, or Python iterators</em> to larger-than-memory or distributed environments. These parallel collections run on top of dynamic task schedulers.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*1QtqzWLJpI5HcQqmVhgDEA.png" /></figure><p>(Picture from <a href="https://docs.dask.org/en/latest/">https://docs.dask.org/en/latest/</a>)</p><p><em>Design (</em><a href="https://docs.dask.org/en/latest/dataframe.html">https://docs.dask.org/en/latest/dataframe.html</a><em>) :</em></p><p>Dask DataFrames coordinate many Pandas DataFrames/Series arranged along the index. A Dask DataFrame is partitioned <em>row-wise</em>, grouping rows by index value for efficiency. These Pandas objects may live on disk or on other machines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*-LqITxR8Mp1takCqbwzAcg.png" /></figure><p>Dask DataFrame copies Pandas API because the dask.dataframe application programming interface (API) is a subset of the Pandas API, <strong>it should be familiar to Pandas users.</strong> There are some slight alterations due to the parallel nature of Dask:</p><pre>from dask.distributed import Client<br>client = Client(n_workers=4)<br>import dask<br>import dask.dataframe as dd</pre><pre>df = dd.read_csv(&#39;2015-*-*.csv&#39;)<br>df2 = df[df.y == &#39;a&#39;].x + 1</pre><pre># <strong>As with all Dask collections, one triggers computation by calling the </strong><strong>.compute() method:</strong></pre><pre>df.groupby(df.user_id).value.mean().compute()<br>df2.compute()<br>df_filter=df[df[2]==&#39;SU3037_0112&#39;].compute()<br>df_filter.describe()</pre><ul><li>Spark (<a href="https://spark.apache.org/">https://spark.apache.org/</a>)</li></ul><p>Definition (<a href="https://spark.apache.org/docs/latest/">link</a>) : Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html">Spark SQL</a> for SQL and structured data processing, <a href="https://spark.apache.org/docs/latest/ml-guide.html">MLlib</a> for machine learning, <a href="https://spark.apache.org/docs/latest/graphx-programming-guide.html">GraphX</a> for graph processing, and <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html">Structured Streaming</a> for incremental computation and stream processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*kkiaO2sKr7WaIFovgbja5A.png" /></figure><p>(Picture from <a href="https://databricks.com/spark/about">databricks.com/spark/about</a>)</p><pre>Run PySpark in Google Colab:<br>!apt-get update<br>!apt-get install openjdk-8-jdk-headless -qq &gt; /dev/null<br>!wget -q <a href="https://downloads.apache.org/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz">https://downloads.apache.org/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz</a><br>!tar -xvf spark-3.0.0-bin-hadoop3.2.tgz<br>!pip install -q findspark</pre><pre>import os<br>os.environ[&quot;JAVA_HOME&quot;] = &quot;/usr/lib/jvm/java-8-openjdk-amd64&quot;<br>os.environ[&quot;SPARK_HOME&quot;] = &quot;/content/spark-3.0.0-bin-hadoop3.2&quot;</pre><pre>import findspark<br>findspark.init()<br>import spark<br>from pyspark.sql import SparkSession<br>spark = SparkSession.builder.master(&quot;local[*]&quot;).getOrCreate()</pre><pre>path=&#39;Link to your data file&#39;<br>df = spark.read.option(&quot;delimiter&quot;, &quot;\t&quot;).csv(path)<br>df.show(5)<br>print((df.count(), len(df.columns)))<br>product_type=&#39;_c2&#39;<br>df.groupBy(product_type).count().show()<br>product=&#39;SU3037_0112&#39; <br>df_product=df.filter(df._c2 == product)<br>df_product.describe().show()</pre><p><strong>2. Data Visualization</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*9OtFqS1efNvkTjec3_5AOA.png" /></figure><p><strong>Not Interactive chart:</strong></p><ul><li>Matplotlib (<a href="https://matplotlib.org/">https://matplotlib.org/</a>)</li></ul><p>Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python. From my point of view: this is very complex library (lot of customization) for data visualization compared to ggplot (R) or Seaborn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*rfR5wfrErSSuHeDepXKIsQ.png" /></figure><p>(Picture from <a href="https://matplotlib.org/stable/tutorials/index.html">https://matplotlib.org/stable/tutorials/index.html</a>)</p><ul><li>Seaborn (<a href="https://seaborn.pydata.org/">https://seaborn.pydata.org/</a>)</li></ul><p>Seaborn is a Python data visualization library based on <a href="https://matplotlib.org/">matplotlib</a>. It provides a high-level interface for drawing attractive and informative statistical graphics. [My opinion] : with seaborn you can create beautiful plot with less code than matplotlib</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*3M31QIvSF_P_kY6cyppSsw.png" /></figure><p>(Picture from <a href="https://seaborn.pydata.org/index.html">https://seaborn.pydata.org/index.html</a>)</p><ul><li>Plotnine (<a href="https://plotnine.readthedocs.io/en/stable/">https://plotnine.readthedocs.io/en/stable/</a>)</li></ul><p>plotnine is an implementation of a <em>grammar of graphics</em> in Python, it is based on <a href="https://ggplot2.tidyverse.org/">ggplot2</a> . The grammar allows users to compose plots by explicitly mapping data to the visual objects that make up the plot.Plotting with a grammar is powerful, it makes custom (and otherwise complex) plots easy to think about and then create, while the simple plots remain simple.</p><p>*** ggplot2 : the very best data visualization tool in R [author]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*wkC2ri5Njm38NMQou8uimw.png" /></figure><p>(Picture from <a href="https://plotnine.readthedocs.io/en/stable/gallery.html">https://plotnine.readthedocs.io/en/stable/gallery.html</a>)</p><p><strong>Interactive chart:</strong></p><ul><li>Plotly (<a href="https://plotly.com/python/">https://plotly.com/python/</a>)</li></ul><p>Plotly’s Python graphing library makes <strong>interactive</strong>, publication-quality graphs. Examples of how to make line plots, scatter plots, area charts, bar charts, error bars, box plots, histograms, heatmaps, ubplots, multiple-axes, polar charts, and bubble charts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*WJkvpRvT2hxVW17Gw-S6Qg.png" /></figure><p>(Picture from <a href="https://plotly.com/python/">https://plotly.com/python/</a>)</p><ul><li>Bokeh (<a href="https://docs.bokeh.org/en/latest/index.html#">https://docs.bokeh.org/en/latest/index.html#</a>)</li></ul><p>Bokeh is a Python library for creating <strong>interactive</strong> visualizations for modern web browsers. It helps you build beautiful graphics, ranging from simple plots to complex dashboards with streaming datasets. With Bokeh, you can create JavaScript-powered visualizations without writing any JavaScript yourself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*T8NEMUgxf79ZA9sb2IqKeQ.png" /></figure><p>(Picture from <a href="https://docs.bokeh.org/en/latest/docs/gallery.html">https://docs.bokeh.org/en/latest/docs/gallery.html</a>)</p><ul><li>Folium (<a href="https://python-visualization.github.io/folium/">https://python-visualization.github.io/folium/</a>)</li></ul><p>folium builds on the data wrangling strengths of the Python ecosystem and the mapping strengths of the leaflet.js library. Manipulate your data in Python, then visualize it in on a Leaflet map via folium.</p><p>The library has a number of built-in tilesets from OpenStreetMap, Mapbox, and Stamen, and supports custom tilesets with Mapbox or Cloudmade API keys. folium supports both Image, Video, GeoJSON and TopoJSON overlays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*Yf_5co6WImkw1qxg5O-mew.png" /></figure><p><strong>3. Feature engineering</strong></p><ul><li>Scikit-learn (<a href="https://scikit-learn.org/stable/index.html">https://scikit-learn.org/stable/index.html</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*zVrze82lknJ2jYg8f1CVnA.png" /></figure><p>Scikit-learn is an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection and evaluation, and many other utilities. We will discuss about <strong>data reprocessing</strong> for feature engineer from scikit learn:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*Sy0mXbjibIPDBFVqKK4gBQ.png" /></figure><p>(Picture from <a href="https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing">https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing</a>)</p><p>From picture above, we can use lot of popular method for feature engineer in scikit learn: one hot encoder, label encoder, scale, power_transform…However from my point of view, the <a href="https://recipes.tidymodels.org/"><strong>recipe</strong></a> package (R) is easy to use with lot of pre-build function than scikit learn . You should try recipe package to learn more. Picture below : number of reprocessing data included in the recipe package</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*GLjfTlsDfILGmJQtriCGSA.png" /></figure><p>(Picture from: <a href="https://recipes.tidymodels.org/articles/Simple_Example.html">https://recipes.tidymodels.org/articles/Simple_Example.html</a>)</p><ul><li>Featuretools (<a href="https://www.featuretools.com/">https://www.featuretools.com/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*JXdeLbYYRkVCHMLF6B88gQ.png" /></figure><p>Featuretools is an open source python framework for <strong>automated feature engineering</strong>. It excels at transforming temporal and relational datasets into feature matrices for machine learning.</p><p>Please see demo of featuretools in this <a href="https://www.featuretools.com/demos/">link</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rp-nUR3mNmqPOiua0QUwBw.png" /></figure><p>(Picture from <a href="https://www.featuretools.com/demos/">https://www.featuretools.com/demos/</a>)</p><p>Feature tool workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*LHGpi6wV12XloGI1mf02DQ.png" /></figure><p>(Picture from <a href="https://github.com/Featuretools/predict-customer-churn/blob/main/churn/5.%20Modeling.ipynb">https://github.com/Featuretools/predict-customer-churn/blob/main/churn/5.%20Modeling.ipynb</a>)</p><p><strong>4. Build &amp; Validate machine learning model</strong></p><p><strong>Manual set up &amp; tune machine learning:</strong></p><ul><li>Scikit-learn (<a href="https://scikit-learn.org/stable/">https://scikit-learn.org/stable/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*zVrze82lknJ2jYg8f1CVnA.png" /></figure><p>Scikit-learn is an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection and evaluation, and many other utilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1d7DGTACMwa81os6ajylpw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1IBUPMoyFYNpFsYXu0panw.png" /></figure><p>(Picture from <a href="https://scikit-learn.org/stable/index.html">https://scikit-learn.org/stable/index.html</a>)</p><p>Tutorial: <a href="https://scikit-learn.org/stable/auto_examples/index.html">https://scikit-learn.org/stable/auto_examples/index.html</a></p><p>Scikit -learn wrapper for Keras: <a href="https://github.com/adriangb/scikeras">https://github.com/adriangb/scikeras</a></p><p>Scikit -learn wrapper for Pytorch: <a href="https://github.com/skorch-dev/skorch/tree/2734b00a7e197bc79d0ccac8444b82390638ba12">https://skorch.readthedocs.io/en/stable/</a></p><ul><li>H2O (<a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/welcome.html">https://docs.h2o.ai/h2o/latest-stable/h2o-docs/welcome.html</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/158/1*LHjAxf5XWqhZs5LeRUMbzQ.png" /></figure><p>[Definition <a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/welcome.html">Link</a>] H2O is an open source, in-memory, distributed, fast, and scalable machine learning and predictive analytics platform that allows you to build machine learning models on big data and provides easy productionalization of those models in an enterprise environment.</p><p>H2O’s core code is written in Java. Inside H2O, a Distributed Key/Value store is used to access and reference data, models, objects, etc., across all nodes and machines. The algorithms are implemented on top of H2O’s distributed Map/Reduce framework and utilize the Java Fork/Join framework for multi-threading. H2O’s REST API allows access to all the capabilities of H2O from an external program or script via JSON over HTTP. The Rest API is used by H2O’s web interface (Flow UI),<strong> R binding (H2O-R), and Python binding (H2O-Python)</strong>.</p><p>The speed, quality, ease-of-use, and model-deployment for the various cutting edge Supervised and Unsupervised algorithms like Deep Learning, Tree Ensembles, and GLRM make H2O a highly sought after API for big data data science.</p><p>Please see H2O user guide for Python and R in this <a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/welcome.html">link</a></p><ul><li>Keras (<a href="https://keras.io/">https://keras.io/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*y2s8lODJzqAhhAOZzclCdg.png" /></figure><p>[Definition <a href="https://keras.io/">link</a>] Keras is an API designed for human beings, not machines. Keras follows best practices for reducing cognitive load: it offers consistent &amp; simple APIs, it minimizes the number of user actions required for common use cases, and it provides clear &amp; actionable error messages. It also has extensive documentation and developer guides.</p><p>Keras tutorial : <a href="https://keras.io/examples/">https://keras.io/examples/</a> (Audio, video, text, image, structure data, time series, RL, GAN…)</p><p>Keras tutorial for R user (R interface to Keras): <a href="https://keras.rstudio.com/articles/examples/index.html">https://keras.rstudio.com/articles/examples/index.html</a></p><ul><li>Tensorflow (<a href="https://www.tensorflow.org/">https://www.tensorflow.org/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/203/1*8KRdyBYkq4kBuJtVTW1N9A.png" /></figure><p><strong>[</strong><a href="https://en.wikipedia.org/wiki/TensorFlow"><strong>Wiki</strong></a><strong>] TensorFlow</strong> is a <a href="https://en.wikipedia.org/wiki/Free_and_open-source_software">free and open-source</a> <a href="https://en.wikipedia.org/wiki/Library_(computing)">software library</a> for <a href="https://en.wikipedia.org/wiki/Machine_learning">machine learning</a>. It can be used across a range of tasks but has a particular focus on <a href="https://en.wikipedia.org/wiki/Types_of_artificial_neural_networks#Training">training</a> and <a href="https://en.wikipedia.org/wiki/Statistical_inference">inference</a> of <a href="https://en.wikipedia.org/wiki/Deep_neural_networks">deep neural networks</a>. TensorFlow was developed by the <a href="https://en.wikipedia.org/wiki/Google_Brain">Google Brain</a> team for internal <a href="https://en.wikipedia.org/wiki/Google">Google</a> use. It was released under the <a href="https://en.wikipedia.org/wiki/Apache_License_2.0">Apache License 2.0</a> in 2015</p><p>Tensorflow is a very best framework for deep learning application (NLP, Computer vision, Reinforcement learning….).</p><p><em>Tensorflow </em>: Please see this tutorial (<a href="https://www.tensorflow.org/tutorials">https://www.tensorflow.org/tutorials</a>) for Tensorflow application example (images, text, audio, structured data, Generative, Interpretability, Reinforcement learning….)</p><p><em>Tensorflow for R users </em>: <a href="https://tensorflow.rstudio.com/tutorials/">https://tensorflow.rstudio.com/tutorials/</a></p><ul><li>Pytorch (<a href="https://pytorch.org/">https://pytorch.org/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*AS4YHiJNudUwtgvuCnuqug.png" /></figure><p><strong>[W</strong><a href="https://en.wikipedia.org/wiki/PyTorch"><strong>iki</strong></a><strong>] PyTorch</strong> is an <a href="https://en.wikipedia.org/wiki/Open_source">open source</a> <a href="https://en.wikipedia.org/wiki/Machine_learning">machine learning</a> <a href="https://en.wikipedia.org/wiki/Library_(computing)">library</a> based on the <a href="https://en.wikipedia.org/wiki/Torch_(machine_learning)">Torch</a> library,used for applications such as <a href="https://en.wikipedia.org/wiki/Computer_vision">computer vision</a> and <a href="https://en.wikipedia.org/wiki/Natural_language_processing">natural language processing</a>,primarily developed by <a href="https://en.wikipedia.org/wiki/Facebook">Facebook</a>’s AI Research lab (FAIR).It is <a href="https://en.wikipedia.org/wiki/Free_and_open-source_software">free and open-source software</a> released under the <a href="https://en.wikipedia.org/wiki/Modified_BSD_License">Modified BSD license</a>. Although the <a href="https://en.wikipedia.org/wiki/Python_(programming_language)">Python</a> interface is more polished and the primary focus of development, PyTorch also has a <a href="https://en.wikipedia.org/wiki/C%2B%2B">C++</a> interface</p><p>PyTorch tutorial : <a href="https://pytorch.org/tutorials/">https://pytorch.org/tutorials/</a> (image, video, audio, text…)</p><p>rTorch tutorial (for R users) : <a href="https://f0nzie.github.io/rTorch/">https://f0nzie.github.io/rTorch/</a></p><ul><li>Dask for <strong>scalable</strong> Machine learning (<a href="https://ml.dask.org/">https://ml.dask.org/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/199/1*2tkmixVPTRlp8C1v7OCO4w.png" /></figure><p>Dask-ML provides scalable machine learning in Python using <a href="https://dask.org/">Dask</a> alongside popular machine learning libraries like <a href="http://scikit-learn.org/">Scikit-Learn</a>, <a href="https://ml.dask.org/xgboost.html">XGBoost</a>, and others.</p><p>Tutorial : <a href="https://ml.dask.org/">https://ml.dask.org/</a></p><ul><li>Spark for <strong>scalable </strong>Machine learning (<a href="https://spark.apache.org/docs/1.2.1/mllib-guide.html">https://spark.apache.org/docs/1.2.1/mllib-guide.html</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*e3IO4dAeQ4s1L1xoYE0EVQ.png" /></figure><p>[<a href="https://spark.apache.org/docs/1.2.1/mllib-guide.html">Link</a>] MLlib is Spark’s<strong> </strong>scalable machine learning<strong> </strong>library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives</p><p>Tutorial: <a href="https://spark.apache.org/docs/1.2.1/mllib-guide.html">https://spark.apache.org/docs/1.2.1/mllib-guide.html</a> (available for Scala, Python, Java)</p><p><strong>Automated set up &amp; tune machine learning</strong>:</p><ul><li>Pycaret (<a href="https://pycaret.org/">https://pycaret.org/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/1*fCJh_Fbl4-S4VkZ4BupgcA.png" /></figure><p>[<a href="https://pycaret.org/">Link</a>] PyCaret is an open source, <strong>low-code</strong> machine learning library in Python that allows you to go from preparing your data to deploying your model within minutes in your choice of notebook environment.</p><p>Please see Pycaret tutorial in this <a href="https://github.com/pycaret/pycaret/tree/master/tutorials">link</a> : (Anomaly detection, binary classification, Clustering, Multi-class Classification, NLP, Regression ….)</p><ul><li>Auto-sklearn (<a href="https://automl.github.io/auto-sklearn/master/">https://automl.github.io/auto-sklearn/master/</a>)</li></ul><p><em>[</em><a href="https://automl.github.io/auto-sklearn/master/"><em>Link</em></a><em>] auto-sklearn</em> is an automated machine learning toolkit and a drop-in replacement for a scikit-learn estimator:</p><pre><strong>import</strong> <strong>autosklearn.classification</strong><br>cls = autosklearn.classification.AutoSklearnClassifier()<br>cls.fit(X_train, y_train)<br>predictions = cls.predict(X_test)</pre><p>Tutorial: <a href="https://automl.github.io/auto-sklearn/master/examples/index.html">link</a></p><ul><li>EvalML (<a href="https://evalml.alteryx.com/en/stable/">https://evalml.alteryx.com/en/stable/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*AUdCPRIUM3ahQVwYynYL2g.png" /></figure><p>[<a href="https://evalml.alteryx.com/en/stable/">Link</a>] EvalML is an AutoML library that builds, optimizes, and evaluates machine learning pipelines using domain-specific objective functions.</p><p>Combined with <a href="https://featuretools.featurelabs.com/">Featuretools</a> and <a href="https://compose.featurelabs.com/">Compose</a>, EvalML can be used to create end-to-end supervised machine learning solutions.</p><p>Tutorial: <a href="https://evalml.alteryx.com/en/stable/tutorials.html">Link</a></p><ul><li>TPOT (<a href="http://epistasislab.github.io/tpot/">http://epistasislab.github.io/tpot/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*44ruo9WW6vEm5Jgt1VTyGg.png" /></figure><p>[<a href="http://epistasislab.github.io/tpot/">Link</a>] Consider TPOT your <strong>Data Science Assistant</strong>. TPOT is a Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.</p><p>Tutorial: <a href="http://epistasislab.github.io/tpot/examples/">Link</a></p><ul><li>AutoKeras (<a href="https://autokeras.com/">https://autokeras.com/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*kOcMz9TOSircuHlTMZavxQ.png" /></figure><p>[<a href="https://autokeras.com/">Link</a>] AutoKeras: An AutoML system based on Keras. It is developed by <a href="http://faculty.cs.tamu.edu/xiahu/index.html">DATA Lab</a> at Texas A&amp;M University. The goal of AutoKeras is to make machine learning accessible to everyone.</p><pre>import autokeras as ak  <br>clf = ak.ImageClassifier() <br>clf.fit(x_train, y_train) <br>results = clf.predict(x_test)</pre><p>Tutorial: <a href="https://autokeras.com/tutorial/overview/">Link</a></p><ul><li>H2O AutoML (<a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html">https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*Y7vbA7Y3tCnY8NuvQRUY7w.png" /></figure><p>[<a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html">Link</a>] H2O’s AutoML can be used for automating the machine learning workflow, which includes automatic training and tuning of many models within a user-specified time-limit. <a href="http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/stacked-ensembles.html">Stacked Ensembles</a> — one based on all previously trained models, another one on the best model of each family — will be automatically trained on collections of individual models to produce highly predictive ensemble models which, in most cases, will be the top performing models in the AutoML Leaderboard.</p><p>H2O offers a number of <a href="http://docs.h2o.ai/h2o/latest-stable/h2o-docs/explain.html">model explainability</a> methods that apply to AutoML objects (groups of models), as well as individual models (e.g. leader model). Explanations can be generated automatically with a single function call, providing a simple interface to exploring and explaining the AutoML models.</p><p>Tutorial in Python and R: <a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html#code-examples">Link</a></p><pre>Example from: <a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html#code-examples">https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html#code-examples</a></pre><pre><strong>import</strong> h2o<br><strong>from</strong> h2o.automl <strong>import</strong> H2OAutoML<br><br>h2o<strong>.</strong>init()<br><br><em># Import a sample binary outcome train/test set into H2O</em><br>train <strong>=</strong> h2o<strong>.</strong>import_file(&quot;https://s3.amazonaws.com/erin-data/higgs/higgs_train_10k.csv&quot;)<br>test <strong>=</strong> h2o<strong>.</strong>import_file(&quot;https://s3.amazonaws.com/erin-data/higgs/higgs_test_5k.csv&quot;)<br><br><em># Identify predictors and response</em><br>x <strong>=</strong> train<strong>.</strong>columns<br>y <strong>=</strong> &quot;response&quot;<br>x<strong>.</strong>remove(y)<br><br><em># For binary classification, response should be a factor</em><br>train[y] <strong>=</strong> train[y]<strong>.</strong>asfactor()<br>test[y] <strong>=</strong> test[y]<strong>.</strong>asfactor()<br><br><em># Run AutoML for 20 base models (limited to 1 hour max runtime by default)</em><br>aml <strong>=</strong> H2OAutoML(max_models<strong>=</strong>20, seed<strong>=</strong>1)<br>aml<strong>.</strong>train(x<strong>=</strong>x, y<strong>=</strong>y, training_frame<strong>=</strong>train)<br><br><em># View the AutoML Leaderboard</em><br>lb <strong>=</strong> aml<strong>.</strong>leaderboard<br>lb<strong>.</strong>head(rows<strong>=</strong>lb<strong>.</strong>nrows)</pre><ul><li>AutoGluon (<a href="https://auto.gluon.ai/stable/index.html">https://auto.gluon.ai/stable/index.html</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/237/1*_5xV6IHfLr7rMn3geTVmUg.png" /></figure><p>[<a href="https://auto.gluon.ai/stable/index.html">Link</a>] AutoGluon enables easy-to-use and easy-to-extend AutoML with a focus on automated stack ensembling, deep learning, and real-world applications spanning text, image, and tabular data. Intended for both ML beginners and experts</p><p>Tutorial : <a href="https://auto.gluon.ai/stable/tutorials/index.html">Link</a></p><ul><li>GluonTS-Probabilistic Time Series Modeling (<a href="https://ts.gluon.ai/">https://ts.gluon.ai/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/214/1*tIp13GXjSZX5uggxU4puvQ.png" /></figure><p>[<a href="https://ts.gluon.ai/">Link</a>] Gluon Time Series (GluonTS) is the Gluon toolkit for probabilistic time series modeling, focusing on deep learning-based models.</p><p>GluonTS provides utilities for loading and iterating over time series datasets, state of the art models ready to be trained, and building blocks to define your own models and quickly experiment with different solutions.</p><p>Tutorial: <a href="https://ts.gluon.ai/tutorials/index.html">Link</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*UbU9UC54Nh21Aq6p-aYWKg.png" /></figure><p>(Picture from this link : <a href="https://github.com/awslabs/gluon-ts/">https://github.com/awslabs/gluon-ts/</a>)</p><p><strong>5. Explain model</strong></p><ul><li><a href="https://dalex.drwhy.ai/">Dalex</a> package (please refer very best book for this wonderful package in this <a href="https://ema.drwhy.ai/">link</a>)</li></ul><p>Explaining machine learning model is very important step because it will help us understanding why our model predict the result. With Dalex package, the machine learning is not a black box anymore, we can find which important features affect our prediction and make recommendation or explanation to our stakeholder / customer.</p><p>According to Dalex package, we have two explain levels: instance level (one sample) and dataset level (whole samples). Instance level includes break down plots, shapley additive explanations, Lime, ceteris-paribus profiles, ceteris-paribus oscillations, local-diagnostics plots. Dataset level includes model performance, variable importance, partial dependence profiles, local dependence and accumulated local profiles, residual-diagnostic plots</p><p>Overview Dalex package (picture link from <a href="https://ema.drwhy.ai/introduction.html">https://ema.drwhy.ai/introduction.html</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/0*1IliCQlfScNVb93j.png" /></figure><p>Instance level explanation (picture link: <a href="https://ema.drwhy.ai/summaryInstanceLevel.html">https://ema.drwhy.ai/summaryInstanceLevel.html</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/0*tB5KRck3CDLkAVJ_.png" /></figure><p>Dataset level explanation (picture link: <a href="https://ema.drwhy.ai/summaryModelLevel.html">https://ema.drwhy.ai/summaryModelLevel.html</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*irZveRaa-NOC-pt9.png" /></figure><p><strong>6. Communicate results</strong></p><ul><li>Jupiter Lab (<a href="https://jupyter.org/">https://jupyter.org/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*mbbzMW3e2kzwmN7GrCjWvw.png" /></figure><p>(Picture from this link: <a href="https://jupyter.org/">https://jupyter.org/</a>)</p><p>[<a href="https://jupyter.org/">Link</a>] JupyterLab is a web-based interactive development environment for Jupyter notebooks, code, and data. JupyterLab is flexible: configure and arrange the user interface to support a wide range of workflows in data science, scientific computing, and machine learning. JupyterLab is extensible and modular: write plugins that add new components and integrate with existing ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*Lo0JXTYCnK4MrhfLtTDX5A.png" /></figure><p>(Picture from this link: <a href="https://jupyter.org/">https://jupyter.org/</a>)</p><ul><li>nbviewer (<a href="https://nbviewer.jupyter.org/">https://nbviewer.jupyter.org/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*K9ndGq830A77kSz70Jrf_A.png" /></figure><p>[<a href="http://Jupyter NBViewer is the web application behind The Jupyter Notebook Viewer, which is graciously hosted by OVHcloud.  Run this locally to get most of the features of nbviewer on your own network.  If you need help using or installing Jupyter Notebook Viewer, please use the jupyter/help issue tracker. If you would like to propose an enhancement to nbviewer or file a bug report, please open an issue here, in the jupyter/nbviewer project.">Link</a>] Jupyter NBViewer is the web application behind <a href="http://nbviewer.jupyter.org/">The Jupyter Notebook Viewer</a>, which is graciously hosted by <a href="https://ovhcloud.com/">OVHcloud</a>.</p><p>Please note: with simple notebook you can share with github link with your code, however, with large notebooks and complex javascript libraries, this nbviewer is a better choice</p><ul><li>Voilà (<a href="https://github.com/voila-dashboards/voila">https://github.com/voila-dashboards/voila</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*64P0tP6WwQQUvb71WppxLw.png" /></figure><p>[<a href="https://github.com/voila-dashboards/voila">Link</a>] Voilà turns Jupyter notebooks into standalone web applications. Unlike the usual HTML-converted notebooks, each user connecting to the Voilà tornado application gets a dedicated Jupyter kernel which can execute the callbacks to changes in Jupyter interactive widgets.</p><p>See voila gallery in this link : <a href="https://voila-gallery.org/">https://voila-gallery.org/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*hQcEYCn7T25CgL2kzdmaaQ.png" /></figure><p>(Picture from <a href="https://voila-gallery.org/">https://voila-gallery.org/</a>)</p><p>Voila tutorial: <a href="https://github.com/voila-dashboards/voila/tree/master/notebooks">Link</a></p><p><strong>7. Deployment (web app)</strong></p><p>Complex level increase: Streamlit → Dash → Flask</p><ul><li>Streamlit (<a href="https://streamlit.io/">https://streamlit.io/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/1*T0IYjk0F9zSS0bPc4cPdgw.png" /></figure><p>Streamlit turns data scripts into shareable web apps in minutes.<br><strong>All in Python. All for free. No front‑end experience required.</strong></p><p>[<a href="https://docs.streamlit.io/en/stable/">Link</a>] <a href="https://streamlit.io/">Streamlit</a> is an open-source Python library that makes it easy to create and share beautiful, custom web apps for machine learning and data science. In just a few minutes you can build and deploy powerful data apps</p><p>My point of view: although with streamlit we can build web app very fast with only python language (for front end and back end) but Shiny (R) is still superior with many strong supportive features for web app development. The main drawback of Shiny is that you should learn R :-)</p><p>Tutorial: <a href="https://docs.streamlit.io/en/stable/getting_started.html">Link</a></p><p>Streamlit Galery: <a href="https://streamlit.io/gallery">Link</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*uzctXmlK-c-24Z_0WTNUkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*xwMU_IGbtEN8RHHM1mlVww.png" /></figure><p>(Picture form <a href="https://streamlit.io/gallery">https://streamlit.io/gallery</a>)</p><ul><li>Dash (<a href="https://plotly.com/dash/">https://plotly.com/dash/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*3Nyetcv8Actvq2IHAAETug.png" /></figure><p>[<a href="https://plotly.com/dash/">Link</a>] <strong>Dash apps give a point-&amp;-click interface to models written in Python, R, and Julia — vastly expanding the notion of what’s possible in a traditional “dashboard.”</strong> With Dash apps, data scientists and engineers put complex Python analytics in the hands of business decision makers and operators.</p><p>Python has taken over the world, and Dash Enterprise is the leading vehicle for delivering Python analytics to business users.Traditional BI dashboards no longer cut it in today’s AI and ML driven world. Production-grade, low-code Python apps are needed for the complex analytics of emerging industries such as autonomous vehicles, renewable energy, quantum computing, novel therapeutics, and more</p><p>My point of view: Dash is a powerful tool for web app development (mostly in pure python language) but it is more complex than streamlit. I suggest we can use streamlit for simple app and Dash for complex app</p><p>Dash tutorial: <a href="https://dash.plotly.com/">https://dash.plotly.com/</a></p><p>Dash gallery: <a href="https://dash-gallery.plotly.host/Portal/">https://dash-gallery.plotly.host/Portal/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*joCSMR49vicgSpBBd3MQUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cDYoSotpS_sFv1X6Uk-DLQ.png" /></figure><p>(Picture from <a href="https://dash-gallery.plotly.host/Portal/">https://dash-gallery.plotly.host/Portal/</a>)</p><ul><li>Flask (<a href="https://palletsprojects.com/p/flask/">https://palletsprojects.com/p/flask/</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*I7bWjX7QLNlZW8r2GLF4nA.png" /></figure><p>[Link] Flask is a lightweight <a href="https://wsgi.readthedocs.io/">WSGI</a> web application framework. It is designed to make getting started quick and easy, with the ability to scale up to complex applications. It began as a simple wrapper around <a href="https://palletsprojects.com/p/werkzeug">Werkzeug</a> and <a href="https://palletsprojects.com/p/jinja">Jinja</a> and has become one of the most popular Python web application frameworks.</p><p>Flask tutorial: <a href="https://flask.palletsprojects.com/en/1.1.x/tutorial/">https://flask.palletsprojects.com/en/1.1.x/tutorial/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8068290be097" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/explore-python-library-for-end-to-end-data-science-project-8068290be097">Explore Python Libraries for End to End Data Science Project</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Explore R libraries for end to end data science projects]]></title>
            <link>https://bluemind1988.medium.com/explore-r-libraries-for-end-to-end-data-science-projects-b4d0af3a9f5c?source=rss-55170f58a88f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4d0af3a9f5c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[duy ngọc]]></dc:creator>
            <pubDate>Tue, 20 Apr 2021 13:55:42 GMT</pubDate>
            <atom:updated>2021-05-06T23:19:49.383Z</atom:updated>
            <content:encoded><![CDATA[<p>[ You can see my another post about Explore Python libraries for end to end data science projects in this <a href="https://bluemind1988.medium.com/explore-python-library-for-end-to-end-data-science-project-8068290be097">link</a>]</p><p><strong>Important note</strong>: this post is just a collection of most popular R libraries for data science I’ve known before, some other important libraries are not in this list therefore please leave me your feedback for improvement. Almost information related to these libraries I quote directly from the their website (<strong>with link/source</strong>) to keep objectivity. For my personal opinions I also have a specific note. Happy reading !</p><p>R is a powerful language for statistic analysis, visualization, time series analysis, pre-processing data and analytic web app development . <em>It means R can cover full stack (end to end) data science project with some important steps below:</em></p><p>Data preparation→ Data visualization → Feature engineering→Build &amp; validate ML model → Explain model → Communicate results → Deployment (web app)</p><p>Today we will explore some useful R libraries that can be used for full stack data science purpose :</p><ol><li><strong>Data preparation:</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*X4Farel8WHoGahmjWHvMzg.png" /></figure><p>Tidiverse home page: <a href="https://www.tidyverse.org/">https://www.tidyverse.org/</a><br>Data.table home page: <a href="https://rdatatable.gitlab.io/data.table/">https://rdatatable.gitlab.io/data.table/</a><br>Sparklyr home page: <a href="https://spark.rstudio.com/">https://spark.rstudio.com/</a></p><p>Tidyverse and data.table are the best tools for acquiring &amp; preparing data in R. Tidyverse has user-friendly syntax (select, mutate, filter, summarise, group_by…) compared to not friendly syntax like data.table (R) or pandas (python). Howerver,<strong> data.table</strong> <strong>is very fast compared to tidyverse or pandas (python)</strong>. I often choose data.table for large csv file (best memory management and speed). For very large data set need distributed data processing across multiple computers, <strong>spark</strong> is the best choice</p><p>Update from <a href="https://dtplyr.tidyverse.org/">https://dtplyr.tidyverse.org/</a>: tidyverse also have package dtplyr (it provides a data.table backend for dplyr) . The goal of dtplyr is to allow you to write dplyr code (select, filter….) that is automatically translated to the equivalent, but usually much faster, data.table code.</p><p>Tidyverse ecosystem picture (author: Silvia Canelón, PhD. Original <a href="https://spcanelon.github.io/tour-of-the-tidyverse/?panelset16=overview9&amp;panelset17=read-data-in2&amp;panelset18=overview10&amp;panelset19=code3&amp;panelset20=overview11&amp;panelset21=view-the-data2&amp;panelset22=overview12&amp;panelset23=select2&amp;panelset24=overview13&amp;panelset25=code4&amp;panelset26=overview14&amp;panelset27=mutate4&amp;panelset28=overview15&amp;panelset29=un-tidying2&amp;panelset30=overview16&amp;panelset31=time-for-a-change%253F2#5">link</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*ckRqx4kCTUd--c9vZ1lc9Q.png" /></figure><ul><li>Csv, txt, tsv … file: readr (tidyverse), fread (data.table)</li></ul><pre>Example from <a href="https://readr.tidyverse.org/reference/read_delim.html">https://readr.tidyverse.org/reference/read_delim.html</a>:<br><strong>library</strong>(tidyverse)<br>read_csv(&quot;mtcars.csv&quot;)<br>read_tsv(&quot;a\tb\n1.0\t2.0&quot;)<br>read_delim(&quot;a|b\n1.0|2.0&quot;, delim = &quot;|&quot;)<br>starwars %&gt;% <a href="https://dplyr.tidyverse.org/reference/filter.html">filter</a>(species == &quot;Droid&quot;)<br>starwars %&gt;% <br>  <a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate</a>(name, bmi = mass / ((height / 100)  ^ 2)) %&gt;%<br>  <a href="https://dplyr.tidyverse.org/reference/select.html">select</a>(name:mass, bmi)</pre><pre>Example from this <a href="https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html">link</a>:</pre><pre><strong>library</strong>(data.table)<br>flights &lt;- fread (&quot;flights14.csv&quot;)<br>flights[origin == &quot;JFK&quot; &amp; month == 6L] # Get all the flights with “JFK” as the origin airport in the month of June.<br>flights[, <strong>c</strong>(&quot;arr_delay&quot;, &quot;dep_delay&quot;)] # select column</pre><ul><li>Excel file: readxl (tidyverse)</li></ul><pre><strong>library</strong>(readxl)<br>#import chickwts sheet<br>read_excel(&quot;madrid_temp.xlsx&quot;, sheet = &quot;chickwts&quot;)</pre><ul><li>Database (MySQL…) : dplyr (backend is <a href="https://dbplyr.tidyverse.org/">dbplyr</a> ), DBI . <strong>You can query data directly from database with dplyr (no SQL require)</strong>, the dbplyr backend will automate convert dplyr syntax to sql code as example below:</li></ul><pre>Example from this link :</pre><pre><strong>library</strong>(dplyr) <br>con &lt;- DBI::<a href="https://dbi.r-dbi.org/reference/dbConnect.html">dbConnect</a>(RSQLite::<a href="https://rsqlite.r-dbi.org/reference/SQLite.html">SQLite</a>(), path = &quot;:memory:&quot;)<br>flights &lt;- <a href="https://dplyr.tidyverse.org/reference/copy_to.html">copy_to</a>(con, nycflights13::flights)<br>flights %&gt;%<br>  <a href="https://dplyr.tidyverse.org/reference/select.html">select</a>(<a href="https://dplyr.tidyverse.org/reference/reexports.html">contains</a>(&quot;delay&quot;)) %&gt;%<br>  <a href="https://dplyr.tidyverse.org/reference/explain.html">show_query</a>() <br>#&gt; &lt;SQL&gt;<br>#&gt; SELECT `dep_delay`, `arr_delay`<br>#&gt; FROM `nycflights13::flights`</pre><pre># if databases don&#39;t live in a file, but instead live on another server:</pre><pre>con &lt;- DBI::dbConnect(RMySQL::MySQL(),    host = &quot;database.rstudio.com&quot;,   user = &quot;hadley&quot;,   password = rstudioapi::askForPassword(&quot;Database password&quot;) )</pre><ul><li>Web: rvest, RCurl</li></ul><p><a href="https://rvest.tidyverse.org/">Rvest</a> is web scrapping tool in R that is nearly the same as beautifulshop in python. The advantage of this tool is that you can apply “tidy way&quot; like %&gt;% when coding (better understanding code than python)</p><pre>Example from <a href="https://rvest.tidyverse.org/">https://rvest.tidyverse.org/</a> :</pre><pre><strong>library</strong>(rvest)<br><br># Start by reading a HTML page with read_html():<br>starwars &lt;- read_html(&quot;https://rvest.tidyverse.org/articles/starwars.html&quot;)<br>films &lt;- starwars %&gt;% html_elements(&quot;section&quot;)<br>title &lt;- films %&gt;% <br>  html_element(&quot;h2&quot;) %&gt;% <br>  html_text2()</pre><pre><strong>library</strong>(RCurl)<br>web &lt;- &quot;<a href="https://raw.githubusercontent.com/alstat/Analysis-with-Programming/master/2013/R/How%20to%20Enter%20Your%20Data/Data.dat">https://raw.githubusercontent.com/alstat/Analysis-with-Programming/master/2013/R/How%20to%20Enter%20Your%20Data/Data.dat</a>&quot;<br>x &lt;- getURL(web)<br>y &lt;- read.table(text = x, header = TRUE)</pre><p>Sparklyr: R interface for Apache Spark (<a href="https://spark.rstudio.com/">https://spark.rstudio.com/</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*rY8qRDKuaqCJ7WO3MzFwXQ.png" /></figure><p>[<a href="https://spark.apache.org/docs/latest/">Definition link</a>]Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html">Spark SQL</a> for SQL and structured data processing, <a href="https://spark.apache.org/docs/latest/ml-guide.html">MLlib</a> for machine learning, <a href="https://spark.apache.org/docs/latest/graphx-programming-guide.html">GraphX</a> for graph processing, and <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html">Structured Streaming</a> for incremental computation and stream processing.</p><p>Sparklyr [<a href="https://spark.rstudio.com/">Definition link]</a>: R interface for spark in ‘tidyverse’ way. With sparklyr you can connect to spark from R (with dplyr backend), Filter and aggregate Spark datasets then bring them into R for analysis and visualization, Use Spark’s distributed <a href="http://spark.apache.org/docs/latest/mllib-guide.html">machine learning</a> library from R,Create <a href="http://spark.rstudio.com/extensions.html">extensions</a> that call the full Spark API and provide interfaces to Spark packages</p><p>Picture from <a href="https://spark.rstudio.com/guides/data-lakes/">https://spark.rstudio.com/guides/data-lakes/</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*Y0fXyg8zkMJb2xWOwg02lQ.png" /></figure><p>Resource to learn sparklyr and big data in R:<br>- Web page: <a href="https://spark.rstudio.com/">https://spark.rstudio.com/</a><br>- Mastering Spark with R book: <a href="https://therinspark.com/">https://therinspark.com/</a><br>- Exploring, Visualizing, and Modeling Big Data with R: <a href="https://okanbulut.github.io/bigdata/">https://okanbulut.github.io/bigdata/</a></p><pre>Example from <a href="https://spark.rstudio.com/">https://spark.rstudio.com/</a>:<br># Connect to spark:<br><strong>library</strong>(sparklyr) <br>sc <strong>&lt;-</strong> <strong>spark_connect</strong>(master <strong>=</strong> &quot;local&quot;)<br># We can now use all of the available dplyr verbs against the tables within the cluster.</pre><p><strong>2. Data Visualization:</strong></p><p>You can visit this excellent website (<a href="https://www.r-graph-gallery.com/index.html">https://www.r-graph-gallery.com/index.html</a>) to discovery hundreds charts made with R programming language. These libraries below is just for reference</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*hIwI_PURJE08WHuP2OxZow.png" /></figure><ul><li>ggplot2: <a href="https://ggplot2.tidyverse.org/">https://ggplot2.tidyverse.org/</a></li></ul><p>[<a href="https://ggplot2.tidyverse.org/">Link</a>] ggplot2 is a system for declaratively creating graphics, based on <a href="https://amzn.to/2ef1eWp">The Grammar of Graphics</a>. You provide the data, tell ggplot2 how to map variables to aesthetics, what graphical primitives to use, and it takes care of the details.</p><p>[My point of view] : This is the best visualization library for chart, <strong>the only drawback is that it is not interactive chart</strong></p><p>Picture from <a href="https://www.r-graph-gallery.com/ggplot2-package.html">https://www.r-graph-gallery.com/ggplot2-package.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9dhnCRvmY-dIB8IdCqoDCA.png" /></figure><pre>Example from <a href="https://ggplot2.tidyverse.org/">https://ggplot2.tidyverse.org/</a>:</pre><pre><a href="https://rdrr.io/r/base/library.html"><strong>library</strong></a>(<a href="http://ggplot2.tidyverse.org/">ggplot2</a>)<br><br><a href="https://ggplot2.tidyverse.org/reference/ggplot.html">ggplot</a>(mpg, <a href="https://ggplot2.tidyverse.org/reference/aes.html">aes</a>(displ, hwy, colour = class)) + <br>  <a href="https://ggplot2.tidyverse.org/reference/geom_point.html">geom_point</a>()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*VYC3mX0AZ4CE_Z7pnYSgbQ.png" /></figure><ul><li>H<a href="https://www.highcharts.com/blog/tutorials/highcharts-for-r-users/">ighchart</a>: <a href="https://www.highcharts.com/blog/tutorials/highcharts-for-r-users/">https://www.highcharts.com/blog/tutorials/highcharts-for-r-users/</a></li></ul><p>[<a href="https://www.highcharts.com/blog/products/highcharts/">Definition link</a>] Highcharts is a modern SVG-based, multi-platform charting library. It makes it easy to add <strong>interactive charts </strong>to web and mobile projects.</p><p>[My point of view] It can be used with <strong>medium</strong> — <strong>large data set </strong>(very effective compared to plotly or dygraph)</p><p>Picture from <a href="https://www.highcharts.com/blog/products/highcharts/">https://www.highcharts.com/blog/products/highcharts/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YDe7hW6AeB4orlEO1YwASw.png" /></figure><pre>Example from <a href="https://www.highcharts.com/blog/tutorials/highcharts-for-r-users/">https://www.highcharts.com/blog/tutorials/highcharts-for-r-users/</a>:<br><strong>library</strong>(highcharter)<br>highchart(type = &quot;stock&quot;) %&gt;% <br>  hc_add_series(asset_returns_xts$SPY, type = &quot;line&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*sL4lgGCdgEd9mvNhDUY42g.png" /></figure><ul><li>Plotly: <a href="https://plotly.com/r/">https://plotly.com/r/</a></li></ul><p>Plotly’s R graphing library makes <strong>interactive</strong>, publication-quality graphs. [My point of view]: It can be used with <strong>small — medium data set</strong></p><p>Picture from <a href="https://plotly.com/r/">https://plotly.com/r/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*g9pwxhrRjSYDZ7YFL8mOEQ.png" /></figure><pre>Example from <a href="https://plotly.com/r/candlestick-charts/">https://plotly.com/r/candlestick-charts/</a> :<br><strong>library</strong>(plotly) <br><strong>library</strong>(quantmod)  <br>getSymbols(&quot;AAPL&quot;,src=&#39;yahoo&#39;)<br><em># basic example of ohlc charts</em> df &lt;- data.frame(Date=index(AAPL),coredata(AAPL)) df &lt;- tail(df, 30)  fig &lt;- df %&gt;% plot_ly(x = ~Date, type=&quot;candlestick&quot;,           open = ~AAPL.Open, close = ~AAPL.Close,           high = ~AAPL.High, low = ~AAPL.Low)  fig &lt;- fig %&gt;% layout(title = &quot;Basic Candlestick Chart&quot;)  fig</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*jXmy99vQ2sS8zLbRcGOQNg.png" /></figure><p><strong>3. </strong><a href="https://recipes.tidymodels.org/"><strong>Feature engineering</strong></a></p><ul><li><a href="https://recipes.tidymodels.org/"><strong>recipes</strong></a> (from tidymodels packages <a href="https://recipes.tidymodels.org/">https://recipes.tidymodels.org/</a>):</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*ZQqcRz_4Wps3mZV8B0PpUg.png" /></figure><p>The idea of the recipes package is to define a recipe or blueprint that can be used to sequentially define the encodings and preprocessing of the data (i.e. “feature engineering”).</p><p><strong>This package is extremely useful because it reduces ton of time for clean, transform &amp; add new feature to raw dataset </strong>( compared to<strong> </strong><a href="https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing">sklearn.preprocessing</a> &amp; <a href="https://scikit-learn.org/stable/modules/classes.html#module-sklearn.pipeline">sklearn.pipeline</a> in python). I hope scikit-learn team will develop something like recipes in near future to help speed up feature engineer tasks in python environment</p><p>Some examples from <a href="https://recipes.tidymodels.org/reference/index.html">https://recipes.tidymodels.org/reference/index.html</a> :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*uYMVGzlG28XIaFNclU5rpw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*KUDHVEMtlpLMRPCtPhURlg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*N8L78306cdnR__j518x_Vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*pa0zyWriA58fqbo0_ElQDA.png" /></figure><p>… (see more in the link above)</p><ul><li><a href="https://business-science.github.io/timetk/"><strong>Timetk </strong></a>(for time series feature engineering)</li></ul><p><a href="https://business-science.github.io/timetk/">https://business-science.github.io/timetk/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*fWMfN-mNYaBz7ZGKi46vQA.png" /></figure><p>Timetk is the very best tool from business science for time series feature engineering. It is atidyverse toolkit to visualize, wrangle, and transform time series data (from business-science.github.io)</p><p>The timetk has step_timeseries_signature(), which is used to add a number of features that can help machine learning models (new features such as day, week, month, quarter, interaction between day, week, month…). You can see examples in their excellent post in this <a href="https://www.business-science.io/time-series/2020/03/18/time-series-machine-learning.html">link</a></p><p>Other wonderful open source packages from business-science you can see in this <a href="https://www.business-science.io/software.html">link</a> (regarding to forecasting, timeseries, feature engineer, EDA… ). Check more information from <a href="http://www.business-science.io/">www.business-science.io</a> ( I learn a lot from this website and it helps me save time to master R skill applied to business)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*wsbgtfoKCmNcN1357iuyfA.png" /></figure><p><strong>4. Machine learning model building &amp; validation</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*QPuLA9JoOrO9-kXgkPHAiQ.png" /></figure><ul><li><a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/faq/r.html">H2O</a> (<a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/faq/r.html">https://docs.h2o.ai/h2o/latest-stable/h2o-docs/faq/r.html</a>)<br>Picture from <a href="https://www.h2o.ai/community/glossary/h2o-h2o-3">https://www.h2o.ai/community/glossary/h2o-h2o-3</a> :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X9mVVF_JS7Uh1JRPAm5x-Q.png" /></figure><p>H2O is the machine learning package that I used the most in R. The speed of H2O ML is very fast compared to traditional R machine learning packages (caret). It has multi languages interface ( R/Python), handle big data well with distributed processing, reduce time for selecting and tuning model with <strong>autoML</strong>, ready for production deployment… Moreover H2O also has full eco-system for almost data science tasks as description below:</p><pre>Example from this <a href="http://h2o-release.s3.amazonaws.com/h2o/master/1292/docs-website/tutorial/rtutorial.html">link</a> and <a href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/explain.html">link</a>:<br><strong># 1. Getting start H2O:</strong><br>library(h2o)<br>localH2O = h2o.init(ip = &quot;localhost&quot;, port = 54321, startH2O = TRUE)</pre><pre><strong># 2. Importing Data:</strong><br>h2o.importFile(), h2o.importFolder(),h2o.importURL()</pre><pre><strong># 3. Data Manipulation and Description:</strong><br>iris.h2o &lt;- as.h2o(localH2O, iris.r, key=&quot;iris.h2o&quot;) # convert dataframe to h2o frame<br>summary(iris.h2o)<br>colnames(iris.h2o)<br>....</pre><pre><strong># 4. Running Machine learning models:</strong><br>H2O offers some famous ML models: GBM, GLM, Xgboost, Lightgbm, Randomforest, K-Means, PCA ...</pre><pre>h2o.gbm(y = dependent, x = independent, data = australia.hex,<br>n.trees = 10, interaction.depth = 3,<br>   n.minobsinnode = 2, shrinkage = 0.2, distribution= &quot;gaussian&quot;)</pre><pre><strong># 5. Model Explainability (picture below):</strong><br><em># Run AutoML for 1 minute</em><br>aml &lt;- h2o.automl(y = y, training_frame = train, max_runtime_secs = 60, seed = 1)<br><em># Explain a single H2O model (e.g. leader model from AutoML)</em><br>exm &lt;- h2o.explain(aml@leader, test)<br><em># Explain first row with a single H2O model (e.g. leader model from AutoML)</em><br>h2o.explain_row(aml@leader, test, row_index = 1)<br><em># Methods for an AutoML object</em><br>h2o.varimp_heatmap()<br>h2o.model_correlation_heatmap()<br>h2o.pd_multi_plot()<br><em># Methods for an H2O model</em><br>h2o.residual_analysis_plot()<br>h2o.varimp_plot()<br>h2o.shap_explain_row_plot()<br>h2o.shap_summary_plot()<br>h2o.pd_plot()<br>h2o.ice_plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*mJWHxMwP7d1qgVMNzrN8Nw.png" /></figure><p>The main drawback of H2O ML is that the saved model cannot be run in different version H2O package. It means that if I updates H2O package, I must train my model again to make it compatible with the new environmental<br>All H2O ML method (<a href="https://www.h2o.ai/community/glossary/h2o-h2o-3">link</a>) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U0f8O_8W9PIKTPg3F9CsVA.png" /></figure><ul><li>C<a href="https://cran.r-project.org/web/packages/caret/vignettes/caret.html#:~:text=The%20caret%20package%20(short%20for,complex%20regression%20and%20classification%20problems.&amp;text=caret%20loads%20packages%20as%20needed,a%20prompt%20to%20install%20it.">aret</a></li></ul><p>Definition from <a href="https://topepo.github.io/caret/">https://topepo.github.io/caret/</a>:</p><p>The <a href="http://cran.r-project.org/web/packages/caret/index.html">caret</a> package (short for <strong>C</strong>lassification <strong>A</strong>nd <strong>RE</strong>gression <strong>T</strong>raining) is a set of functions that attempt to streamline the process for creating predictive models. The package contains tools for: data splitting, pre-processing, feature selection, model tuning using resampling, variable importance estimation… The main drawback of caret it that its speed is low compared to H2O and scikit-learn (some caret model were written in C language to improve speed). R is not the best language for speed optimization.</p><pre>Example from this <a href="https://cran.r-project.org/web/packages/caret/vignettes/caret.html#:~:text=The%20caret%20package%20(short%20for,complex%20regression%20and%20classification%20problems.&amp;text=caret%20loads%20packages%20as%20needed,a%20prompt%20to%20install%20it.">link</a>:</pre><pre><strong>library</strong>(caret)<br><strong>library</strong>(mlbench)<br><strong>data</strong>(Sonar)</pre><pre># Train / test split:<br><strong>set.seed</strong>(107)<br>inTrain &lt;- <strong>createDataPartition</strong>(<br>  y = Sonar$Class,<br>  <em>## the outcome data are needed</em><br>  p = .75,<br>  <em>## The percentage of data in the</em><br>  <em>## training set</em><br>  list = FALSE<br>)<br>training &lt;- Sonar[ inTrain,]<br>testing  &lt;- Sonar[-inTrain,]</pre><pre># Apply caret model with hyperparameter tunning and repeated cv </pre><pre>ctrl &lt;- <strong>trainControl</strong>(<br>  method = &quot;repeatedcv&quot;, <br>  repeats = 3,<br>  classProbs = TRUE, <br>  summaryFunction = twoClassSummary<br>)<br><br><strong>set.seed</strong>(123)<br>plsFit &lt;- <strong>train</strong>(<br>  Class ~ .,<br>  data = training,<br>  method = &quot;pls&quot;,<br>  preProc = <strong>c</strong>(&quot;center&quot;, &quot;scale&quot;),<br>  tuneLength = 15,<br>  trControl = ctrl,<br>  metric = &quot;ROC&quot;<br>)<br># Visual hyperparameter tuning performance with ggplot<br><strong>ggplot</strong>(plsFit)</pre><pre># Predict and calculate model performance:<br>plsClasses &lt;- <strong>predict</strong>(plsFit, newdata = testing)<br><strong>str</strong>(plsClasses)<br><em>#&gt;  Factor w/ 2 levels &quot;M&quot;,&quot;R&quot;: 2 1 1 1 2 2 1 2 2 2 ...</em><br>plsProbs &lt;- <strong>predict</strong>(plsFit, newdata = testing, type = &quot;prob&quot;)<br><strong>confusionMatrix</strong>(data = plsClasses, testing$Class)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*3SirwARa9n1uqWoWCw9XmA.png" /></figure><ul><li><a href="https://www.tidymodels.org/">Tidymodels</a> (<a href="https://www.tidymodels.org/">https://www.tidymodels.org/</a>)</li></ul><p>Definition from <a href="https://www.tidymodels.org/">https://www.tidymodels.org/</a>: The tidymodels framework is a collection of packages for modeling and machine learning using <a href="https://www.tidyverse.org/">tidyverse</a> principles. Tidymodels help organize all machine learning process (split data, feature engineering, create model, create workflow, tuning….) in the tidy ways. I think it is easy to use than scikit-learn pipeline (python) and I prefer tidymodels than caret for machine learning model set up (personal idea)</p><p>You can see some important packages in tidymodels in the picture below:<br>(Picture from <a href="https://jhudatascience.org/tidyversecourse/model.html">https://jhudatascience.org/tidyversecourse/model.html</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*fq6Eh5t9W8jAIBoB9tqzdw.png" /></figure><pre>Example from <a href="https://www.tidymodels.org/start/case-study/">https://www.tidymodels.org/start/case-study/</a> :<strong><br>library</strong>(tidymodels)<br>hotels &lt;- <br>  read_csv(&#39;https://tidymodels.org/start/case-study/hotels.csv&#39;) %&gt;%<br>  mutate_if(is.character, as.factor) <br><br># DATA SPLITTING &amp; RESAMPLING (package <strong>rsample</strong>)<br>set.seed(123)<br>splits      &lt;- initial_split(hotels, strata = children)<br><br>hotel_other &lt;- training(splits)<br>hotel_test  &lt;- testing(splits)</pre><pre># BUILD THE MODEL (package <strong>parsnip</strong>)<br>lr_mod &lt;- <br>  logistic_reg(penalty = tune(), mixture = 1) %&gt;% <br>  set_engine(&quot;glmnet&quot;)</pre><pre># CREATE THE RECIPE (package <strong>recipes</strong>)<br>holidays &lt;- c(&quot;AllSouls&quot;, &quot;AshWednesday&quot;, &quot;ChristmasEve&quot;, &quot;Easter&quot;, <br>              &quot;ChristmasDay&quot;, &quot;GoodFriday&quot;, &quot;NewYearsDay&quot;, &quot;PalmSunday&quot;)<br><br>lr_recipe &lt;- <br>  recipe(children ~ ., data = hotel_other) %&gt;% <br>  step_date(arrival_date) %&gt;% <br>  step_holiday(arrival_date, holidays = holidays) %&gt;% <br>  step_rm(arrival_date) %&gt;% <br>  step_dummy(all_nominal(), -all_outcomes()) %&gt;% <br>  step_zv(all_predictors()) %&gt;% <br>  step_normalize(all_predictors())</pre><pre># CREATE THE WORKFLOW (package <strong>workflows</strong>)<br>lr_workflow &lt;- <br>  workflow() %&gt;% <br>  add_model(lr_mod) %&gt;% <br>  add_recipe(lr_recipe)</pre><pre># CREATE THE GRID FOR TUNING<br>lr_reg_grid &lt;- tibble(penalty = 10^seq(-4, -1, length.out = 30))</pre><pre># TRAIN AND TUNE THE MODEL (package <strong>tune</strong>)<br>lr_res &lt;- <br>  lr_workflow %&gt;% <br>  tune_grid(val_set,<br>            grid = lr_reg_grid,<br>            control = control_grid(save_pred = <strong>TRUE</strong>),<br>            metrics = metric_set(roc_auc))<br>lr_best &lt;- <br>  lr_res %&gt;% <br>  collect_metrics() %&gt;% <br>  arrange(penalty) %&gt;% <br>  slice(12)</pre><p><strong>5. Explain model</strong></p><ul><li><a href="https://dalex.drwhy.ai/">Dalex</a> package (please refer very best book for this wonderful package in this <a href="https://ema.drwhy.ai/">link</a>) <a href="https://dalex.drwhy.ai/">https://dalex.drwhy.ai/</a></li></ul><p>Explaining machine learning model is very important step because it will help us understanding why our model predict the result. With Dalex package, the machine learning is not a black box anymore, we can find which important features affect our prediction and make recommendation or explanation to our stakeholder / customer.</p><p>According to Dalex package, we have two explain levels: instance level (one sample) and dataset level (whole samples). Instance level includes break down plots, shapley additive explanations, Lime, ceteris-paribus profiles, ceteris-paribus oscillations, local-diagnostics plots. Dataset level includes model performance, variable importance, partial dependence profiles, local dependence and accumulated local profiles, residual-diagnostic plots</p><p>Overview Dalex package (link from <a href="https://ema.drwhy.ai/introduction.html">https://ema.drwhy.ai/introduction.html</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*8jwdoYlTPrGUuOmsawzE6w.png" /></figure><p>Instance level explanation (link: <a href="https://ema.drwhy.ai/summaryInstanceLevel.html">https://ema.drwhy.ai/summaryInstanceLevel.html</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*UgUIDhpFH7oowKkhPTYs6Q.png" /></figure><p>Dataset level explanation (link: <a href="https://ema.drwhy.ai/summaryModelLevel.html">https://ema.drwhy.ai/summaryModelLevel.html</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*v6DcaK3YT4Hgch0PgcGCGA.png" /></figure><p><strong>6. Communicate Results</strong></p><ul><li><a href="https://rmarkdown.rstudio.com/index.html">R markdown</a> ( <a href="https://rmarkdown.rstudio.com/index.html">https://rmarkdown.rstudio.com/index.html</a>)</li></ul><p>Overview (<a href="https://rmarkdown.rstudio.com/lesson-1.html">link</a>) : R Markdown provides an authoring framework for data science. With R markdown you can save and execute code, generate high quality reports that can be shared with an audience</p><p>Support format (<a href="https://rmarkdown.rstudio.com/index.html">link</a>): R Markdown supports dozens of static and dynamic output formats including <a href="https://bookdown.org/yihui/rmarkdown/html-document.html">HTML</a>, <a href="https://bookdown.org/yihui/rmarkdown/pdf-document.html">PDF</a>, <a href="https://bookdown.org/yihui/rmarkdown/word-document.html">MS Word</a>, <a href="https://bookdown.org/yihui/rmarkdown/beamer-presentation.html">Beamer</a>, <a href="https://bookdown.org/yihui/rmarkdown/ioslides-presentation.html">HTML5 slides</a>, <a href="https://bookdown.org/yihui/rmarkdown/tufte-handouts.html">Tufte-style handouts</a>, <a href="https://bookdown.org/">books</a>, <a href="https://rmarkdown.rstudio.com/flexdashboard/">dashboards</a>, <a href="https://bookdown.org/yihui/rmarkdown/shiny-documents.html">shiny applications</a>, <a href="https://github.com/rstudio/rticles">scientific articles</a>, <a href="https://bookdown.org/yihui/rmarkdown/rmarkdown-site.html">websites</a>, and more.</p><p>Notebook output(<a href="https://rmarkdown.rstudio.com/lesson-10.html">link</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*M93ryA_vhzCiyqOh3RSmNg.png" /></figure><p>HTML output (link) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*ymj1e-O4vw1W25HpRzc4kg.png" /></figure><p>Slide presentation (<a href="https://rmarkdown.rstudio.com/lesson-11.html">link</a>) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*FRjYaOUTaDL2IKM-BmRY6Q.png" /></figure><p>Dashboards (<a href="https://rmarkdown.rstudio.com/lesson-12.html">link</a>) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*dXf48cqqNy8Pft51Oomqpg.png" /></figure><p>Websites (<a href="https://rmarkdown.rstudio.com/lesson-13.html">link</a>) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*vMG30jvcqT1Uhv0t.png" /></figure><p>Interactive documents (<a href="https://rmarkdown.rstudio.com/lesson-14.html">link</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*wabdKuRnoubS3aPX.png" /></figure><p><strong>7. Deployment (web app)</strong></p><p>Today business need application to make decision / report automatically , not just reporting in notebook or power point. Fortunately, R has some powerful packages for this purpose as discussion below:</p><ul><li><a href="https://shiny.rstudio.com/">Shiny</a> (<a href="https://shiny.rstudio.com/">https://shiny.rstudio.com/</a>)</li></ul><p>Overview (<a href="https://shiny.rstudio.com/">link</a>) : Shiny is an R package that makes it easy to build interactive web apps straight from R. You can host standalone apps on a webpage or embed them in <a href="http://rmarkdown.rstudio.com/">R Markdown</a> documents or build <a href="http://rstudio.github.io/shinydashboard/">dashboards</a>. You can also extend your Shiny apps with <a href="http://rstudio.github.io/shinythemes/">CSS themes</a>, <a href="http://www.htmlwidgets.org/">htmlwidgets</a>, and JavaScript <a href="https://github.com/daattali/shinyjs/blob/master/README.md">actions</a>. Shiny combines the computational power of R with the interactivity of the modern web.</p><p>My opinion : with shiny, we can program front end &amp; back end with only R language, no need html, css, javascript…. requirement. Shiny is very useful for data scientist to build stand alone data analytics web app for idea demo or production purpose (deployment with AWS, Shiny server, heroku….)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*EHAwbCPVsvemygyAlWapSA.png" /></figure><p>Very good book to learn shiny: Mastering Shiny by Hadley Wickham (link: <a href="https://mastering-shiny.org/">https://mastering-shiny.org/</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*M6YiCXDp6VHznfB2fU7FYg.png" /></figure><p>Shiny for real estate investment web app (<a href="https://shiny.rstudio.com/gallery/real-estate-investment.html">link</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MugcGZ-jEf2EDv-kGreznQ.png" /></figure><p>Shiny for radiant web app (<a href="https://shiny.rstudio.com/gallery/radiant.html">link</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*bbbE5wEok2WQRk5qeNmrhA.png" /></figure><p>Shiny for trading web app (<a href="https://shiny.rstudio.com/gallery/nz-trade-dash.html">link</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xopKsG7MNxE-Vkw14Ts6JQ.png" /></figure><p>many more….</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4d0af3a9f5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>