<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akmal Abdillah on Medium]]></title>
        <description><![CDATA[Stories by Akmal Abdillah on Medium]]></description>
        <link>https://medium.com/@akmalabdillah?source=rss-4a0a5dd3696f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*IlUv46IMxlj8IzHb1SFdFg.jpeg</url>
            <title>Stories by Akmal Abdillah on Medium</title>
            <link>https://medium.com/@akmalabdillah?source=rss-4a0a5dd3696f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:26:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akmalabdillah/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Python Fundamental #3 — Exploratory Data Analysis (EDA): A Practical Guide for Data Science and…]]></title>
            <link>https://medium.com/data-science-collective/python-fundamental-3-exploratory-data-analysis-eda-a-practical-guide-for-data-science-and-0caa950b8954?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/0caa950b8954</guid>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Sun, 08 Nov 2026 10:53:04 GMT</pubDate>
            <atom:updated>2026-11-08T10:53:04.011Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python Fundamental #3 — Exploratory Data Analysis (EDA): A Practical Guide for Data Science and Machine Learning</h3><p><em>A practical, step-by-step guide to inspecting, cleaning, visualizing, and understanding data with Python before building Machine Learning models.</em></p><h3><strong>1. Introduction</strong></h3><p>In Data Science, building a Machine Learning model is not always the first step toward solving a problem. Before training an algorithm, we need to understand the data we are working with.</p><p>What does the dataset contain? Are there missing values? Are the variables distributed as expected? Do certain features have relationships with one another? Are there unusual observations that could indicate data quality issues or potentially meaningful anomalies?</p><p>These are some of the questions answered through Exploratory Data Analysis (EDA).</p><p>EDA is an essential stage of the Data Science workflow that involves examining, summarizing, and visualizing data to discover patterns, identify inconsistencies, generate hypotheses, and guide subsequent analytical decisions.</p><p>Python provides a rich ecosystem of libraries for this process:</p><ul><li>Pandas for data manipulation and inspection.</li><li>NumPy for numerical operations.</li><li>Matplotlib for flexible data visualization.</li><li>Seaborn for statistical graphics.</li><li>Plotly for interactive data exploration.</li></ul><p>In this article, we will explore EDA step by step, from inspecting a dataset and evaluating data quality to investigating relationships between features and preparing data for Machine Learning. Rather than simply learning individual Python functions, we will focus on how to use them to answer meaningful questions about data.</p><h3><strong>2. What Is Exploratory Data Analysis?</strong></h3><p>Exploratory Data Analysis is the process of investigating a dataset before making assumptions, drawing conclusions, or building predictive models. EDA helps us understand the characteristics of our data through statistical summaries, visualizations, and systematic checks.</p><p>Common EDA activities include:</p><ul><li>Understanding dataset structure and data types.</li><li>Measuring central tendency and variability.</li><li>Identifying missing values and duplicate records.</li><li>Examining distributions and outliers.</li><li>Exploring relationships and correlations between variables.</li><li>Discovering trends and unusual behavior.</li><li>Assessing whether data is suitable for a particular Machine Learning task.</li></ul><p><strong>Why Is Exploratory Data Analysis Important?</strong></p><p>Imagine that you are developing a Machine Learning model to predict industrial equipment temperature. Before training the model, you should investigate whether measurements are recorded consistently, whether timestamps or sensor readings are missing, whether pressure is related to temperature, and whether unusual values represent errors or genuine operating conditions.</p><p>The key principle is simple: understand your data before asking a model to learn from it. Think of EDA as a routine health check-up or inspecting a used car before buying it. You need to understand the data’s overall condition before pushing it through a heavy Machine Learning pipeline.</p><h3><strong>3. EDA in the Data Science Workflow</strong></h3><p>EDA connects raw data collection with data preparation, statistical analysis, and Machine Learning. A typical workflow includes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*qH54oAC5fTafu3L3RTA1mQ.png" /></figure><p>EDA is not strictly linear. Findings discovered during modeling or evaluation may require us to revisit earlier analysis, investigate additional variables, or correct data quality issues.</p><h3><strong>4. Setting Up the Python Environment</strong></h3><p>Install the required libraries:</p><pre>pip install pandas numpy matplotlib seaborn plotly scikit-learn</pre><p>Import the libraries:</p><pre>import pandas as pd<br>import numpy as np<br> <br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import plotly.express as px<br> <br>from sklearn.model_selection import train_test_split<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import classification_report</pre><p>Pandas and NumPy support data manipulation and numerical analysis. Matplotlib and Seaborn provide statistical visualizations, Plotly supports interactive charts, and Scikit-learn provides tools for the Machine Learning demonstration.</p><h3><strong>5. Creating a Sample Dataset</strong></h3><p>To make the tutorial reproducible, we will create a synthetic industrial sensor dataset. It represents equipment measurements collected at regular intervals and includes temperature, pressure, vibration, and an operating-status label. The data is synthetic and intended for education, not for diagnosing real equipment.</p><pre>np.random.seed(42)<br> <br>n = 1000<br>timestamp = pd.date_range(<br>    start=&quot;2026-01-01&quot;,<br>    periods=n,<br>    freq=&quot;h&quot;<br>)<br> <br>temperature = np.random.normal(75, 8, n)<br>pressure = np.random.normal(30, 4, n)<br>vibration = np.random.normal(2.5, 0.6, n)<br> <br># Introduce unusual readings for exploration<br>temperature[50] = 125<br>pressure[120] = np.nan<br>vibration[200] = 8.5<br> <br># Create a synthetic binary target<br>anomaly_score = (<br>    (temperature &gt; 95).astype(int)<br>    + (pressure &gt; 36).astype(int)<br>    + (vibration &gt; 4).astype(int)<br>)<br>anomaly = (anomaly_score &gt;= 1).astype(int)<br> <br>df = pd.DataFrame({<br>    &quot;Timestamp&quot;: timestamp,<br>    &quot;Temperature&quot;: temperature,<br>    &quot;Pressure&quot;: pressure,<br>    &quot;Vibration&quot;: vibration,<br>    &quot;Anomaly&quot;: anomaly<br>})<br> <br># Introduce additional missing values<br>df.loc[[10, 25, 80], &quot;Temperature&quot;] = np.nan<br> <br># Add duplicate records for demonstration<br>df = pd.concat([df, df.iloc[[5, 10]]], ignore_index=True)<br> <br>print(df.head())</pre><p>The dataset contains the following columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*g7vGsvcwSVLfAh_G6FMjDQ.png" /></figure><p>And your result should be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JV7NZ4jxR04zf1Li-oLnlg.png" /></figure><p>We intentionally introduced missing readings, duplicate rows, and unusual values to practice common EDA techniques.</p><h3><strong>6. Understanding the Data Structure</strong></h3><ul><li><em>Inspecting the First, Last, and Random Rows</em></li></ul><pre>print(df.head())<br>print(df.tail())<br>print(df.sample(5, random_state=42))</pre><p>The head() method displays the first five rows by default, while tail() displays the last five. Random sampling helps inspect records beyond the beginning of the dataset.</p><ul><li><em>Checking Dimensions and Data Types</em></li></ul><pre>print(&quot;Dataset shape:&quot;, df.shape)<br>print(&quot;Column names:&quot;, df.columns.tolist())<br>print(&quot;Data types:&quot;)<br>print(df.dtypes)<br>df.info()</pre><p>The shape attribute returns the number of rows and columns. The info() method summarizes column names, non-null counts, data types, and memory usage.</p><ul><li><em>Generating Descriptive Statistics</em></li></ul><pre>print(df.describe())<br>print(df.describe(include=&quot;all&quot;))</pre><p>The mean (average) can be deceiving. If a billionaire walks into a room, the average net worth spikes, but nobody actually got richer. That is why we also look at the median (middle value) for a more realistic picture.</p><p>Information about Descriptitve Statistics parameter:</p><ul><li>Count: number of non-missing observations.</li><li>Mean: arithmetic average.</li><li>Standard deviation: measure of variability.</li><li>Minimum and maximum: smallest and largest observed values.</li><li>Quartiles: values that divide observations into four groups.</li></ul><p>The 50th percentile is the median. Comparing the mean and median can provide an early indication of skewness or unusually large observations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zodc9PUtXs55UZzBXy-FoQ.png" /></figure><p>Data Science insight: Descriptive statistics provide a quick overview, but they do not tell the whole story. Combine statistical summaries with visualizations to understand distributions more completely.</p><h3><strong>7. Data Quality Assesment</strong></h3><p><strong>7.1 Detecting Missing Values</strong></p><p>Missing data is like a temperature sensor temporarily dropping out. Deleting that row might cause you to miss key patterns, but blindly filling in values could trigger false alarms.</p><pre>missing_count = df.isnull().sum()<br>missing_percentage = (df.isnull().mean() * 100).round(2)<br> <br>missing_summary = pd.DataFrame({<br>    &quot;Missing_Count&quot;: missing_count,<br>    &quot;Missing_Percentage&quot;: missing_percentage<br>})<br> <br>print(missing_summary.sort_values(<br>    &quot;Missing_Percentage&quot;,<br>    ascending=False<br>))</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*olCIen_8EDJEaU4Ops3r9g.png" /></figure><p>Missing data can occur because of sensor communication failures, incomplete data entry, equipment downtime, or errors during data integration. The reason for missingness matters when choosing a treatment.</p><p><strong>7.2 Visualizing Missing Values</strong></p><pre>missing = df.isnull().sum()<br>fig, axes = plt.subplots(1, 2, figsize=(12, 4))<br><br># 1) Number of missing values per column<br>missing.plot(kind=&quot;bar&quot;, color=&quot;red&quot;, rot=0, ax=axes[0])<br>axes[0].bar_label(axes[0].containers[0])<br>axes[0].set_title(&quot;Missing Values per Column&quot;)<br><br># 2) Location of missing values (row index)<br>rows, cols = np.where(df.isnull())<br>axes[1].scatter(rows, df.columns[cols], marker=&quot;|&quot;, s=500, color=&quot;red&quot;)<br>axes[1].margins(y=0.5)<br>axes[1].set_title(&quot;Location of Missing Values&quot;)<br>axes[1].set_xlabel(&quot;Row index&quot;)<br><br>plt.tight_layout()<br>plt.show()<br><br>display(df[df.isnull().any(axis=1)])</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Sg6MAD-umsivrDKxjLCPQ.png" /></figure><p>This visualization helps reveal where missing values occur. For very large datasets, a summary table or sampled view may be more efficient than plotting every row.</p><p><strong>7.3 Detecting Duplicate Records</strong></p><pre>duplicate_count = df.duplicated().sum()<br>print(&quot;Duplicate rows:&quot;, duplicate_count)<br><br>display(df[df.duplicated(keep=False)].sort_values(&quot;Timestamp&quot;))<br><br># Remove exact duplicate rows for this demonstration<br>df_clean = df.drop_duplicates().copy()<br>print(&quot;Rows before:&quot;, len(df))<br>print(&quot;Rows after:&quot;, len(df_clean))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_8M-PoYaWdIHXY_alII_BA.png" /></figure><p>Duplicates may result from repeated data ingestion, merged files, or repeated records from an upstream system. However, duplicate timestamps are not necessarily duplicate records. Remove duplicates only when they violate the intended data structure.</p><p><strong>7.4 Checking Data Types</strong></p><pre>df_clean[&quot;Timestamp&quot;] = pd.to_datetime(<br>    df_clean[&quot;Timestamp&quot;],<br>    errors=&quot;coerce&quot;<br>)<br><br>print(df_clean.dtypes)<br>print(&quot;Invalid timestamps:&quot;, df_clean[&quot;Timestamp&quot;].isna().sum())</pre><p>The errors=’coerce’ parameter converts unparseable timestamps to NaT, which should then be investigated. Data type validation is especially important for dates, categorical variables, identifiers, and measurements imported from external systems.</p><h3><strong>8. Handling Missing Values</strong></h3><p>Finding missing values is only the beginning. The next step is deciding how to handle them. There is no single method appropriate for every dataset.</p><p><em>Option 1: Remove Rows with Missing Values</em></p><pre>df_dropped = df_clean.dropna(<br>    subset=[&quot;Temperature&quot;, &quot;Pressure&quot;, &quot;Vibration&quot;]<br>)</pre><p>This can be appropriate when only a small number of rows are affected and their removal does not introduce significant bias. Dropping many observations can reduce available information.</p><p><em>Option 2: Impute Using Median</em></p><pre>df_imputed = df_clean.copy()<br> <br>for column in [&quot;Temperature&quot;, &quot;Pressure&quot;, &quot;Vibration&quot;]:<br>    df_imputed[column] = df_imputed[column].fillna(<br>        df_imputed[column].median()<br>    )</pre><p>The median is often more robust to extreme values than the mean. However, median imputation can distort distributions and relationships. Choose it based on the data and analytical objective.</p><p><em>Option 3: Forward-Fill Time-Series Data</em></p><pre>df_time = df_clean.sort_values(&quot;Timestamp&quot;).copy()<br>df_time[&quot;Temperature&quot;] = df_time[&quot;Temperature&quot;].ffill()</pre><p>Forward-fill carries the previous observed value into the next missing observation. It may be appropriate when readings are frequent and the underlying process changes slowly, but it can be misleading during shutdowns, rapid process changes, or long data gaps.</p><p>Machine Learning practice: Fit imputation rules using the training data only, then apply those rules to validation and test data. This helps prevent information leakage.</p><h3><strong>9. Detecting and Investigating Outliers</strong></h3><p>Outliers are observations that differ substantially from the rest of the data. They may represent measurement errors, sensor faults, unusual operating conditions, or genuine events of interest.</p><p><strong>Why It Matters:</strong> Extreme values are not always errors. In industrial systems, a sudden temperature spike might actually be an early warning sign of equipment failure. Don’t remove outliers without investigation!</p><p><strong>9.1. Using Box Plots</strong></p><pre>plt.figure(figsize=(10, 5))<br>sns.boxplot(<br>    data=df_imputed[[&quot;Temperature&quot;, &quot;Pressure&quot;, &quot;Vibration&quot;]]<br>)<br>plt.title(&quot;Box Plot of Industrial Sensor Variables&quot;)<br>plt.tight_layout()<br>plt.show()</pre><p>Box plots show the median, quartiles, and potential outliers. Points outside the whiskers are flagged according to the plotting convention, not automatically classified as errors.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*lgyDJPakyjjcDAPtnOsKDQ.png" /></figure><p><strong>9.2. Using the Interquartile Range (IQR)</strong></p><p>The IQR method flags values outside a range based on the first and third quartiles.</p><pre>column = &quot;Temperature&quot;<br> <br>Q1 = df_imputed[column].quantile(0.25)<br>Q3 = df_imputed[column].quantile(0.75)<br>IQR = Q3 - Q1<br> <br>lower_bound = Q1 - 1.5 * IQR<br>upper_bound = Q3 + 1.5 * IQR<br> <br>outliers = df_imputed[<br>    (df_imputed[column] &lt; lower_bound)<br>    | (df_imputed[column] &gt; upper_bound)<br>]<br> <br>print(outliers[[&quot;Timestamp&quot;, column]])</pre><p>This method is useful for identifying potential outliers, but it does not determine whether those values are incorrect. Investigate their source, context, and possible significance before removing or transforming them.</p><p>Industrial data insight: An extreme temperature reading could be an invalid sensor value — or an early warning of a real problem. Removing it without investigation could eliminate an important event.</p><h3><strong>10. Univariate Analysis: Understanding Individual Variables</strong></h3><p>Univariate analysis examines one variable at a time. Its goal is to understand the distribution, central tendency, variability, and possible irregularities in each feature.</p><p><strong>10.1. Histograms</strong></p><pre>numeric_columns = [&quot;Temperature&quot;, &quot;Pressure&quot;, &quot;Vibration&quot;]<br> <br>df_imputed[numeric_columns].hist(<br>    figsize=(12, 8),<br>    bins=30,<br>    edgecolor=&quot;black&quot;<br>)<br>plt.suptitle(&quot;Distribution of Sensor Measurements&quot;)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8b4s39gCOvxdlEP0-6pUgg.png" /></figure><p><strong>10.2. Comparing Mean and Median (Summary Statistics)</strong></p><pre>summary = df_imputed[numeric_columns].agg(<br>    [&quot;mean&quot;, &quot;median&quot;, &quot;std&quot;, &quot;min&quot;, &quot;max&quot;]<br>)<br>print(summary)</pre><p>The mean and median provide complementary information about central tendency. A substantial difference may indicate skewness or influential observations, but it is not sufficient by itself to establish distribution shape.</p><p><strong>10.3. Categorical Frequency Analysis</strong></p><pre>print(df_imputed[&quot;Anomaly&quot;].value_counts())<br> <br>print(<br>    df_imputed[&quot;Anomaly&quot;]<br>    .value_counts(normalize=True)<br>    .mul(100)<br>    .round(2)<br>)</pre><p>Class frequency is important in classification problems. A model can achieve high accuracy while performing poorly on a rare but important class.</p><h3><strong>11. Bivariate Analysis: Exploring Relationships Between Variables</strong></h3><p>Bivariate analysis investigates relationships between two variables. It helps us understand whether changes in one variable are associated with changes in another.</p><p><strong>11.1. Scatter Plots</strong></p><pre>fig = px.scatter(<br>    df_imputed,<br>    x=&quot;Temperature&quot;,<br>    y=&quot;Pressure&quot;,<br>    color=&quot;Anomaly&quot;,<br>    hover_data=[&quot;Timestamp&quot;, &quot;Vibration&quot;],<br>    title=&quot;Temperature vs. Pressure by Anomaly Label&quot;<br>)<br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CeIX4AEfAtAt62m3ev9pnw.png" /></figure><p>This interactive Plotly chart allows users to zoom, inspect individual observations, and compare points by anomaly label. Scatter plots can reveal relationships, clusters, and unusual observations.</p><p>A relationship between two variables does not prove that one causes the other. Operating conditions, load, ambient temperature, and other factors may influence both.</p><p><strong>11.2. Comparing Distributions by Category</strong></p><pre>plt.figure(figsize=(9, 5))<br>sns.boxplot(<br>    data=df_imputed,<br>    x=&quot;Anomaly&quot;,<br>    y=&quot;Temperature&quot;<br>)<br>plt.title(&quot;Temperature Distribution by Anomaly Label&quot;)<br>plt.xlabel(&quot;Anomaly Label (0 = Normal, 1 = Anomaly)&quot;)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*2-6tsE3TNGSlieK6If_Tmw.png" /></figure><p>This comparison helps assess whether temperature differs across the two groups. In this tutorial, labels were generated using predefined rules, so the resulting relationships are partly built into the synthetic data. Real projects require independent evidence and reliable labels.</p><h3><strong>12. Correlation Analysis</strong></h3><p>Correlation measures the strength and direction of an association between two variables. It can help explore relationships, identify potentially redundant features, and understand a dataset before modeling.</p><p><strong>Remember:</strong> Correlation does not equal causation. Ice cream sales and shark attacks both rise in the summer, but buying ice cream won’t make a shark attack you.</p><p><strong>12.1. Pearson Correlation</strong></p><ul><li>+1: perfect positive linear correlation.</li><li>0: no linear correlation.</li><li>-1: perfect negative linear correlation.</li></ul><p>A correlation close to zero does not necessarily mean there is no relationship; the relationship may be nonlinear.</p><p><strong>12.2. Creating Correlation Matrix</strong></p><pre>correlation_matrix = df_imputed[<br>    numeric_columns + [&quot;Anomaly&quot;]<br>].corr(method=&quot;pearson&quot;)<br> <br>plt.figure(figsize=(9, 6))<br>sns.heatmap(<br>    correlation_matrix,<br>    annot=True,<br>    cmap=&quot;coolwarm&quot;,<br>    fmt=&quot;.2f&quot;,<br>    vmin=-1,<br>    vmax=1<br>)<br>plt.title(&quot;Correlation Matrix&quot;)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*dII96jrbxwphS0BAKQGOAg.png" /></figure><p>A high correlation between two input features may indicate redundancy or multicollinearity, which can affect certain models and complicate interpretation.</p><p><em>Spearman Correlation</em></p><pre>spearman_corr = df_imputed[<br>    numeric_columns + [&quot;Anomaly&quot;]<br>].corr(method=&quot;spearman&quot;)<br> <br>print(spearman_corr)</pre><p>Spearman correlation can be more informative for monotonic relationships that are not necessarily linear. Correlation should not be the only basis for feature selection: a feature with low linear correlation may still be valuable to a nonlinear model.</p><p><em>Kendall’s Tau Correlation</em></p><p>A non-parametric correlation metric that evaluates the strength and direction of association between two variables based on concordant and discordant pairs. Unlike Pearson, which measures linear relationships, and Spearman, which relies on rank differences, Kendall’s Tau measures rank consistency. It is particularly robust for small sample sizes and dataset structures containing numerous tied ranks (identical values).</p><h3><strong>13. EDA for Time Series Data</strong></h3><p>Time-series analysis is relevant when observations are recorded over time, such as industrial sensor readings, financial prices, energy consumption, or product demand. Unlike ordinary tabular data, time-series data has a temporal structure that must be preserved.</p><p><strong>13.1. Preparing the Time Index</strong></p><pre>df_ts = (<br>    df_imputed<br>    .sort_values(&quot;Timestamp&quot;)<br>    .set_index(&quot;Timestamp&quot;)<br>)<br>print(df_ts.head())</pre><p><strong>13.2. Visualizing Sensor Trends</strong></p><pre>fig = px.line(<br>    df_ts.reset_index(),<br>    x=&quot;Timestamp&quot;,<br>    y=&quot;Temperature&quot;,<br>    title=&quot;Temperature Over Time&quot;<br>)<br>fig.show()</pre><p><strong>13.3. Checking Time Intervals</strong></p><pre>time_differences = (<br>    df_ts.index.to_series()<br>    .diff()<br>    .value_counts()<br>)<br>print(time_differences.head())</pre><p>This helps investigate whether observations arrive at consistent intervals. For irregularly sampled data, distinguish legitimate gaps from missing records.</p><p><strong>13.4. Rolling Statistics</strong></p><pre>df_ts[&quot;Temperature_Rolling_Mean&quot;] = (<br>    df_ts[&quot;Temperature&quot;]<br>    .rolling(window=24, min_periods=12)<br>    .mean()<br>)<br> <br>df_ts[&quot;Temperature_Rolling_Std&quot;] = (<br>    df_ts[&quot;Temperature&quot;]<br>    .rolling(window=24, min_periods=12)<br>    .std()<br>)<br> <br>fig = px.line(<br>    df_ts.reset_index(),<br>    x=&quot;Timestamp&quot;,<br>    y=[&quot;Temperature&quot;, &quot;Temperature_Rolling_Mean&quot;],<br>    title=&quot;Temperature and 24-Observation Rolling Mean&quot;<br>)<br>fig.show()</pre><p>Because the synthetic dataset uses hourly observations, a 24-observation window represents approximately 24 hours when the data has no gaps. Rolling statistics are descriptive tools, not proof of an anomaly or a forecast. For time-series Machine Learning, consider chronological validation, seasonality, and information available at prediction time.</p><h3><strong>14. EDA for Machine Learning</strong></h3><p>EDA helps determine whether a dataset is suitable for a Machine Learning task and what preprocessing or modeling challenges may arise. The approach depends on whether the problem involves <em>classification, regression, clustering, forecasting, or anomaly detection</em>.</p><p><strong>14.1. Understanding the Target Variable</strong></p><pre>target_counts = df_imputed[&quot;Anomaly&quot;].value_counts()<br>print(target_counts)<br> <br>print(<br>    df_imputed[&quot;Anomaly&quot;]<br>    .value_counts(normalize=True)<br>    .mul(100)<br>    .round(2)<br>)</pre><p>If one class is rare, pay attention to recall, precision, class weighting, or resampling methods. The right approach depends on the consequences of false positives and false negatives.</p><p><strong>14.2. Investigating Feature Relationships with the Target</strong></p><pre>feature_summary = (<br>    df_imputed<br>    .groupby(&quot;Anomaly&quot;)[numeric_columns]<br>    .mean()<br>)<br>print(feature_summary)</pre><p>This provides an initial comparison of average sensor measurements between groups. Differences in group averages do not guarantee that a feature will improve model performance.</p><p><strong>14.3. Preparing a Baseline Classification Model</strong></p><p>We can demonstrate the next step by training a simple <em>Random Forest classifier</em>. First, remove rows that contain missing feature values and split the dataset into training and test sets.</p><pre>features = [&quot;Temperature&quot;, &quot;Pressure&quot;, &quot;Vibration&quot;]<br> <br>model_data = df[features + [&quot;Anomaly&quot;]].dropna().copy()<br>X = model_data[features]<br>y = model_data[&quot;Anomaly&quot;]<br> <br>X_train, X_test, y_train, y_test = train_test_split(<br>    X, y,<br>    test_size=0.2,<br>    random_state=42,<br>    stratify=y<br>)<br> <br>model = RandomForestClassifier(<br>    n_estimators=100,<br>    random_state=42,<br>    class_weight=&quot;balanced&quot;<br>)<br> <br>model.fit(X_train, y_train)<br>y_pred = model.predict(X_test)<br> <br>print(classification_report(<br>    y_test, y_pred, zero_division=0<br>))</pre><p>The classification report provides precision, recall, F1-score, and support for each class.</p><ul><li><strong>Precision</strong>: among observations predicted as anomalous, how many were actually labeled anomalous?</li><li><strong>Recall</strong>: among observations labeled anomalous, how many did the model identify?</li><li><strong>F1-score</strong>: the harmonic mean of precision and recall.</li><li><strong>Support</strong>: the number of true observations in each class.</li></ul><p><strong>Recall </strong>may be especially important when missing a genuine abnormal event is costly. However, high <strong>recall </strong>with too many false alarms may also be impractical.</p><p><strong><em>Important limitation</em></strong>: This is an educational baseline, not a validated industrial solution. The synthetic target is based on rules involving the same input features, so the experiment does not demonstrate real-world predictive performance.</p><p><strong><em>Avoiding Data Leakage</em></strong></p><p>Data leakage occurs when information unavailable at prediction time influences model training or evaluation. To reduce this risk:</p><ul><li>Split data before learning preprocessing parameters.</li><li>Fit imputers and scalers on the training set only.</li><li>Keep test data separate until final evaluation.</li><li>Use chronological splits for forecasting and time-dependent prediction tasks.</li><li>Avoid features that directly reveal the target.</li><li>Check whether labels or features use information from the future.</li></ul><p>A model should be evaluated on data that realistically represents the conditions in which it will be used.</p><h3><strong>15. From EDA to Feature Engineering</strong></h3><p>After exploring the dataset, the next step is to decide how findings should influence data preparation and feature engineering. Feature engineering involves creating, transforming, or selecting variables that help a Machine Learning model learn useful patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6hnakfrg_5hPgjOUlHGjg.png" /></figure><p>Not every dataset requires all these steps. The appropriate preprocessing strategy depends on the data, model, and scientific or business objective. For time-series problems, features used to predict the future must be calculated using information available at the relevant prediction time.</p><p>EDA should help us make informed decisions , not encourage us to apply every preprocessing technique automatically.</p><h3><strong>16. Common Mistake to Avoid During EDA</strong></h3><ul><li><strong>Removing outliers without investigating them. </strong>Extreme observations may represent meaningful events. Removing them automatically can hide important patterns.</li><li><strong>Treating correlation as causation. </strong>Two variables can move together without one causing the other. Additional analysis and domain knowledge are needed.</li><li><strong>Ignoring missing-data mechanisms. </strong>Missingness can be associated with operating conditions, collection processes, or the variable itself.</li><li><strong>Using visualizations without asking a question. </strong>Charts should help answer specific questions and communicate meaningful findings.</li><li><strong>Overlooking class imbalance. </strong>A classification dataset dominated by one class can make accuracy misleading.</li><li><strong>Allowing data leakage. </strong>Using test data to make preprocessing or modeling decisions can produce overly optimistic evaluation.</li><li><strong>Treating EDA as a one-time activity. </strong>EDA is iterative; model evaluation and domain feedback can lead to further investigation.</li></ul><h3><strong>17. Conclusion</strong></h3><p>Exploratory Data Analysis is one of the most important steps in the Data Science workflow because it helps us understand what our data contains before making decisions based on it.</p><p>Using Python libraries such as Pandas, NumPy, Matplotlib, Seaborn, and Plotly, we can inspect dataset structures, evaluate data quality, analyze distributions, discover relationships, and investigate unusual observations.</p><p>More importantly, EDA connects technical analysis with real-world problems. In industrial analytics, for example, exploring temperature, pressure, vibration, and time-series behavior can help identify patterns worth investigating before developing forecasting or anomaly-detection models.</p><p>However, finding a pattern does not automatically establish its cause, and detecting an unusual observation does not necessarily mean that a failure has occurred. Meaningful analysis requires statistical reasoning, domain knowledge, and careful validation.</p><p>EDA is not just about exploring data. It is about asking the right questions, discovering meaningful evidence, and making better decisions about what to do next.</p><p><strong>Key Takeaway:</strong> <em>Machine Learning values data quality over algorithm complexity. High-performing models are rarely born from complex algorithms alone; they are built on a foundation of deep data understanding through structured EDA.</em></p><p>Iam Akmal Abdillah, keep exploring, keep experimenting and keep learning! Thank You!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0caa950b8954" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/python-fundamental-3-exploratory-data-analysis-eda-a-practical-guide-for-data-science-and-0caa950b8954">Python Fundamental #3 — Exploratory Data Analysis (EDA): A Practical Guide for Data Science and…</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Fundamental #2 — Data Visualization in Python]]></title>
            <link>https://medium.com/data-science-collective/python-fundamental-2-data-visualization-in-python-2b38cac206dd?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b38cac206dd</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 13:30:26 GMT</pubDate>
            <atom:updated>2026-10-11T13:30:26.019Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T8QlsTnpLXaBzcE3" /></figure><p>In my previous article, we discussed two fundamental Python libraries for data science: <strong>NumPy and Pandas</strong>.</p><p><a href="https://medium.com/@akmalabdillah/python-fundamental-1-pandas-and-numpy-2ce5265b445d">Python Fundamental #1 — Pandas and Numpy</a></p><p>NumPy is commonly used for numerical computing, while Pandas is widely used to manipulate and analyze tabular data. After processing our data using Pandas and NumPy, the next step is learning how to <strong>present that data in a visual format that is easy to understand</strong>.</p><p>This is known as <strong>Data Visualization</strong>.</p><p>Data visualization is not simply about creating attractive charts. Its primary purpose is to help us discover patterns, trends, relationships, distributions, and anomalies that may be difficult to identify by looking at numbers in a table.</p><p>Python provides many libraries for data visualization. Some of the most commonly used are:</p><ul><li><strong>Matplotlib</strong> — fundamental plotting library</li><li><strong>Seaborn</strong> — statistical data visualization</li><li><strong>Plotly</strong> — interactive visualization</li><li><strong>Dash</strong> — building analytical dashboards with Plotly</li></ul><p>In this article, we will explore each of them step by step.</p><h3>1. Preparing the Dataset</h3><p>Before creating visualizations, we need a dataset.</p><p>To make our examples easy to run without depending on external files, we will create a simple dataset using Pandas.</p><p>The dataset represents sales data from a company.</p><pre>import pandas as pd<br>import numpy as np<br><br># Create dataset<br>data = {<br>    &quot;Date&quot;: [<br>        &quot;2026-01-01&quot;,<br>        &quot;2026-01-02&quot;,<br>        &quot;2026-01-03&quot;,<br>        &quot;2026-01-04&quot;,<br>        &quot;2026-01-05&quot;,<br>        &quot;2026-01-06&quot;,<br>        &quot;2026-01-07&quot;,<br>        &quot;2026-01-08&quot;,<br>        &quot;2026-01-09&quot;,<br>        &quot;2026-01-10&quot;<br>    ],<br>    &quot;Product&quot;: [<br>        &quot;Laptop&quot;,<br>        &quot;Mouse&quot;,<br>        &quot;Keyboard&quot;,<br>        &quot;Monitor&quot;,<br>        &quot;Laptop&quot;,<br>        &quot;Mouse&quot;,<br>        &quot;Keyboard&quot;,<br>        &quot;Monitor&quot;,<br>        &quot;Laptop&quot;,<br>        &quot;Mouse&quot;<br>    ],<br>    &quot;Category&quot;: [<br>        &quot;Computer&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Accessories&quot;<br>    ],<br>    &quot;Sales&quot;: [<br>        15000000,<br>        500000,<br>        800000,<br>        4000000,<br>        12000000,<br>        700000,<br>        900000,<br>        4500000,<br>        17000000,<br>        600000<br>    ],<br>    &quot;Quantity&quot;: [<br>        2,<br>        10,<br>        8,<br>        3,<br>        2,<br>        14,<br>        9,<br>        3,<br>        2,<br>        12<br>    ]<br>}<br><br>df = pd.DataFrame(data)<br><br># Convert Date to datetime<br>df[&quot;Date&quot;] = pd.to_datetime(df[&quot;Date&quot;])<br><br>print(df)</pre><p>The resulting DataFrame will look approximately like this:</p><pre>        Date   Product     Category     Sales  Quantity<br>0 2026-01-01    Laptop     Computer  15000000         2<br>1 2026-01-02     Mouse  Accessories    500000        10<br>2 2026-01-03  Keyboard  Accessories    800000         8<br>3 2026-01-04   Monitor     Computer   4000000         3<br>4 2026-01-05    Laptop     Computer  12000000         2<br>5 2026-01-06     Mouse  Accessories    700000        14<br>6 2026-01-07  Keyboard  Accessories    900000         9<br>7 2026-01-08   Monitor     Computer   4500000         3<br>8 2026-01-09    Laptop     Computer  17000000         2<br>9 2026-01-10     Mouse  Accessories    600000        12</pre><p>Now we have a dataset that can be used to create various types of visualizations.</p><h3>2. Matplotlib</h3><h3>What is Matplotlib?</h3><p><strong>Matplotlib</strong> is one of the fundamental Python libraries for creating data visualizations.</p><p>Matplotlib provides detailed control over various elements of a chart, including:</p><ul><li>title</li><li>axis</li><li>labels</li><li>legend</li><li>grid</li><li>markers</li><li>lines</li><li>figure size</li></ul><p>Therefore, understanding Matplotlib is an important foundation before moving to higher-level visualization libraries.</p><p>Import the library:</p><pre>import matplotlib.pyplot as plt</pre><h3>3. Line Chart</h3><p>A line chart is commonly used to observe <strong>changes or trends over time</strong>.</p><p>For example, we want to see how sales change each day.</p><pre>import matplotlib.pyplot as plt<br><br>plt.figure(figsize=(10, 5))<br><br>plt.plot(<br>    df[&quot;Date&quot;],<br>    df[&quot;Sales&quot;],<br>    marker=&quot;o&quot;<br>)<br><br>plt.title(&quot;Daily Sales&quot;)<br>plt.xlabel(&quot;Date&quot;)<br>plt.ylabel(&quot;Sales&quot;)<br><br>plt.xticks(rotation=45)<br>plt.grid(True)<br><br>plt.tight_layout()<br>plt.show()</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*Y5WzBmxlWSb-KdHDY6-XVw.png" /></figure><h3>Explanation</h3><p>This section:</p><pre>plt.plot(<br>    df[&quot;Date&quot;],<br>    df[&quot;Sales&quot;],<br>    marker=&quot;o&quot;<br>)</pre><p>means:</p><ul><li>X-axis → Date</li><li>Y-axis → Sales</li><li>marker=&quot;o&quot; → adds a marker to each data point</li></ul><p>Line charts are useful for:</p><ul><li>sales trends</li><li>stock prices</li><li>temperature</li><li>production</li><li>sensor data</li><li>machine performance</li><li>time series forecasting</li></ul><p>In real-world Data Science and Machine Learning projects, line charts are also useful for examining <strong>sensor trends before building forecasting models</strong>.</p><h3>4. Bar Chart</h3><p>If we want to compare values across different categories, we can use a <strong>bar chart</strong>.</p><p>First, we aggregate the data using Pandas.</p><pre>sales_by_product = (<br>    df.groupby(&quot;Product&quot;)[&quot;Sales&quot;]<br>    .sum()<br>    .sort_values(ascending=False)<br>)<br><br>print(sales_by_product)</pre><p>Output can be:</p><pre>Product<br>Laptop      44000000<br>Monitor      8500000<br>Mouse        1800000<br>Keyboard     1700000<br>Name: Sales, dtype: int64</pre><p>Then visualize the result:</p><pre>plt.figure(figsize=(8, 5))<br><br>plt.bar(<br>    sales_by_product.index,<br>    sales_by_product.values<br>)<br><br>plt.title(&quot;Total Sales by Product&quot;)<br>plt.xlabel(&quot;Product&quot;)<br>plt.ylabel(&quot;Total Sales&quot;)<br><br>plt.xticks(rotation=45)<br>plt.tight_layout()<br><br>plt.show()</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*rOffNvt40qY867OpjCM9_Q.png" /></figure><p>A bar chart is useful for answering questions such as:</p><blockquote><em>Which product has the highest total sales?</em></blockquote><p>Or:</p><blockquote><em>Which area has the highest production?</em></blockquote><h3>5. Scatter Plot</h3><p>A scatter plot is used to examine the <strong>relationship between two numerical variables</strong>.</p><p>For example:</p><pre>plt.figure(figsize=(8, 5))<br><br>plt.scatter(<br>    df[&quot;Quantity&quot;],<br>    df[&quot;Sales&quot;]<br>)<br><br>plt.title(&quot;Quantity vs Sales&quot;)<br>plt.xlabel(&quot;Quantity&quot;)<br>plt.ylabel(&quot;Sales&quot;)<br><br>plt.grid(True)<br>plt.show()</pre><p>Output :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*AX888LdePIhXnYGGDL7Yeg.png" /></figure><p>Here:</p><p><strong>X-axis</strong></p><pre>Quantity</pre><p><strong>Y-axis</strong></p><pre>Sales</pre><p>A scatter plot can help us identify whether there is a relationship or pattern between two variables.</p><p>In Machine Learning, this type of visualization is frequently used during <strong>Exploratory Data Analysis (EDA)</strong> before selecting features for a model.</p><h3>6. Histogram</h3><p>A histogram is used to visualize the <strong>distribution of a numerical variable</strong>.</p><p>For example:</p><pre>plt.figure(figsize=(8, 5))<br><br>plt.hist(<br>    df[&quot;Sales&quot;],<br>    bins=5<br>)<br><br>plt.title(&quot;Sales Distribution&quot;)<br>plt.xlabel(&quot;Sales&quot;)<br>plt.ylabel(&quot;Frequency&quot;)<br><br>plt.show()</pre><p>Output can be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*VDhAlwOp1MkFsw08xCRrrg.png" /></figure><p>A histogram can help answer questions such as:</p><ul><li>How is the data distributed?</li><li>Is the data concentrated within a particular range?</li><li>Is there an indication of skewness?</li><li>Are there potential outliers?</li></ul><p>This becomes important when we move into <strong>statistical analysis and machine learning</strong>.</p><h3>7. Seaborn</h3><p>After understanding Matplotlib, we can move on to <strong>Seaborn</strong>.</p><p>Seaborn is built on top of Matplotlib and provides a higher-level interface for creating statistical visualizations.</p><p>Install Seaborn:</p><pre>pip install seaborn</pre><p>How to import the library:</p><pre>import seaborn as sns</pre><h3>8. Seaborn Bar Plot</h3><p>For example:</p><pre>plt.figure(figsize=(8, 5))<br><br>sns.barplot(<br>    data=df,<br>    x=&quot;Product&quot;,<br>    y=&quot;Sales&quot;<br>)<br><br>plt.title(&quot;Average Sales by Product&quot;)<br>plt.xticks(rotation=45)<br><br>plt.tight_layout()<br>plt.show()</pre><p>Output can be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*s5EE8l_5otBoJGzGRhbIFA.png" /></figure><p>Compared with manually creating charts using Matplotlib, Seaborn provides a simpler API for many statistical visualization tasks.</p><h3>9. Box Plot</h3><p>One of the most useful visualizations in data analysis is the <strong>box plot</strong>.</p><pre>plt.figure(figsize=(8, 5))<br><br>sns.boxplot(<br>    data=df,<br>    x=&quot;Category&quot;,<br>    y=&quot;Sales&quot;<br>)<br><br>plt.title(&quot;Sales Distribution by Category&quot;)<br><br>plt.show()</pre><p>Output can be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*wnXGpFtv3S77Qg5r8sCIKg.png" /></figure><p>A box plot can be used to examine:</p><ul><li>median</li><li>quartiles</li><li>spread</li><li>potential outliers</li></ul><p>For example, in a Machine Learning workflow:</p><pre>Sensor Data<br>     ↓<br>Exploratory Data Analysis<br>     ↓<br>Box Plot<br>     ↓<br>Detect Potential Outliers<br>     ↓<br>Data Cleaning<br>     ↓<br>Machine Learning</pre><h3>10. Heatmap</h3><p>A heatmap is a matrix-based visualization that uses color intensity to represent values.</p><p>Heatmaps are particularly popular for visualizing a <strong>correlation matrix</strong>.</p><p>First, select the numerical data:</p><pre>numeric_df = df[<br>    [&quot;Sales&quot;, &quot;Quantity&quot;]<br>]</pre><p>Then calculate the correlation:</p><pre>correlation = numeric_df.corr()<br><br>print(correlation)</pre><p>The result will look approximately like:</p><pre>             Sales  Quantity<br>Sales     1.000000 -0.787926<br>Quantity -0.787926  1.000000</pre><p>Then visualize it:</p><pre>plt.figure(figsize=(7, 5))<br><br>sns.heatmap(<br>    correlation,<br>    annot=True,<br>    cmap=&quot;coolwarm&quot;,<br>    fmt=&quot;.2f&quot;<br>)<br><br>plt.title(&quot;Correlation Matrix&quot;)<br><br>plt.show()</pre><p>Then the result can be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*S04xHH3IPfdK1xGMjtZFwQ.png" /></figure><p>The parameter:</p><pre>annot=True</pre><p>is used to display the correlation value inside each cell.</p><p>Meanwhile:</p><pre>fmt=&quot;.2f&quot;</pre><p>means that the values will be displayed with two decimal places.</p><p>Heatmaps are frequently used during <strong>feature analysis</strong>.</p><h3>11. Plotly</h3><p>Matplotlib and Seaborn are excellent for static visualizations.</p><p>However, sometimes we need charts that allow users to:</p><ul><li>hover</li><li>zoom</li><li>pan</li><li>display tooltips</li><li>interact with data</li></ul><p>For these requirements, we can use <strong>Plotly</strong>.</p><p>Plotly provides Python tools for creating interactive visualizations and supports various chart types such as line, bar, scatter, histogram, box plot, and heatmap.</p><p>Install Plotly:</p><pre>pip install plotly</pre><p>Then:</p><pre>import plotly.express as px</pre><h3>12. Interactive Line Chart</h3><p>For example:</p><pre>fig = px.line(<br>    df,<br>    x=&quot;Date&quot;,<br>    y=&quot;Sales&quot;,<br>    title=&quot;Daily Sales&quot;<br>)<br><br>fig.show()</pre><p>Unlike:</p><pre>plt.show()</pre><p>Plotly produces an interactive visualization.</p><p>Users can:</p><ul><li>hover over data points</li><li>zoom</li><li>pan</li><li>inspect individual values</li></ul><p>Plotly also uses the concept of a <strong>Figure</strong> as the main structure for a chart, while Plotly Express provides a high-level interface for creating common visualizations.</p><h3>13. Interactive Bar Chart</h3><p>We can create an interactive bar chart using:</p><pre>sales_by_product = (<br>    df.groupby(&quot;Product&quot;, as_index=False)[&quot;Sales&quot;]<br>    .sum()<br>)<br><br>fig = px.bar(<br>    sales_by_product,<br>    x=&quot;Product&quot;,<br>    y=&quot;Sales&quot;,<br>    title=&quot;Total Sales by Product&quot;<br>)<br><br>fig.show()</pre><p>Output can be (the result allows users to explore the data interactively):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cgVbS-6nNb-JBT0kH3rEug.png" /></figure><h3>14. Interactive Scatter Plot</h3><p>A scatter plot can also be created using Plotly.</p><pre>fig = px.scatter(<br>    df,<br>    x=&quot;Quantity&quot;,<br>    y=&quot;Sales&quot;,<br>    color=&quot;Category&quot;,<br>    hover_data=[&quot;Product&quot;],<br>    title=&quot;Quantity vs Sales&quot;<br>)<br><br>fig.show()</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RKvKILQf9wLk2rj5mxbwDA.png" /></figure><p>One useful feature is the ability to add:</p><pre>color=&quot;Category&quot;</pre><p>This allows each category to be visually distinguished.</p><p>We can also add:</p><pre>hover_data=[&quot;Product&quot;]</pre><p>so that product information appears when the user hovers over a data point.</p><h3>15. Dashboard</h3><p>After understanding data visualization, the next step is combining multiple visualizations into an <strong>analytical dashboard</strong>.</p><p>For example:</p><pre>+---------------------------------------+<br>|           SALES DASHBOARD             |<br>+------------------+--------------------+<br>|                  |                    |<br>|   Sales Trend    |  Sales by Product  |<br>|                  |                    |<br>+------------------+--------------------+<br>|                                       |<br>|          Quantity vs Sales            |<br>|                                       |<br>+---------------------------------------+</pre><p>A dashboard can combine several components:</p><ul><li>KPIs</li><li>line charts</li><li>bar charts</li><li>scatter plots</li><li>histograms</li><li>filters</li><li>tables</li></ul><p>One Python ecosystem that can be used to build analytical applications is <strong>Dash</strong>, which integrates with Plotly.</p><p>A simple example (use dataframe that we used before):</p><pre>import pandas as pd<br>import numpy as np<br>from dash import Dash, dcc, html<br>import plotly.express as px<br><br>app = Dash(__name__)<br><br># Create dataset<br>data = {<br>    &quot;Date&quot;: [<br>        &quot;2026-01-01&quot;,<br>        &quot;2026-01-02&quot;,<br>        &quot;2026-01-03&quot;,<br>        &quot;2026-01-04&quot;,<br>        &quot;2026-01-05&quot;,<br>        &quot;2026-01-06&quot;,<br>        &quot;2026-01-07&quot;,<br>        &quot;2026-01-08&quot;,<br>        &quot;2026-01-09&quot;,<br>        &quot;2026-01-10&quot;<br>    ],<br>    &quot;Product&quot;: [<br>        &quot;Laptop&quot;,<br>        &quot;Mouse&quot;,<br>        &quot;Keyboard&quot;,<br>        &quot;Monitor&quot;,<br>        &quot;Laptop&quot;,<br>        &quot;Mouse&quot;,<br>        &quot;Keyboard&quot;,<br>        &quot;Monitor&quot;,<br>        &quot;Laptop&quot;,<br>        &quot;Mouse&quot;<br>    ],<br>    &quot;Category&quot;: [<br>        &quot;Computer&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Accessories&quot;<br>    ],<br>    &quot;Sales&quot;: [<br>        15000000,<br>        500000,<br>        800000,<br>        4000000,<br>        12000000,<br>        700000,<br>        900000,<br>        4500000,<br>        17000000,<br>        600000<br>    ],<br>    &quot;Quantity&quot;: [<br>        2,<br>        10,<br>        8,<br>        3,<br>        2,<br>        14,<br>        9,<br>        3,<br>        2,<br>        12<br>    ]<br>}<br><br>df = pd.DataFrame(data)<br><br># Convert Date to datetime<br>df[&quot;Date&quot;] = pd.to_datetime(df[&quot;Date&quot;])<br><br><br>fig = px.line(<br>    df,<br>    x=&quot;Date&quot;,<br>    y=&quot;Sales&quot;,<br>    title=&quot;Daily Sales&quot;<br>)<br><br>app.layout = html.Div(<br>    children=[<br>        html.H1(&quot;Sales Dashboard&quot;),<br><br>        dcc.Graph(<br>            figure=fig<br>        )<br>    ]<br>)<br><br>if __name__ == &quot;__main__&quot;:<br>    app.run(debug=True)</pre><p>Then run (example we save as app.py):</p><pre>python app.py</pre><p>The dashboard can be used as an interface for users to explore data interactively.</p><h3>16. Comparing Matplotlib, Seaborn, and Plotly</h3><p>These libraries have slightly different purposes.</p><p>LibraryFocusInteractiveMatplotlibFundamental &amp; custom visualizationNo by defaultSeabornStatistical visualizationNo by defaultPlotlyInteractive visualizationYesDashAnalytical dashboardsYes</p><p>A simple overview:</p><pre>|     Library     |                Focus               |     Interactive     |<br>|-----------------|------------------------------------|---------------------|<br>| Matplotlib      | Fundamental &amp; custom visualization | No by default       |<br>| Seaborn         | Statistical visualization          | No by default       |<br>| Plotly          | Interactive visualization          | Yes                 |<br>| Dash            | Analytical dashboards              | Yes                 |</pre><p>A simple overview:</p><pre>                 DATA<br>                   │<br>                   ▼<br>                Pandas<br>                   │<br>                   ▼<br>          ┌────────┼────────┐<br>          │        │        │<br>          ▼        ▼        ▼<br>      Matplotlib Seaborn   Plotly<br>          │        │        │<br>          │        │        ▼<br>          │        │       Dash<br>          │        │        │<br>          └────────┴────────┘<br>                   │<br>                   ▼<br>             Visualization</pre><h3>17. When Should You Use Each Chart?</h3><p>Choosing a chart should not be based solely on which chart looks attractive.</p><p>We should choose a chart based on the question we want to answer.</p><h3>Line Chart</h3><p>Use it to answer:</p><blockquote><em>How does a value change over time?</em></blockquote><p>Examples:</p><pre>Production vs Time<br>Sales vs Time<br>Temperature vs Time</pre><h3>Bar Chart</h3><p>Use it to answer:</p><blockquote><em>How do different categories compare?</em></blockquote><p>Examples:</p><pre>Sales by Product<br>Production by Area<br>Revenue by Department</pre><h3>Scatter Plot</h3><p>Use it to answer:</p><blockquote><em>Is there a relationship between two variables?</em></blockquote><p>Examples:</p><pre>Temperature vs Pressure<br>Quantity vs Sales<br>Age vs Income</pre><h3>Histogram</h3><p>Use it to answer:</p><blockquote><em>How is a variable distributed?</em></blockquote><p>Examples:</p><pre>Age Distribution<br>Salary Distribution<br>Sensor Distribution</pre><h3>Box Plot</h3><p>Use it to answer:</p><blockquote><em>How is data distributed and spread across different groups?</em></blockquote><p>Examples:</p><pre>Temperature by Machine<br>Salary by Department<br>Sales by Category</pre><h3>Heatmap</h3><p>Use it to answer:</p><blockquote><em>What patterns exist within a matrix of values?</em></blockquote><p>Examples:</p><pre>Correlation Matrix<br>Monthly Performance<br>Sensor Correlation</pre><h3>18. Complete Data Visualization Script</h3><p>If we combine the concepts discussed above, we can create a more structured script:</p><pre>&quot;&quot;&quot;<br>Python Fundamental #2<br>Data Visualization in Python<br><br>Libraries:<br>- Pandas<br>- NumPy<br>- Matplotlib<br>- Seaborn<br>- Plotly<br>&quot;&quot;&quot;<br><br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import plotly.express as px<br><br><br># ============================================<br># 1. CREATE DATASET<br># ============================================<br><br>data = {<br>    &quot;Date&quot;: pd.date_range(<br>        start=&quot;2026-01-01&quot;,<br>        periods=10,<br>        freq=&quot;D&quot;<br>    ),<br>    &quot;Product&quot;: [<br>        &quot;Laptop&quot;,<br>        &quot;Mouse&quot;,<br>        &quot;Keyboard&quot;,<br>        &quot;Monitor&quot;,<br>        &quot;Laptop&quot;,<br>        &quot;Mouse&quot;,<br>        &quot;Keyboard&quot;,<br>        &quot;Monitor&quot;,<br>        &quot;Laptop&quot;,<br>        &quot;Mouse&quot;<br>    ],<br>    &quot;Category&quot;: [<br>        &quot;Computer&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Accessories&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Computer&quot;,<br>        &quot;Accessories&quot;<br>    ],<br>    &quot;Sales&quot;: [<br>        15_000_000,<br>        500_000,<br>        800_000,<br>        4_000_000,<br>        12_000_000,<br>        700_000,<br>        900_000,<br>        4_500_000,<br>        17_000_000,<br>        600_000<br>    ],<br>    &quot;Quantity&quot;: [<br>        2,<br>        10,<br>        8,<br>        3,<br>        2,<br>        14,<br>        9,<br>        3,<br>        2,<br>        12<br>    ]<br>}<br><br>df = pd.DataFrame(data)<br><br><br># ============================================<br># 2. DATA INSPECTION<br># ============================================<br><br>print(&quot;Dataset:&quot;)<br>print(df)<br><br>print(&quot;\nDataset Information:&quot;)<br>print(df.info())<br><br>print(&quot;\nStatistical Summary:&quot;)<br>print(df.describe())<br><br><br># ============================================<br># 3. MATPLOTLIB - LINE CHART<br># ============================================<br><br>plt.figure(figsize=(10, 5))<br><br>plt.plot(<br>    df[&quot;Date&quot;],<br>    df[&quot;Sales&quot;],<br>    marker=&quot;o&quot;<br>)<br><br>plt.title(&quot;Daily Sales&quot;)<br>plt.xlabel(&quot;Date&quot;)<br>plt.ylabel(&quot;Sales&quot;)<br><br>plt.xticks(rotation=45)<br>plt.grid(True)<br><br>plt.tight_layout()<br>plt.show()<br><br><br># ============================================<br># 4. MATPLOTLIB - BAR CHART<br># ============================================<br><br>sales_by_product = (<br>    df.groupby(&quot;Product&quot;)[&quot;Sales&quot;]<br>    .sum()<br>    .sort_values(ascending=False)<br>)<br><br>plt.figure(figsize=(8, 5))<br><br>plt.bar(<br>    sales_by_product.index,<br>    sales_by_product.values<br>)<br><br>plt.title(&quot;Total Sales by Product&quot;)<br>plt.xlabel(&quot;Product&quot;)<br>plt.ylabel(&quot;Sales&quot;)<br><br>plt.xticks(rotation=45)<br>plt.tight_layout()<br><br>plt.show()<br><br><br># ============================================<br># 5. MATPLOTLIB - SCATTER PLOT<br># ============================================<br><br>plt.figure(figsize=(8, 5))<br><br>plt.scatter(<br>    df[&quot;Quantity&quot;],<br>    df[&quot;Sales&quot;]<br>)<br><br>plt.title(&quot;Quantity vs Sales&quot;)<br>plt.xlabel(&quot;Quantity&quot;)<br>plt.ylabel(&quot;Sales&quot;)<br><br>plt.grid(True)<br>plt.show()<br><br><br># ============================================<br># 6. SEABORN - BOX PLOT<br># ============================================<br><br>plt.figure(figsize=(8, 5))<br><br>sns.boxplot(<br>    data=df,<br>    x=&quot;Category&quot;,<br>    y=&quot;Sales&quot;<br>)<br><br>plt.title(&quot;Sales Distribution by Category&quot;)<br><br>plt.show()<br><br><br># ============================================<br># 7. SEABORN - HEATMAP<br># ============================================<br><br>numeric_df = df[<br>    [&quot;Sales&quot;, &quot;Quantity&quot;]<br>]<br><br>correlation = numeric_df.corr()<br><br>plt.figure(figsize=(7, 5))<br><br>sns.heatmap(<br>    correlation,<br>    annot=True,<br>    cmap=&quot;coolwarm&quot;,<br>    fmt=&quot;.2f&quot;<br>)<br><br>plt.title(&quot;Correlation Matrix&quot;)<br><br>plt.show()<br><br><br># ============================================<br># 8. PLOTLY - INTERACTIVE LINE CHART<br># ============================================<br><br>fig = px.line(<br>    df,<br>    x=&quot;Date&quot;,<br>    y=&quot;Sales&quot;,<br>    title=&quot;Interactive Daily Sales&quot;<br>)<br><br>fig.show()<br><br><br># ============================================<br># 9. PLOTLY - INTERACTIVE SCATTER<br># ============================================<br><br>fig = px.scatter(<br>    df,<br>    x=&quot;Quantity&quot;,<br>    y=&quot;Sales&quot;,<br>    color=&quot;Category&quot;,<br>    hover_data=[&quot;Product&quot;],<br>    title=&quot;Interactive Quantity vs Sales&quot;<br>)<br><br>fig.show()</pre><p>This script demonstrates a representative workflow:</p><pre>Raw Data<br>   │<br>   ▼<br>Pandas DataFrame<br>   │<br>   ▼<br>Data Inspection<br>   │<br>   ├───────────────┐<br>   ▼               ▼<br>Matplotlib       Seaborn<br>   │               │<br>   │               ├── Box Plot<br>   │               └── Heatmap<br>   │<br>   ├── Line<br>   ├── Bar<br>   └── Scatter<br>                   │<br>                   ▼<br>                 Plotly<br>                   │<br>                   ├── Interactive Line<br>                   └── Interactive Scatter</pre><h3>19. Data Visualization in Data Science</h3><p>Data visualization is not an isolated step.</p><p>In a Data Science workflow, visualization is usually an important part of <strong>Exploratory Data Analysis (EDA)</strong>.</p><p>A simplified workflow looks like this:</p><pre>                Raw Data<br>                   │<br>                   ▼<br>              Data Loading<br>                   │<br>                   ▼<br>             Data Cleaning<br>                   │<br>                   ▼<br>              Data Analysis<br>                   │<br>                   ▼<br>                   EDA<br>                   │<br>          ┌────────┴────────┐<br>          ▼                 ▼<br>    Statistics        Visualization<br>          │                 │<br>          └────────┬────────┘<br>                   ▼<br>             Feature Engineering<br>                   │<br>                   ▼<br>             Machine Learning<br>                   │<br>                   ▼<br>             Model Evaluation<br>                   │<br>                   ▼<br>              Visualization<br>                   │<br>                   ▼<br>               Reporting</pre><p>Therefore, visualization is not only used after the analysis is complete.</p><p>Visualization can be used <strong>before, during, and after the machine learning process</strong>.</p><h3>20. Example in Machine Learning</h3><p>Suppose we have machine data:</p><pre>Datetime<br>Temperature<br>Pressure<br>Vibration<br>RPM</pre><p>Before building an anomaly detection model, we can perform:</p><pre>sns.boxplot(<br>    data=df,<br>    x=&quot;Temperature&quot;<br>)</pre><p>to examine the temperature distribution.</p><p>Then:</p><pre>sns.heatmap(<br>    df.corr(),<br>    annot=True<br>)</pre><p>to examine relationships between sensors.</p><p>And:</p><pre>plt.plot(<br>    df[&quot;Datetime&quot;],<br>    df[&quot;Temperature&quot;]<br>)</pre><p>to examine sensor trends over time.</p><p>Visualization can therefore help us understand:</p><ul><li>trends</li><li>seasonality</li><li>outliers</li><li>correlations</li><li>distributions</li><li>anomalies</li><li>changes in machine behavior</li></ul><p>This is especially relevant when visualization is used as part of <strong>industrial data analytics, predictive maintenance, forecasting, and anomaly detection</strong>.</p><h3>21. My Opinion (About Visualization Libraries)</h3><h4>Why I Prefer Plotly for Data Visualization</h4><p>Among the data visualization libraries available in Python, Plotly is one of my personal favorites because of its powerful capabilities and interactive features.</p><p>While Matplotlib and Seaborn are excellent for static charts and statistical analysis, I often prefer Plotly when I need to explore data dynamically, build interactive visualizations, or present analytical results to other people.</p><p>What makes Plotly particularly powerful is that it allows us to create sophisticated, interactive charts with relatively little code. Many interactive features are available by default, without requiring us to implement them manually.</p><p>Here are some of the reasons why I find Plotly so useful.</p><h4>21.1. Interactive by Default</h4><p>One of Plotly’s biggest advantages is that its charts are interactive by default.</p><p>When viewing a Plotly chart, users can:</p><ul><li>Hover over data points to inspect detailed information.</li><li>Zoom in and out to examine specific regions of a chart.</li><li>Pan across the visualization.</li><li>Toggle legends to show or hide individual data series.</li><li>Select data points or regions for further exploration.</li></ul><p>These capabilities make data analysis more flexible because users can investigate the data themselves instead of relying entirely on predefined static charts.</p><p>For example :</p><pre>import plotly.express as px<br><br>fig = px.line(<br>    df,<br>    x=&quot;Date&quot;,<br>    y=&quot;Sales&quot;,<br>    title=&quot;Daily Sales Trend&quot;<br>)<br><br>fig.show()</pre><p>Interactive output can be like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tUxGAoiH2PASUose_uB-mA.png" /></figure><p>With just a few lines of code, we can create an interactive line chart that allows users to inspect individual data points, zoom into specific dates, and explore changes in sales over time.</p><h3>21.2. Supports More Than 70 Chart Types</h3><p>Plotly is not limited to basic line charts and bar charts. Its ecosystem supports more than 70 chart and trace types across statistical, scientific, financial, geographic, and three-dimensional visualization.</p><p>Some examples include:</p><ul><li>Time Series and Line Charts</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*bjrRtL2SEdf407hd.png" /></figure><ul><li>Scatter Plots and Bubble Charts</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/0*FL9oS8QihLeezhyi" /></figure><ul><li>Heatmaps and Statistical Charts</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*hqVSedIYZDo471B5" /></figure><ul><li>3D Visualizations</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/0*n_RGaWPwSE4LfKmk" /></figure><ul><li>Geographic Visualizations</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RPsk0F-wzVPFtoh1" /></figure><p>Having these visualization options within one ecosystem makes Plotly a versatile tool for different data science use cases.</p><h3>21.3. Advanced Customization</h3><p>Another reason I like Plotly is its flexibility in customizing charts.</p><p>We can control chart titles, axis labels, colors, templates, annotations, hover information, legends, and layout. We can also combine multiple chart types into a single figure.</p><p>For example, we can create a scatter plot that uses different colors for different product categories and displays additional information when the user hovers over a data point.</p><p>For example:</p><pre>import plotly.express as px<br><br>fig = px.scatter(<br>    df,<br>    x=&quot;Quantity&quot;,<br>    y=&quot;Sales&quot;,<br>    color=&quot;Category&quot;,<br>    hover_data=[&quot;Product&quot;],<br>    title=&quot;Sales vs Quantity by Category&quot;,<br>    template=&quot;plotly_white&quot;<br>)<br><br>fig.update_layout(<br>    title_x=0.5,<br>    xaxis_title=&quot;Quantity Sold&quot;,<br>    yaxis_title=&quot;Sales Revenue&quot;<br>)<br><br>fig.show()</pre><p>Output can be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OuuowdQ03RhM3BNesnJ6mw.png" /></figure><p>In this example:</p><ul><li>color=&quot;Category&quot; distinguishes different categories.</li><li>hover_data=[&quot;Product&quot;] displays product information on hover.</li><li>template=&quot;plotly_white&quot; applies a clean visual theme.</li><li>update_layout() allows us to customize the chart layout.</li></ul><p>Plotly also provides a lower-level API called plotly.graph_objects, which offers greater control over individual traces and figure properties when building more complex visualizations.</p><h3>21.4. Handles Large Datasets with WebGL Support</h3><p>Plotly also supports WebGL-based rendering for certain chart types. This can improve rendering performance when visualizing large numbers of data points.</p><p>For example, in industrial data analytics, a single sensor may generate thousands or millions of measurements over time. Rendering every measurement as an ordinary SVG element can become expensive, depending on the browser and visualization.</p><p>For suitable use cases, we can use Scattergl to render scatter plots through WebGL.</p><p>For example:</p><pre>import plotly.graph_objects as go<br><br>fig = go.Figure(<br>    data=[<br>        go.Scattergl(<br>            x=df[&quot;Quantity&quot;],<br>            y=df[&quot;Sales&quot;],<br>            mode=&quot;markers&quot;,<br>            name=&quot;Sales Data&quot;<br>        )<br>    ]<br>)<br><br>fig.update_layout(<br>    title=&quot;Sales Data Exploration&quot;,<br>    xaxis_title=&quot;Quantity&quot;,<br>    yaxis_title=&quot;Sales&quot;<br>)<br><br>fig.show()</pre><p>Scattergl uses WebGL rendering rather than the standard SVG-based scatter trace. This can help with large datasets, although actual performance depends on data size, browser capabilities, available memory, and chart complexity.</p><p>For very large datasets, we should still consider aggregation, downsampling, and data filtering rather than assuming that every point must always be rendered.</p><h3>21.5. Easy Integration with Dash for Building Dashboards</h3><p>One of Plotly’s most valuable capabilities is its integration with Dash, an open-source Python framework for building interactive analytical web applications.</p><p>This means we can progress from creating a single chart to building a complete dashboard without having to implement the entire frontend in JavaScript.</p><p>For example, a dashboard could contain:</p><ul><li>KPI cards displaying key metrics.</li><li>Interactive line charts showing historical trends.</li><li>Dropdown filters for selecting machines or product categories.</li><li>Date-range controls for examining specific periods.</li><li>Scatter plots for investigating relationships between variables.</li><li>Tables displaying detailed analytical results.</li></ul><p>A simple example:</p><pre>import pandas as pd<br>import plotly.express as px<br><br>from dash import Dash, dcc, html<br><br># Create the application<br>app = Dash(__name__)<br><br># Create an interactive chart<br>fig = px.line(<br>    df,<br>    x=&quot;Date&quot;,<br>    y=&quot;Sales&quot;,<br>    title=&quot;Daily Sales Trend&quot;<br>)<br><br># Define the dashboard layout<br>app.layout = html.Div(<br>    [<br>        html.H1(&quot;Sales Analytics Dashboard&quot;),<br><br>        dcc.Graph(<br>            id=&quot;sales-trend&quot;,<br>            figure=fig<br>        )<br>    ]<br>)<br><br>if __name__ == &quot;__main__&quot;:<br>    app.run(debug=True)</pre><p>Output can be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MecRpP-Q4kMpfBjMijROzA.png" /></figure><p>This example creates a basic web dashboard containing an interactive Plotly chart. More advanced dashboards can include callbacks, filters, multiple pages, and connections to external data sources.</p><h3>21.6. Suitable for Industrial Data Analytics and Machine Learning</h3><p>For my own work in Data Science and Machine Learning, I find Plotly particularly useful because industrial and machine learning datasets often require more than a static visualization.</p><p>Consider a gas turbine monitoring application with the following data:</p><pre>Datetime<br>Temperature<br>Pressure<br>Vibration<br>RPM<br>Actual_Value<br>Predicted_Value<br>Anomaly_Status</pre><p>Using Plotly, we can create an interactive chart to compare actual sensor measurements against model predictions.</p><p>For example:</p><pre>import plotly.graph_objects as go<br><br>fig = go.Figure()<br><br># Actual sensor measurements<br>fig.add_trace(<br>    go.Scatter(<br>        x=df[&quot;Datetime&quot;],<br>        y=df[&quot;Actual_Value&quot;],<br>        mode=&quot;lines&quot;,<br>        name=&quot;Actual Value&quot;<br>    )<br>)<br><br># Model predictions<br>fig.add_trace(<br>    go.Scatter(<br>        x=df[&quot;Datetime&quot;],<br>        y=df[&quot;Predicted_Value&quot;],<br>        mode=&quot;lines&quot;,<br>        name=&quot;Predicted Value&quot;<br>    )<br>)<br><br>fig.update_layout(<br>    title=&quot;Actual vs Predicted Sensor Values&quot;,<br>    xaxis_title=&quot;Datetime&quot;,<br>    yaxis_title=&quot;Sensor Value&quot;,<br>    hovermode=&quot;x unified&quot;,<br>    template=&quot;plotly_white&quot;<br>)<br><br>fig.show()</pre><p>This visualization allows us to compare actual measurements with predicted values over time. We can inspect individual timestamps, zoom into specific periods, and investigate where the model’s predictions differ from the actual measurements.</p><p>We can extend the same concept to visualize anomaly detection results, forecasting performance, and equipment behavior.</p><p>For example, we can add a third trace to represent an upper operating limit or highlight detected anomalies. These additions can make it easier for engineers and analysts to investigate potentially abnormal operating conditions.</p><p>It is important to remember that a visualization can help identify suspicious patterns, but it does not independently confirm that equipment is faulty. Anomalies should be evaluated against operating conditions, sensor quality, engineering limits, and other relevant evidence.</p><h3>22. Conclusion</h3><p>Data visualization is an essential part of the data science workflow. It helps us understand data distributions, identify patterns and trends, detect outliers, and communicate insights more effectively.</p><p>Python provides several powerful visualization libraries, each with its own strengths. Matplotlib offers fundamental plotting capabilities and detailed customization, while Seaborn simplifies statistical visualization and produces informative charts with less code. Meanwhile, Plotly provides interactive and highly customizable visualizations that allow us to explore data more dynamically.</p><p>Personally, I often prefer Plotly because of its powerful interactive features, flexibility, and ability to create engaging visualizations. Features such as hover information, zooming, filtering through legends, and interactive exploration make it particularly useful when analyzing complex datasets. Plotly can also be integrated with Dash to build interactive dashboards, making it suitable for more advanced data analytics and monitoring applications.</p><p>In practical data science projects, visualization is not just about making charts look attractive. It is about transforming raw data into meaningful information that supports better decisions. For example, in time-series forecasting and industrial anomaly detection, visualizing actual values, model predictions, and anomaly thresholds can help us understand model behavior and investigate potential abnormal conditions.</p><p>The key is to choose the right visualization technique for the problem, the audience, and the analytical objective. A simple line chart may be sufficient to communicate a trend, while an interactive dashboard may be more appropriate for exploring large datasets or monitoring multiple variables.</p><p>Ultimately, mastering data visualization will help you become a more effective data scientist by connecting data analysis, machine learning, and business understanding.</p><p>In the next article, we will continue our Python for Data Science journey by exploring Exploratory Data Analysis (EDA), where we will learn how to inspect, clean, summarize, and discover insights from datasets before building machine learning models.</p><p>Stay tuned, and keep learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b38cac206dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/python-fundamental-2-data-visualization-in-python-2b38cac206dd">Python Fundamental #2 — Data Visualization in Python</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Fundamental #1 — Pandas and Numpy]]></title>
            <link>https://medium.com/data-science-collective/python-fundamental-1-pandas-and-numpy-2ce5265b445d?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ce5265b445d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Thu, 30 Apr 2026 07:57:57 GMT</pubDate>
            <atom:updated>2026-04-30T07:57:57.420Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lOjIOV-VTH-9zHP_.jpg" /></figure><p>Python is widely used for data science, machine learning, and scientific computing. Among them, <strong>NumPy</strong> and <strong>Pandas</strong> are essential tools for handling numerical and structured data efficiently.</p><ul><li><strong>NumPy</strong> (Numerical Python) provides high-performance support for multi-dimensional arrays and mathematical operations, making it the foundation for numerical computing in Python.</li><li><strong>Pandas</strong>, built on top of NumPy, offers flexible data structures like <em>DataFrames </em>and <em>Series</em>, making data manipulation and analysis easier.</li></ul><p>Both libraries are crucial for data processing, analysis, and visualization, commonly used in fields like finance, artificial intelligence, and scientific research.</p><p><strong>1.</strong> <strong>Numpy</strong></p><p><em>NumPy (Numerical Python)</em> is a powerful library for numerical computing in Python. It provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these data structures efficiently. NumPy is widely used in scientific computing, data analysis, and machine learning due to its speed and efficiency.</p><p><strong>Key Features of NumPy:</strong></p><p>· Supports multi-dimensional arrays (ndarrays).</p><p>· Provides mathematical and statistical functions.</p><p>· Enables fast array operations using vectorization.</p><p>· Supports broadcasting for arithmetic operations on arrays of different shapes.</p><p>· Essential for data preprocessing and numerical analysis in AI and machine learning.</p><p>First of all to work with <em>Numpy </em>in python script, we must to import the library :</p><pre>import numpy as np</pre><p>Then, we can use <em>np </em>to declare the <em>numpy.</em></p><ol><li><em>Creating an Array</em></li></ol><p>In <strong>Python</strong>, an <strong>array</strong> is a data structure used to store multiple values in a single variable.</p><pre>arr = np.array([1, 2, 3, 4, 5])<br>print(&quot;\nNumPy Array:\n&quot;, arr)</pre><p>Result :</p><pre>NumPy Array:<br> [1 2 3 4 5]</pre><ul><li>np.array([...]) creates a <strong>NumPy array</strong> from a Python list [1, 2, 3, 4, 5].</li><li>Unlike a Python list, a NumPy array supports <strong>vectorized operations</strong> (e.g., mathematical operations on the entire array at once).</li></ul><p>2. <em>Multiply the Array (Mathematical Operations)</em></p><pre>arr2 = arr * 2<br>print(&quot;\nArray after multiplying by 2:\n&quot;, arr2)</pre><p>Result :</p><pre>Array after multiplying by 2:<br> [ 2  4  6  8 10]</pre><p>In conclusion, arr2 is a new array that contains the result of multiplying each element of arr by 2. This kind of operation is called an element-wise operation because the calculation is performed on each individual element of the array.</p><p>3. <em>Matrices and Linear Operations</em></p><p>In <strong>NumPy</strong>, a matrix is represented using a <strong>2D array (nested lists)</strong>. NumPy provides powerful tools for matrix operations such as <strong>addition, multiplication, transposition, and inversion</strong>.</p><pre>matrix = np.array([[3, 4], [5, 6]])<br>print(&quot;\nMatrix:\n&quot;, matrix)</pre><p>Result :</p><pre>Matrix:<br> [[3 4]<br> [5 6]]</pre><ul><li><em>Transpose Matrix</em></li></ul><pre># Transposing a Matrix<br>print(&quot;\nTransposed Matrix:\n&quot;, matrix.T)</pre><p>Result :</p><pre>Transposed Matrix:<br> [[3 5]<br> [4 6]]</pre><p>4. <em>Another way to create a NumPy array</em></p><p>In addition to <em>np.array()</em>, which accepts lists, NumPy provides various functions for creating arrays with specific patterns. This is very useful for initialization and testing.</p><p>4.1 <em>np.arange() — Array with Range</em></p><p><em>np.arange(start, stop, step)</em> is similar to the <em>range()</em> function in Python, but returns a NumPy array. It is very useful for creating arrays with sequential values.</p><pre># np.arange(start, end, step) - similar with range() but returns array<br># start: starting value (inclusive)<br># end: ending value (exclusive)<br># step: increment<br><br>arr_arange = np.arange(0, 10, 2)  # 0, 2, 4, 6, 8<br>print(&quot;arange(0, 10, 2):&quot;, arr_arange)<br><br># another case<br>arr_arange2 = np.arange(5)  # 0 sampai 4<br>print(&quot;arange(5):&quot;, arr_arange2)<br><br>arr_arange3 = np.arange(1, 8, 1.5)  # with decimal step<br>print(&quot;arange(1, 8, 1.5):&quot;, arr_arange3)</pre><p>Result:</p><pre>arange(0, 10, 2): [0 2 4 6 8]<br>arange(5): [0 1 2 3 4]<br>arange(1, 8, 1.5): [1.  2.5 4.  5.5 7. ]</pre><p>4.2 <em>np.zeros() and np.ones()</em></p><p>Create an array containing all zeros or all ones. This is very useful for initializing an array before further operations, or for creating a mask or binary value.</p><p>Example 1:</p><pre># np.zeros(shape) - an array filled with zeros<br># The shape can be 1D (size) or 2D (rows, columns)<br><br>arr_zeros = np.zeros(5)  # array 1D with 5 zero elements<br>print(&quot;zeros(5):&quot;, arr_zeros)<br><br>arr_zeros_2d = np.zeros((3, 3))  # 3x3 zero matrix<br>print(&quot;zeros((3, 3)):&quot;)<br>print(arr_zeros_2d)</pre><p>Result:</p><pre>zeros(5): [0. 0. 0. 0. 0.]<br>zeros((3, 3)):<br>[[0. 0. 0.]<br> [0. 0. 0.]<br> [0. 0. 0.]]</pre><p>So, np.zeros(5) makes a one-row array with five zeros, while np.zeros((3, 3)) makes a 3 × 3 matrix where every value is zero.</p><p>The zeros appear as 0. instead of 0 because NumPy creates the values as floating-point numbers by default.</p><p>Example 2:</p><pre># np.ones(shape) - an array filled with ones<br># Same as zeros but filled with 1s<br><br>arr_ones = np.ones((2, 4))  # a 2x4 matrix containing 1<br>print(&quot;ones((2, 4)):&quot;)<br>print(arr_ones)</pre><p>Result:</p><pre>ones((2, 4)):<br>[[1. 1. 1. 1.]<br> [1. 1. 1. 1.]]</pre><p>This means NumPy created a 2 × 4 matrix, and every element inside the matrix is 1.</p><p>The values appear as 1. instead of 1 because NumPy creates the array using floating-point numbers by default.</p><p>4.3 <em>np.linspace() — Array with equal spacing</em></p><p><em>np.linspace(start, end, num)</em> Creates an array with a specified number of elements and equal spacing between them. Unlike `arange`, here we specify the number of elements, not the step size.</p><pre># np.linspace(start, end, num) - an array with num equally spaced elements<br># start: starting value<br># end: ending value<br># num: number of elements<br><br>arr_linspace = np.linspace(0, 1, 5)  # 5 values from 0 to 1<br>print(&quot;linspace(0, 1, 5):&quot;, arr_linspace)<br><br>arr_linspace2 = np.linspace(0, 100, 11)  # 11 values from 0 to 100<br>print(&quot;linspace(0, 100, 11):&quot;, arr_linspace2)</pre><p>Result:</p><pre>linspace(0, 1, 5): [0.   0.25 0.5  0.75 1.  ]<br>linspace(0, 100, 11): [  0.  10.  20.  30.  40.  50.  60.  70.  80.  90. 100.]</pre><p>This creates 11 values from 0 to 100. The spacing between each value is 10.</p><p>So, np.linspace() is useful when you want to generate a specific number of evenly spaced values between two numbers.</p><p>4.4 <em>Random Number Array</em></p><p>NumPy has a <em>np.random</em> module for creating arrays with random values. This is very useful for simulations, initializing neural network weights, and data sampling.</p><p>Example 1:</p><pre># np.random.rand(shape) - random values with uniform dist [0, 1)<br># Each value have same probability<br>arr_random = np.random.rand(3, 3)<br>print(&quot;random.rand(3, 3) - uniform [0, 1):&quot;)<br>print(arr_random)</pre><p>Result :</p><pre>random.rand(3, 3) - uniform [0, 1):<br>[[0.0466752  0.07308115 0.73814515]<br> [0.24918135 0.34692576 0.27054042]<br> [0.40353355 0.5951376  0.86294036]]</pre><p>Each number is between <strong>0 and 1</strong>, including 0 but not including 1. That is why the output shows decimal values such as 0.0466752, 0.73814515, and 0.86294036.</p><p>The term <strong>uniform [0, 1)</strong> means every number in the range from 0 up to 1 has an equal chance of being selected. Since the values are random, the result may be different every time the code is run.</p><p>Example 2 :</p><pre># np.random.randint(low, high, size) - random integer values<br># low: lower bound (inclusive)<br># high: upper bound (exclusive)<br># size: array shape<br><br>arr_randint = np.random.randint(1, 10, (3, 3))  # integer 1-9<br>print(&quot;random.randint(1, 10, (3, 3)):&quot;)<br>print(arr_randint)</pre><p>Result :</p><pre>random.randint(1, 10, (3, 3)):<br>[[9 4 2]<br> [2 3 5]<br> [7 8 7]]</pre><p>The numbers are randomly chosen from <strong>1 to 9</strong>.</p><p>The first value, 1, is included, but the second value, 10, is not included. That is why the possible values are:</p><p><em>1, 2, 3, 4, 5, 6, 7, 8, 9</em></p><p>Because the values are random, the output may be different every time the code is run.</p><p>Example 3 :</p><pre># np.random.randn(shape) - Gaussian Distribution<br># Mean = 0, Std = 1<br><br>arr_normal = np.random.randn(5)<br>print(&quot;random.randn(5) - Normal Distribution:&quot;)<br>print(arr_normal)<br><br># The results will vary each time you run it!</pre><p>Result :</p><pre>random.randn(5) - Normal Distribution:<br>[ 0.64257665  0.25134601 -1.50083963  0.81431326 -0.35981619]</pre><p>This means NumPy generated 5 random decimal numbers. Unlike np.random.rand(), which only gives values between 0 and 1, np.random.randn() can produce <strong>positive and negative values</strong>.</p><p>The values come from a normal distribution with:</p><p><em>mean = 0<br>standard deviation = 1</em></p><p>So most numbers will usually be close to 0, but sometimes the result can be farther away, such as -1.50083963.</p><p>Because the values are random, the result may be different every time the code is run.</p><p>Example 4 :</p><pre># np.random.rand(shape) - a random value from the uniform distribution (0, 1)<br># Each value has an equal probability<br><br>arr_random = np.random.rand(3, 3)<br>print(&quot;random.rand(3, 3) - uniform [0, 1):&quot;)<br>print(arr_random)</pre><p>Result :</p><pre>random.rand(3, 3) - uniform [0, 1):<br>[[0.0466752  0.07308115 0.73814515]<br> [0.24918135 0.34692576 0.27054042]<br> [0.40353355 0.5951376  0.86294036]]</pre><p>This means NumPy generated <strong>9 random numbers</strong> arranged in <strong>3 rows and 3 columns</strong>.</p><p>Each value is between 0 and 1. The number 0 can be included, but 1 is not included. That is why the notation is written as [0, 1).</p><p>Because the numbers are randomly generated, the output can be different each time the program is run.</p><p>4.5 <em>NumPy Array Attributes</em></p><p>Every NumPy array has several important attributes that provide information about the array. Understanding these attributes is essential for debugging and understanding the data structure.</p><p>Example 1 :</p><pre># Creating attribute array<br>arr = np.array([[1, 2, 3], [4, 5, 6]])<br><br>print(&quot;Array:&quot;)<br>print(arr)</pre><p>Result :</p><pre>Array: <br>[[1 2 3]<br> [4 5 6]]</pre><p>So, the array shape is <strong>2 × 3</strong>, meaning it has <strong>2 rows</strong> and <strong>3 columns</strong>. This type of array is often called a <strong>matrix</strong>.</p><p>Example 2 :</p><pre># shape: a tuple specifying the array&#39;s dimensions (rows, columns)<br>print(&quot;Shape:&quot;, arr.shape)<br>print(&quot;Number of rows:&quot;, arr.shape[0], &quot;, Number of columns:&quot;, arr.shape[1])</pre><p>Result :</p><pre>Shape: (2, 3)<br>Number of rows: 2 , Number of columns: 3</pre><p>So, the shape (2, 3) means the array is a <strong>2 × 3 matrix</strong>. Therefore, this array has 2 rows and 3 columns.</p><p>Example 3 :</p><pre># dtype: the data type of the elements in the array<br>print(&quot;\nDtype:&quot;, arr.dtype)<br>print(&quot;Integer:&quot;, arr.dtype, &quot;is integer 64-bit&quot;)</pre><p>Result :</p><pre>Dtype: int32<br>Integer: int32 is integer 64-bit</pre><p>This means the elements inside the array are integers, and NumPy identifies their type as int32. However, there is a small mistake in the printed sentence. The output says: <em>int32 is integer 64-bit. </em>But int32 actually means <strong>32-bit integer</strong>, not 64-bit integer. So the correct explanation should be:<em> Integer: int32 is integer 32-bit. </em>In conclusion, arr.dtype tells us the data type of the array elements. In this example, the array contains integer values with the data type int32.</p><p>Example 4 :</p><pre># ndim: number of array dimensions<br>print(&quot;\nNdim (number of dimensions):&quot;, arr.ndim)</pre><p>Result :</p><pre>Ndim (number of dimensions): 2</pre><p>So, arr.ndim = 2 means the array is a <strong>2D array</strong>, or a <strong>matrix</strong>.</p><p>Example 5:</p><pre># size: total number of elements<br>print(&quot;\nSize (total elements):&quot;, arr.size)</pre><p>Result :</p><pre>Size (total elements): 6</pre><p><strong>5. <em>Indexing and Slicing Array</em></strong></p><p>How to access elements in a NumPy array using a Python list, but with additional features for multidimensional arrays.</p><p>5.1 <em>Indexing Array 1D</em></p><p>Indexing in a 1D array similar to Python lists. Indexing starts from 0.</p><pre># Indexing a 1D array<br>arr = np.array([10, 20, 30, 40, 50])<br>print(&quot;Array:&quot;, arr)<br>print()<br><br># Positive indices (starting from 0)<br>print(&quot;arr[0]:&quot;, arr[0], &quot;(first element)&quot;)<br>print(&quot;arr[1]:&quot;, arr[1], &quot;(second element)&quot;)<br>print(&quot;arr[2]:&quot;, arr[2], &quot;(third element)&quot;)<br><br># Negative indices (starting from the end)<br>print(&quot;\narr[-1]:&quot;, arr[-1], &quot;(last element)&quot;)<br>print(&quot;arr[-2]:&quot;, arr[-2], &quot;(second-to-last element)&quot;)</pre><p>Result:</p><pre>Array: [10 20 30 40 50]<br><br>arr[0]: 10 (first element)<br>arr[1]: 20 (second element)<br>arr[2]: 30 (third element)<br><br>arr[-1]: 50 (last element)<br>arr[-2]: 40 (second-to-last element)</pre><p>This result shows indexing in a 1D NumPy array: for the array [10 20 30 40 50], positive indices start from the beginning, so arr[0] = 10, arr[1] = 20, and arr[2] = 30, while negative indices start from the end, so arr[-1] = 50 is the last element and arr[-2] = 40 is the second-to-last element.</p><p>5.2 <em>Slicing Array 1D</em></p><p>Slicing is a way to access a subrange of elements. The format is <em>array[start:end:step]</em>.</p><ul><li><strong><em>start </em></strong><em>: starting index (inclusive)</em></li><li><strong><em>end </em></strong><em>: ending index (exclusive)</em></li><li><strong><em>step </em></strong><em>: the interval between indices</em></li></ul><pre># Slicing a 1D array<br>arr = np.array([10, 20, 30, 40, 50, 60])<br>print(&quot;Array:&quot;, arr)<br>print()<br><br># arr[start:end] - from start to end-1<br>print(&quot;arr[1:4]:&quot;, arr[1:4], &quot;(index 1, 2, 3)&quot;)<br><br># arr[:end] - from the beginning to end-1<br>print(&quot;arr[:3]:&quot;, arr[:3], &quot;(index 0, 1, 2)&quot;)<br><br># arr[start:] - from start to the end<br>print(&quot;arr[3:]:&quot;, arr[3:], &quot;(index 3, 4, 5)&quot;)<br><br># arr[::step] - with a specific step<br>print(&quot;arr[::2]:&quot;, arr[::2], &quot;(all elements, step 2)&quot;)<br>print(&quot;arr[1::2]:&quot;, arr[1::2], &quot;(from index 1, step 2)&quot;)</pre><p>Result:</p><pre>Array: [10 20 30 40 50 60]<br><br>arr[1:4]: [20 30 40] (index 1, 2, 3)<br>arr[:3]: [10 20 30] (index 0, 1, 2)<br>arr[3:]: [40 50 60] (index 3, 4, 5)<br>arr[::2]: [10 30 50] (all elements, step 2)<br>arr[1::2]: [20 40 60] (from index 1, step 2)</pre><p>5.3 <em>Indexing Array 2D</em></p><p>For a 2D array, we use two indices: <em>array[row, column]</em>.</p><pre># Indexing a 2D array<br>matrix = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])<br>print(&quot;3x3 matrix:&quot;)<br>print(matrix)<br>print()<br><br># Accesing a single element<br>print(&quot;matrix[0, 0]:&quot;, matrix[0, 0], &quot;(row 0, column 0)&quot;)<br>print(&quot;matrix[1, 2]:&quot;, matrix[1, 2], &quot;(row 1, column 2)&quot;)<br>print(&quot;matrix[2, 0]:&quot;, matrix[2, 0], &quot;(row 2, column 0)&quot;)<br><br># Access an entire row atau column<br>print(&quot;\nmatrix[1]:&quot;, matrix[1], &quot;(row index 1)&quot;)<br>print(&quot;matrix[:, 2]:&quot;, matrix[:, 2], &quot;(column index 2)&quot;)</pre><p>Result:</p><pre>3x3 matrix:<br>[[1 2 3]<br> [4 5 6]<br> [7 8 9]]<br><br>matrix[0, 0]: 1 (row 0, column 0)<br>matrix[1, 2]: 6 (row 1, column 2)<br>matrix[2, 0]: 7 (row 2, column 0)<br><br>matrix[1]: [4 5 6] (row index 1)<br>matrix[:, 2]: [3 6 9] (column index 2)</pre><p>5.4 <em>Fancy Indexing dan Boolean Indexing</em></p><p>NumPy supports indexing using boolean arrays or index lists.</p><pre># Fancy Indexing - using an index array<br>arr = np.array([10, 20, 30, 40, 50])<br>indices = [0, 2, 4]  # want to retrieve indices 0, 2, 4<br>print(&quot;Array:&quot;, arr)<br>print(&quot;Retrieve indices [0, 2, 4]:&quot;, arr[indices])<br><br># Boolean Indexing - filtering with conditions<br>arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])<br>print(&quot;\nArray:&quot;, arr)<br># Create a boolean mask<br>mask = arr &gt; 5  # True for elements &gt; 5<br>print(&quot;Mask (arr &gt; 5):&quot;, mask)<br># Use the mask for filtering<br>print(&quot;arr[mask] (elemen &gt; 5):&quot;, arr[mask])</pre><p>Result:</p><pre>Array: [10 20 30 40 50]<br>Retrieve indices [0, 2, 4]: [10 30 50]<br><br>Array: [ 1  2  3  4  5  6  7  8  9 10]<br>Mask (arr &gt; 5): [False False False False False  True  True  True  True  True]<br>arr[mask] (elemen &gt; 5): [ 6  7  8  9 10]</pre><p>This result shows two NumPy indexing methods: <strong>fancy indexing</strong>, where arr[[0, 2, 4]] retrieves the elements at indices 0, 2, and 4 from [10 20 30 40 50], giving [10 30 50]; and <strong>boolean indexing</strong>, where the condition arr &gt; 5 creates a mask that is True only for values greater than 5, so arr[mask] filters the array [1 2 3 4 5 6 7 8 9 10] and returns [6 7 8 9 10].</p><p><strong>5. Reshaping Array</strong></p><p>Reshaping is the process of changing the shape of an array without altering its data. It is very useful for adjusting the data format for machine learning models.</p><p>Example 1:</p><pre># Reshape array - change the shape without altering the data<br>arr = np.arange(1, 13)  # 1 to 11<br>print(&quot;Original array:&quot;, arr)<br>print(&quot;Shape:&quot;, arr.shape)<br><br># Reshape to a 3x4 matrix<br>matrix_reshape = arr.reshape(3, 4)<br>print(&quot;\nReshape (3, 4):&quot;)<br>print(matrix_reshape)<br><br># Reshape to a 2x6 matrix<br>matrix_reshape2 = arr.reshape(2, 6)<br>print(&quot;Reshape (2, 6):&quot;)<br>print(matrix_reshape2)</pre><p>Result:</p><pre>Original array: [ 1  2  3  4  5  6  7  8  9 10 11 12]<br>Shape: (12,)<br><br>Reshape (3, 4):<br>[[ 1  2  3  4]<br> [ 5  6  7  8]<br> [ 9 10 11 12]]<br>Reshape (2, 6):<br>[[ 1  2  3  4  5  6]<br> [ 7  8  9 10 11 12]]</pre><p>Example 2:</p><pre># Flatten - convert a matrix into a 1D array<br>matrix = np.array([[1, 2, 3], [4, 5, 6]])<br>print(&quot;Matrix:&quot;)<br>print(matrix)<br><br>flattened = matrix.flatten()<br>print(&quot;Flattened:&quot;, flattened)<br><br># ravel - another method (more efficient for viewing)<br>raveled = matrix.ravel()<br>print(&quot;Raveled:&quot;, raveled)</pre><p>Result:</p><pre>Matrix:<br>[[1 2 3]<br> [4 5 6]]<br>Flattened: [1 2 3 4 5 6]<br>Raveled: [1 2 3 4 5 6]</pre><p><strong>6. Mathematical Operations and Statistics</strong></p><p>NumPy provides a wide range of mathematical and statistical functions that can be applied to an entire array at once.</p><p>6.1 <em>Mathematical Function</em></p><p>NumPy provides mathematical functions such as sqrt, log, exp, sin, cos, and so on.</p><pre># NumPy Math Functions<br>arr = np.array([1, 4, 9, 16, 25])<br>print(&quot;Array:&quot;, arr)<br>print()<br><br># Square Root<br>print(&quot;np.sqrt (square root):&quot;, np.sqrt(arr))<br><br># Square<br>print(&quot;np.square (square):&quot;, np.square(arr))<br><br># Exponential (e^x)<br>print(&quot;np.exp (e^x):&quot;, np.exp(arr))<br><br># Logarithm<br>print(&quot;np.log (natural logarithm):&quot;, np.log(arr))<br>print(&quot;np.log10 (base 10 logarithm):&quot;, np.log10(arr))<br><br># Trigonometry<br>arr_rad = np.array([0, np.pi/2, np.pi])<br>print(&quot;\narr_rad:&quot;, arr_rad)<br>print(&quot;np.sin:&quot;, np.sin(arr_rad))</pre><p>Result:</p><pre>Array: [ 1  4  9 16 25]<br><br>np.sqrt (square root): [1. 2. 3. 4. 5.]<br>np.square (square): [  1  16  81 256 625]<br>np.exp (e^x): [2.71828183e+00 5.45981500e+01 8.10308393e+03 8.88611052e+06<br> 7.20048993e+10]<br>np.log (natural logarithm): [0.1.38629436 2.19722458 2.77258872 3.21887582]<br>np.log10 (base 10 logarithm): [0.0.60205999 0.95424251 1.20411998 1.39794001]<br><br>arr_rad: [0.1.57079633 3.14159265]<br>np.sin: [0.0000000e+00 1.0000000e+00 1.2246468e-16]</pre><p>6.2 <em>Statistics Function</em></p><p>NumPy provides basic statistical functions such as mean, median, std, and so on.</p><pre># Basic Statistics<br>arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])<br>print(&quot;Array:&quot;, arr)<br>print()<br><br># Mean - average<br>print(&quot;np.mean (rata-rata):&quot;, np.mean(arr))<br><br># Median<br>print(&quot;np.median (median):&quot;, np.median(arr))<br><br># Std - standard deviation<br>print(&quot;np.std (standard deviation):&quot;, np.std(arr))<br><br># Variance<br>print(&quot;np.var (variance):&quot;, np.var(arr))<br><br># Sum - total sum of elements<br>print(&quot;np.sum (total sum):&quot;, np.sum(arr))<br><br># Min/Max - minimum/maximum values<br>print(&quot;np.min (minimum):&quot;, np.min(arr))<br>print(&quot;np.max (maximum):&quot;, np.max(arr))<br><br># Percentile<br>print(&quot;np.percentile (25%):&quot;, np.percentile(arr, 25))<br>print(&quot;np.percentile (75%):&quot;, np.percentile(arr, 75))</pre><p>Result:</p><pre>Array: [ 1  2  3  4  5  6  7  8  9 10]<br><br>np.mean (rata-rata): 5.5<br>np.median (median): 5.5<br>np.std (standard deviation): 2.8722813232690143<br>np.var (variance): 8.25<br>np.sum (total sum): 55<br>np.min (minimum): 1<br>np.max (maximum): 10<br>np.percentile (25%): 3.25<br>np.percentile (75%): 7.75</pre><p><em>6.3 Aggregation by Axis</em></p><pre># Aggregation by axis<br>matrix = np.array([[1, 2, 3], [4, 5, 6]])<br>print(&quot;Matrix:&quot;)<br>print(matrix)<br>print()<br><br># axis=0: column-wise operation (row reduction)<br>print(&quot;Mean axis=0 (per kolom):&quot;, np.mean(matrix, axis=0))<br><br># axis=1: row-wise operation (column reduction)<br>print(&quot;Mean axis=1 (per baris):&quot;, np.mean(matrix, axis=1))<br><br># Without axis: flatten first, then calculate<br>print(&quot;Mean all:&quot;, np.mean(matrix))</pre><p>Result:</p><pre>Matrix:<br>[[1 2 3]<br> [4 5 6]]<br><br>Mean axis=0 (per kolom): [2.5 3.5 4.5]<br>Mean axis=1 (per baris): [2. 5.]<br>Mean all: 3.5</pre><p><strong>7. Pandas — Data Panel</strong></p><p>Pandas is a library for manipulating and analyzing tabular data. The two main data structures are:</p><p>1. <strong>Series:</strong> A 1D array with labels — like a single column in a spreadsheet<br>2. <strong>DataFrame:</strong> A 2D table with named rows and columns — like a full spreadsheet</p><p><em>7.1 Creating a Series</em></p><p>A series is a 1D array with labeled indices. Each element has a label that we can use to access it.</p><p>Example 1:</p><pre># Creating a Series from a list<br>data = [10, 20, 30, 40, 50]<br>series = pd.Series(data)<br>print(&quot;Series (without index):&quot;)<br>print(series)<br>print()<br><br># Creating a Series with a custom index<br>series2 = pd.Series(data, index=[&#39;a&#39;, &#39;b&#39;, &#39;c&#39;, &#39;d&#39;, &#39;e&#39;])<br>print(&quot;Series (with custom index):&quot;)<br>print(series2)</pre><p>Result:</p><pre>Series (without index):<br>0    10<br>1    20<br>2    30<br>3    40<br>4    50<br>dtype: int64<br><br>Series (with custom index):<br>a    10<br>b    20<br>c    30<br>d    40<br>e    50<br>dtype: int64</pre><p>Example 2:</p><pre># Accessing elements in a Series<br>series = pd.Series([10, 20, 30, 40, 50], index=[&#39;a&#39;, &#39;b&#39;, &#39;c&#39;, &#39;d&#39;, &#39;e&#39;])<br>print(&quot;Series:&quot;)<br>print(series)<br>print()<br><br># Access with label<br>print(&quot;series[&#39;a&#39;]:&quot;, series[&#39;a&#39;])<br>print(&quot;series[&#39;c&#39;]:&quot;, series[&#39;c&#39;])<br><br># Access with numeric position<br>print(&quot;series[0]:&quot;, series[0])<br>print(&quot;series[2]:&quot;, series[2])<br><br># Slicing with label<br>print(&quot;\nseries[&#39;a&#39;:&#39;c&#39;]:&quot;)<br>print(series[&#39;a&#39;:&#39;c&#39;])</pre><p>Result:</p><pre>Series:<br>a    10<br>b    20<br>c    30<br>d    40<br>e    50<br>dtype: int64<br><br>series[&#39;a&#39;]: 10<br>series[&#39;c&#39;]: 30<br>series[0]: 10<br>series[2]: 30<br><br>series[&#39;a&#39;:&#39;c&#39;]:<br>a    10<br>b    20<br>c    30<br>dtype: int64</pre><p><em>7.2 Creating a Dataframe</em></p><p>A DataFrame is the primary data structure in Pandas. It is like a table with named rows and columns.</p><pre># Create DataFrame from dictionary<br>data = {<br>    &#39;Name&#39;: [&#39;Alice&#39;, &#39;Bob&#39;, &#39;Charlie&#39;, &#39;Diana&#39;, &#39;Eve&#39;],<br>    &#39;Age&#39;: [25, 30, 35, 28, 32],<br>    &#39;Salary&#39;: [5000000, 6000000, 7000000, 5500000, 6500000]<br>}<br>df = pd.DataFrame(data)<br>print(&quot;DataFrame:&quot;)<br>print(df)</pre><p>Result:</p><pre>DataFrame:<br>      Name  Age   Salary<br>0    Alice   25  5000000<br>1      Bob   30  6000000<br>2  Charlie   35  7000000<br>3    Diana   28  5500000<br>4      Eve   32  6500000</pre><p><em>7.3 Exploring a DataFrame</em></p><p>Pandas provides many methods for examining the structure and content of data.</p><p>Example 1:</p><pre># df.head() - view the first 5 rows<br>print(&quot;df.head() - first 5 rows:&quot;)<br>print(df.head())<br>print()<br><br># df.tail() - view the last 5 rows<br>print(&quot;df.tail() - last 5 rows:&quot;)<br>print(df.tail())<br>print()<br><br># df.info() - DataFrame structure information<br>print(&quot;df.info():&quot;)<br>df.info()</pre><p>Result:</p><pre>df.head() - first 5 rows:<br>      Name  Age   Salary<br>0    Alice   25  5000000<br>1      Bob   30  6000000<br>2  Charlie   35  7000000<br>3    Diana   28  5500000<br>4      Eve   32  6500000<br><br>df.tail() - last 5 rows:<br>      Name  Age   Salary<br>0    Alice   25  5000000<br>1      Bob   30  6000000<br>2  Charlie   35  7000000<br>3    Diana   28  5500000<br>4      Eve   32  6500000<br><br>df.info():<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 5 entries, 0 to 4<br>Data columns (total 3 columns):<br> #   Column  Non-Null Count  Dtype <br>---  ------  --------------  ----- <br> 0   Name    5 non-null      object<br> 1   Age     5 non-null      int64 <br> 2   Salary  5 non-null      int64 <br>dtypes: int64(2), object(1)<br>memory usage: 248.0+ bytes</pre><p>Example 2:</p><pre># df.describe() - descriptive statistics for numeric columns<br>print(&quot;df.describe() - descriptive statistics:&quot;)<br>print(df.describe())<br>print()<br><br># df.shape - Dataframe dimensions (rows, columns)<br>print(&quot;df.shape:&quot;, df.shape)<br><br># df.columns - column names<br>print(&quot;df.columns:&quot;, df.columns.tolist())<br><br># df.dtypes - data types for each column<br>print(&quot;df.dtypes:&quot;)<br>print(df.dtypes)</pre><p>Result:</p><pre>df.describe() - descriptive statistics:<br>             Age        Salary<br>count   5.000000  5.000000e+00<br>mean   30.000000  6.000000e+06<br>std     3.807887  7.905694e+05<br>min    25.000000  5.000000e+06<br>25%    28.000000  5.500000e+06<br>50%    30.000000  6.000000e+06<br>75%    32.000000  6.500000e+06<br>max    35.000000  7.000000e+06<br><br>df.shape: (5, 3)<br>df.columns: [&#39;Name&#39;, &#39;Age&#39;, &#39;Salary&#39;]<br>df.dtypes:<br>Name      object<br>Age        int64<br>Salary     int64<br>dtype: object</pre><p><em>7.4 Choosing a Column</em></p><p>How to select one or more columns from a DataFrame.</p><pre># Select a single column - creates a Series<br>print(&quot;df[&#39;Name&#39;] (single column):&quot;)<br>print(df[&#39;Name&#39;])<br>print()<br><br># Select multiple columns - creates a DataFrame<br>print(&quot;df[[&#39;Name&#39;, &#39;Age&#39;]] (multiple columns):&quot;)<br>print(df[[&#39;Name&#39;, &#39;Age&#39;]])</pre><p>Result:</p><pre># Select a single column - creates a Series<br>print(&quot;df[&#39;Name&#39;] (single column):&quot;)<br>print(df[&#39;Name&#39;])<br>print()<br><br># Select multiple columns - creates a DataFrame<br>print(&quot;df[[&#39;Name&#39;, &#39;Age&#39;]] (multiple columns):&quot;)<br>print(df[[&#39;Name&#39;, &#39;Age&#39;]])</pre><p><em>7.5 Add and Remove Column</em></p><pre># Add new column<br>df[&#39;Department&#39;] = [&#39;HR&#39;, &#39;IT&#39;, &#39;Finance&#39;, &#39;IT&#39;, &#39;Marketing&#39;]<br>print(&quot;DataFrame with new column:&quot;)<br>print(df)<br>print()<br><br># Remove column<br>df_drop = df.drop(&#39;Department&#39;, axis=1)  # axis=1 for columns<br>print(&quot;DataFrame after removing Department column:&quot;)<br>print(df_drop)</pre><p>Result:</p><pre>DataFrame with new column:<br>      Name  Age   Salary Department<br>0    Alice   25  5000000         HR<br>1      Bob   30  6000000         IT<br>2  Charlie   35  7000000    Finance<br>3    Diana   28  5500000         IT<br>4      Eve   32  6500000  Marketing<br><br>DataFrame after removing Department column:<br>      Name  Age   Salary<br>0    Alice   25  5000000<br>1      Bob   30  6000000<br>2  Charlie   35  7000000<br>3    Diana   28  5500000<br>4      Eve   32  6500000</pre><p><em>7.6 Filtering Data (Selecting Row)</em></p><p>Selecting rows based on specific conditions.</p><p>Example 1:</p><pre># Filter with simple condition<br>df_usia = df[df[&#39;Age&#39;] &gt; 30]<br>print(&quot;Employees with Age &gt; 30:&quot;)<br>print(df_usia)<br>print()<br><br># Filter by isin - specific values<br>df_dept = df[df[&#39;Department&#39;].isin([&#39;IT&#39;, &#39;HR&#39;])]<br>print(&quot;Employees in the IT or HR Department:&quot;)<br>print(df_dept)</pre><p>Result:</p><pre>Employees with Age &gt; 30:<br>      Name  Age   Salary Department<br>2  Charlie   35  7000000    Finance<br>4      Eve   32  6500000  Marketing<br><br>Employees in the IT or HR Department:<br>    Name  Age   Salary Department<br>0  Alice   25  5000000         HR<br>1    Bob   30  6000000         IT<br>3  Diana   28  5500000         IT</pre><p>Example 2:</p><pre># Filtering with multiple conditions<br># &amp; (AND), | (OR), ~ (NOT)<br># Note: Use parentheses for each condition!<br>df_filter = df[(df[&#39;Age&#39;] &gt; 28) &amp; (df[&#39;Salary&#39;] &gt; 5500000)]<br>print(&quot;Age &gt; 28 AND Salary &gt; 5500000:&quot;)<br>print(df_filter)<br>print()<br><br># Filter with OR<br>df_or = df[(df[&#39;Department&#39;] == &#39;IT&#39;) | (df[&#39;Salary&#39;] &gt; 6500000)]<br>print(&quot;Department IT OR Salary &gt; 6500000:&quot;)<br>print(df_or)</pre><p>Result:</p><pre># Filtering with multiple conditions<br># &amp; (AND), | (OR), ~ (NOT)<br># Note: Use parentheses for each condition!<br>df_filter = df[(df[&#39;Age&#39;] &gt; 28) &amp; (df[&#39;Salary&#39;] &gt; 5500000)]<br>print(&quot;Age &gt; 28 AND Salary &gt; 5500000:&quot;)<br>print(df_filter)<br>print()<br><br># Filter with OR<br>df_or = df[(df[&#39;Department&#39;] == &#39;IT&#39;) | (df[&#39;Salary&#39;] &gt; 6500000)]<br>print(&quot;Department IT OR Salary &gt; 6500000:&quot;)<br>print(df_or)</pre><p><em>7.7 Group By and Aggregation</em></p><p>Group By is used to group data based on specific values and then perform aggregations.</p><p>Example 1:</p><pre># GroupBy - grouping data<br>grouped = df.groupby(&#39;Department&#39;)[&#39;Salary&#39;].mean()<br>print(&quot;Average Salary per Department:&quot;)<br>print(grouped)<br>print()<br><br># Group with multiple aggregate functions<br>agg_result = df.groupby(&#39;Department&#39;).agg({<br>    &#39;Age&#39;: [&#39;mean&#39;, &#39;min&#39;, &#39;max&#39;],<br>    &#39;Salary&#39;: [&#39;sum&#39;, &#39;mean&#39;]<br>})<br>print(&quot;Aggregated per Department:&quot;)<br>print(agg_result)</pre><p>Result:</p><pre>Average Salary per Department:<br>Department<br>Finance      7000000.0<br>HR           5000000.0<br>IT           5750000.0<br>Marketing    6500000.0<br>Name: Salary, dtype: float64<br><br>Aggregated per Department:<br>             Age            Salary           <br>            mean min max       sum       mean<br>Department                                   <br>Finance     35.0  35  35   7000000  7000000.0<br>HR          25.0  25  25   5000000  5000000.0<br>IT          29.0  28  30  11500000  5750000.0<br>Marketing   32.0  32  32   6500000  6500000.0</pre><p>Example 2:</p><pre># Count<br>print(&quot;Number of employees per department:&quot;)<br>print(df.groupby(&#39;Department&#39;).size())<br>print()<br><br># Sum<br>print(&quot;Total Salary per Department:&quot;)<br>print(df.groupby(&#39;Department&#39;)[&#39;Salary&#39;].sum())</pre><p>Result:</p><pre>Number of employees per department:<br>Department<br>Finance      1<br>HR           1<br>IT           2<br>Marketing    1<br>dtype: int64<br><br>Total Salary per Department:<br>Department<br>Finance       7000000<br>HR            5000000<br>IT           11500000<br>Marketing     6500000<br>Name: Salary, dtype: int64</pre><p><em>7.8 Reading External Files</em></p><p>Pandas can read various file formats, such as CSV, Excel, JSON, and others.</p><p><em>Read CSV File:</em></p><pre># Read CSV file<br>df = pd.read_csv(&#39;DATA_DAILY.csv&#39;)<br>print(df.head())</pre><p>Result:</p><pre>        DATE  COLUMN_A  COLUMN_B  COLUMN_C<br>0  2026-01-01    56.336    71.421    71.421<br>1  2026-01-02    56.336    65.106    68.159<br>2  2026-01-03    56.336    66.147    67.488<br>3  2026-01-04    56.336    72.553    68.635<br>4  2026-01-05    56.336    72.443    69.394</pre><p><em>Read Excel File:</em></p><pre># Read Excel file<br>df = pd.read_excel(&#39;DATA_DAILY.xlsx&#39;)<br>print(df.head())</pre><p>Result:</p><pre>        DATE  COLUMN_A   COLUMN_B   COLUMN_C<br>0 2026-01-01    56.336  71.421367  71.421367<br>1 2026-01-02    56.336  65.105761  68.158622<br>2 2026-01-03    56.336  66.146667  67.487725<br>3 2026-01-04    56.336  72.552641  68.635022<br>4 2026-01-05    56.336  72.442537  69.393978</pre><p>We can also save the dataframe to another file, such as an Excel or CSV file.</p><pre>df.to_csv(&#39;output_name.csv&#39;, index=False)  # save to CSV<br>df.to_excel(&#39;output_name.xlsx&#39;, index=False)  # save to Excel</pre><p><em>7.9 Handling NaN Values</em></p><p>Real-world data often contains missing values. Pandas represents missing values as NaN (Not a Number).</p><p>Example 1:</p><pre># Creating a DataFrame with NaN values<br>df_nan = pd.DataFrame({<br>    &#39;A&#39;: [1, 2, np.nan, 4],<br>    &#39;B&#39;: [5, np.nan, np.nan, 8],<br>    &#39;C&#39;: [9, 10, 11, 12]<br>})<br>print(&quot;DataFrame with NaN:&quot;)<br>print(df_nan)<br>print()<br><br># Check for NaN values<br>print(&quot;Check for NaN values:&quot;)<br>print(df_nan.isna())<br>print()<br><br># Count NaN per column<br>print(&quot;Count of NaN per column:&quot;)<br>print(df_nan.isna().sum())</pre><p>Result:</p><pre>DataFrame with NaN:<br>     A    B   C<br>0  1.0  5.0   9<br>1  2.0  NaN  10<br>2  NaN  NaN  11<br>3  4.0  8.0  12<br><br>Check for NaN values:<br>       A      B      C<br>0  False  False  False<br>1  False   True  False<br>2   True   True  False<br>3  False  False  False<br><br>Count of NaN per column:<br>A    1<br>B    2<br>C    0<br>dtype: int64</pre><p>Example 2:</p><pre># Drop rows/columns containing NaN<br>print(&quot;Drop rows with NaN (any):&quot;)<br>print(df_nan.dropna())<br>print()<br><br># dropna(all) - drop if ALL values are NaN<br>print(&quot;Drop rows if ALL are NaN (none in this example)&quot;)<br><br># dropna(subset=[‘column’]) - check only specific columns<br>print(&quot;Drop if column A is NaN:&quot;)<br>print(df_nan.dropna(subset=[&#39;A&#39;]))</pre><p>Result:</p><pre>Drop rows with NaN (any):<br>     A    B   C<br>0  1.0  5.0   9<br>3  4.0  8.0  12<br><br>Drop rows if ALL are NaN (none in this example)<br>Drop if column A is NaN:<br>     A    B   C<br>0  1.0  5.0   9<br>1  2.0  NaN  10<br>3  4.0  8.0  12</pre><p>Example 3:</p><pre># Replace NaN values with a specific value<br># fillna(value)<br>print(&quot;Replace NaN values with 0:&quot;)<br>print(df_nan.fillna(0))<br>print()<br><br># Fill with mean value<br>mean_A = df_nan[&#39;A&#39;].mean()<br>print(f&quot;Mean column A: {mean_A}&quot;)<br>print(&quot;Fill NaN in column A with mean:&quot;)<br>df_nan[&#39;A&#39;] = df_nan[&#39;A&#39;].fillna(mean_A)<br>print(df_nan)<br><br># Forward fill - fill with the previous value<br>df_nan2 = pd.DataFrame({<br>    &#39;A&#39;: [1, 2, np.nan, 4]<br>})<br>print(&quot;\nForward fill:&quot;)<br>print(df_nan2.fillna(method=&#39;ffill&#39;))</pre><p>Result:</p><pre>Replace NaN values with 0:<br>          A    B   C<br>0  1.000000  5.0   9<br>1  2.000000  0.0  10<br>2  2.333333  0.0  11<br>3  4.000000  8.0  12<br><br>Mean column A: 2.3333333333333335<br>Fill NaN in column A with mean:<br>          A    B   C<br>0  1.000000  5.0   9<br>1  2.000000  NaN  10<br>2  2.333333  NaN  11<br>3  4.000000  8.0  12<br><br>Forward fill:<br>     A<br>0  1.0<br>1  2.0<br>2  2.0<br>3  4.0</pre><p><em>7.10 Sorting Data</em></p><p>Sort data by value or index.</p><pre># Sort by value<br>df_sorted = df.sort_values(&#39;COLUMN_B&#39;, ascending=False)<br>print(&quot;Sort by COLUMN_B (descending):&quot;)<br>print(df_sorted)<br>print()<br><br># Sort by multiple columns<br>df_sorted2 = df.sort_values([&#39;COLUMN_B&#39;, &#39;COLUMN_C&#39;])<br>print(&quot;Sort by COLUMN_B, then COLUMN_C:&quot;)<br>print(df_sorted2)</pre><p>Result:</p><pre>Sort by COLUMN_B (descending):<br>        DATE  COLUMN_A    COLUMN_B   COLUMN_C<br>7 2026-01-08    56.336  102.719665  69.689357<br>3 2026-01-04    56.336   72.552641  68.635022<br>4 2026-01-05    56.336   72.442537  69.393978<br>0 2026-01-01    56.336   71.421367  71.421367<br>9 2026-01-10    56.336   70.608654  69.540826<br>8 2026-01-09    56.336   67.313213  69.411940<br>2 2026-01-03    56.336   66.146667  67.487725<br>5 2026-01-06    56.336   66.012126  69.315716<br>1 2026-01-02    56.336   65.105761  68.158622<br>6 2026-01-07    56.336   64.258783  68.522033<br><br>Sort by COLUMN_B, then COLUMN_C:<br>        DATE  COLUMN_A    COLUMN_B   COLUMN_C<br>6 2026-01-07    56.336   64.258783  68.522033<br>1 2026-01-02    56.336   65.105761  68.158622<br>5 2026-01-06    56.336   66.012126  69.315716<br>2 2026-01-03    56.336   66.146667  67.487725<br>8 2026-01-09    56.336   67.313213  69.411940<br>9 2026-01-10    56.336   70.608654  69.540826<br>0 2026-01-01    56.336   71.421367  71.421367<br>4 2026-01-05    56.336   72.442537  69.393978<br>3 2026-01-04    56.336   72.552641  68.635022<br>7 2026-01-08    56.336  102.719665  69.689357</pre><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p><strong>Summary</strong></p><p><strong>Numpy</strong></p><ul><li><em>ndarray </em>: efficient multidimensional arrays</li><li><em>Vectorized operations </em>: operations applied to all elements at once</li><li><em>Creating Arrays </em>: <em>array(), arange(), zeros(), ones(), linspace(), random.rand()</em></li><li><em>Attributes </em>: <em>shape, dtype, ndim, size</em></li><li><em>Indexing/Slicing </em>: Accessing elements by index or range</li><li><em>Statistics </em>: <em>mean(), median(), std(), min(), max() </em>etc.</li></ul><p><strong>Pandas</strong></p><ul><li><em>Series </em>: 1D array with labels — like a spreadsheet column</li><li><em>DataFrame </em>: 2D table with rows and columns — like a full spreadsheet</li><li><em>Creating a Dataframe</em> : from a dictionary, CSV, Excel, etc.</li><li><em>Choosing a Column</em> : <em>df[‘column’]</em> or <em>df[[‘column1’, ‘column2’]]</em></li><li><em>Filtering </em>: <em>df[condition]</em> to select rows</li><li>Group By : grouping data for aggregation</li><li><em>NaN Handling</em> : <em>isna(), dropna(), fillna()</em></li></ul><p>In conclusion, <strong>NumPy and Pandas are two important Python libraries that are very useful for data processing and analysis</strong>. NumPy makes it easier to work with arrays and mathematical operations, while Pandas helps manage data in table form, such as DataFrames. By understanding the basics of NumPy and Pandas, we can manipulate data, perform calculations, and analyze information more quickly, neatly, and efficiently. Therefore, these two libraries are an essential foundation for anyone who wants to learn data science, machine learning, or data analysis using Python.</p><p>Thank you! Feel free if you want to contact me and brainstorming!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ce5265b445d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/python-fundamental-1-pandas-and-numpy-2ce5265b445d">Python Fundamental #1 — Pandas and Numpy</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SELECT and FUNCTION in MySQL]]></title>
            <link>https://medium.com/@akmalabdillah/select-function-dan-group-by-pada-mysql-afacd0b963aa?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/afacd0b963aa</guid>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Fri, 18 Mar 2022 10:46:31 GMT</pubDate>
            <atom:updated>2026-05-07T08:39:23.949Z</atom:updated>
            <content:encoded><![CDATA[<p>MySQL #2</p><p>Assalamualaikum Wr. Wb.</p><p>After previously discussing how to create a simple database using MySQL, we’ll now briefly discuss how to “manipulate” the data in our database. We can use a command to retrieve data from our database; this is called an SQL query or SELECT. Some commands are simple, while others are a bit more complex. Meanwhile, if we want to execute a command according to a specific function we need, it’s called a function or method. MySQL includes several built-in functions, such as those for calculating averages or summing values, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Dc0DGaiXLh4oD7xGEF2U-w.jpeg" /></figure><blockquote><strong>SELECT Statement</strong></blockquote><p>The SELECT statement in MySQL is one of the commands that can be used to retrieve data from one or more tables in a database.<br>The basic syntax of the SELECT statement is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*IqAwmIztLCeDGJem3-DF0w.png" /><figcaption>Syntax Dasar Perintah Select</figcaption></figure><p>· The SELECT statement in this syntax indicates that we are instructing the system to retrieve data.<br>· The * symbol, commonly known as a wildcard in this syntax, indicates that we will retrieve data from all columns. We can retrieve data from only certain columns; we simply need to specify which columns we want to retrieve.<br>· The FROM [table_name] statement in this syntax specifies which table we will retrieve data from in the database.<br>· The semicolon (;) in the command is used to mark the end of an SQL statement.<br>Now, if we run that command using the same database and table as in the previous MySQL discussion at <a href="https://medium.com/@akmalabdillah/membuat-database-sederhana-menggunakan-mysql-64d3647c7f2d">https://medium.com/@akmalabdillah/membuat-database-sederhana-menggunakan-mysql-64d3647c7f2d</a>, the result will be as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*8hIOxETFIrywyagl4Ic3oA.png" /><figcaption>Hasil Tampilan Data Semua Kolom</figcaption></figure><p>In addition to displaying all columns in a table, we can also display a single column or even multiple columns. For example, suppose we want to display data from the courier table showing the last name and delivery status. In that case, we simply use the basic syntax above but change the column names.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*KO1SvHxOrYZ6e5yasvhSig.png" /><figcaption>Syntax untuk Menampilkan Beberapa Kolom</figcaption></figure><p>When we run that command, the result will be as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*0pUTvZhLgc1oDm4PkyTdwA.png" /><figcaption>Tampilan Kolom last_name dan status kurir</figcaption></figure><p>Well, to display data based on table columns, we just need to combine the commands in MySQL.<br>Now, what if we’re asked to display data but are limited by the number of rows? We can use a basic command like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*fqC2-kPPG5rEje8ltPc70A.png" /><figcaption>SELECT Statement Limit</figcaption></figure><p>That SELECT statement can display a limited number of rows of data. For example, suppose I want to display data from the courier table with all columns but only four rows. In that case, the syntax would be as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*tmkn7HaCrO14jx4Gdym_Vw.png" /><figcaption>Syntax SELECT dengan LIMIT</figcaption></figure><p>Based on that command, the following data will be displayed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*pP9n2_t2lqBmm1PmfcS9GA.png" /><figcaption>Hasil SELECT Statement dengan LIMIT</figcaption></figure><p>As we can see, only four rows of data are displayed, along with all the columns.</p><p><strong>Using DISTINCT in SELECT</strong></p><p>Sometimes, when we display a large amount of data, we encounter duplicates. Well, we can resolve this by using DISTINCT. This means we won’t display duplicate data; in other words, we’ll remove duplicate rows, resulting in unique or distinct data. The general query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*aXjhpz-E0u4vexUjfxVhBQ.png" /></figure><p>Based on that query, the concept is the same as when we display data in a table; we simply need to add the DISTINCT clause after SELECT. For a clearer explanation, let’s try this with the courier table. For example, if we want to find out how many different statuses there are in the courier table, we can write the query as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*WZ1Q4_ako7bZOAsMVseUNw.png" /></figure><p>The results are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/113/1*s_v06fPTIIEuT0kJhY3scg.png" /></figure><p>Based on these results, the courier table shows only two status types in the status column: “Stay” and “Delivering.”</p><p><strong><em>Prefix and Alias</em></strong></p><p>Next, we will discuss two terms used in the SELECT statement: <em>prefix </em>and <em>alias</em>. A prefix is a term used to add the table name before the column name. An alias, on the other hand, is used to rename a column or table that we will use in the SELECT statement.<br>To use a prefix, you simply need to add the table name; this is useful when working with more than one table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*9FwP8BOdYIMG4kh5fMeC5g.png" /></figure><p><strong>Descriptions:</strong></p><p>· <em>table_name </em>: the name of the table that will be used as a prefix for the column to be accessed.</p><p>· <em>column_name </em>: the name of the column in the table from which data will be retrieved</p><p>· <em>table_name (the last one)</em> : the name of the table from which data will be retrieved<br>For a clearer understanding, let’s try applying this to the courier table. I will write the following query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*saQdkb7mPi08w8FGfqDaAw.png" /></figure><p>The following results will then appear:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/1*yPAVaD5DtxKU7StU36l8sw.png" /><figcaption>Hasil Prefix</figcaption></figure><p>As I explained above, using this prefix is very useful when managing and identifying data from multiple tables, as it makes it easier to access data from each table.</p><p>Aliases are used to change the name of a column retrieved by a <strong>SELECT </strong>statement by using the <strong>AS </strong>keyword. The complete AS query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*lMx83czE9RhSCDg72KccYQ.png" /></figure><p><strong>Descriptions:</strong></p><p>· column_name: the name of the column from which data will be retrieved</p><p>· new_column_name: the name of the new column that will replace the previous column name</p><p>· table_name: the name of the table from which data will be retrieved<br>To make this clearer, let’s try it out on the courier table. I’ll write the following query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*4K2Q4Hfk7jfIbo4gW52caQ.png" /></figure><p>After running the query above, it will produce the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/125/1*JKb48VXdGRXJbNBS-oaiCw.png" /></figure><p>Based on that output, we can see that the result returned is the `no_hp` column, and in the result, the column name automatically changes to `phone_number` as we instructed. However, this change is not permanent. If we run the query “SELECT phone_number FROM kurir;”, it will result in an error because that column name does not exist in the table from which we are retrieving data.</p><p>So, the next question is: can we use an alias but omit the AS keyword? The answer is <strong>YES</strong>! The common query used is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*Y_k6HpQ-7qjw696DP2HSyw.png" /></figure><p>For more details, let’s take a look at the courier table. I’ll write the following query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*UQqH61307C9rkZaspX4mWA.png" /></figure><p>This will produce the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/1*hZL9iZrN0BSxhO7-k_lZWw.png" /></figure><p>The results are the same when I use the AS keyword; the output remains unchanged. The only difference is in how the query is written. We can also use two columns at once with this alias. Here is the query I used to display the last_name and no_hp columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*9gD5poXNsiu0v4PvcCYumg.png" /></figure><p>The output as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/206/1*S6AEepZDYOC5YavTGXmGeQ.png" /></figure><p>Next question: Can prefixes and aliases be combined in a single query line? The answer is, of course, yes! Let’s take a look at the following discussion.</p><p><strong>Combining Prefixes and Aliases</strong></p><p>We can combine prefixes and aliases to make it easier to access data, typically using queries like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*Z1sLWM_Fhf0LaezqXdVhUw.png" /></figure><p><strong>Descriptions:</strong></p><p>· table_name: the name of the table that will serve as the prefix for the column we will access<br>· column_name: the name of the column whose data we will access<br>· new_column_name: the name of the new column that will replace the previous column name<br>· table_name (the last one): the name of the table whose data we will access<br>For a clearer understanding, let’s try using that query on the courier table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*6rmzh26sqlK6JyqLRnclvA.png" /></figure><p>The output as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/1*hYF1uvzOYeDa0H2S8S5E2Q.png" /></figure><p><strong>Using <em>Alias </em>on Table</strong></p><p>In addition to being used for column names, aliases can also be used for table names. The syntax for queries is the same as for column aliases, typically using the AS keyword. The general syntax for queries is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*maFYTHzzj-tctzlTw9JgiA.png" /></figure><p><strong>Descriptions :</strong></p><p>· column_name: the name of the column we want to display<br>· table_name: the name of the table we want to rename using an alias<br>· new_table_name: the new table name that will replace the previous one<br>For a clearer understanding, let’s try this out on the <em>kurir </em>table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*oS197mTuLu2gQGQbK24UKw.png" /></figure><p>If we run that query, it will produce the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*a1_WKEZ5bULJm9YEDnPSUg.png" /></figure><p><strong>Using <em>Prefix </em>with <em>Alias </em>Table</strong></p><p>If we use aliases in a table, we can also use aliases in the prefix. The general syntax for writing a query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*ZFAycxBJZJUFYmG-3b-NDw.png" /></figure><p><strong>Descriptions:</strong></p><p>· table: an alias for the table used as a prefix for the column we are accessing<br>· column_name: the name of the column whose data we are accessing<br>· table_name: the name of the table whose data we are accessing<br>· table: the name of the new table (alias) that replaces the previous table name<br>For a clearer understanding, let’s try the query above using the <em>kurir </em>table as an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*VLGHPWLRYWvpUhx-FaSwag.png" /></figure><p>After running that query, the results are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*WCWbisOolshFLvYoUgBdQw.png" /></figure><p>So, aliases can also be applied to prefixes.</p><p><strong>Using <em>WHERE </em>in SELECT</strong></p><p>We often encounter WHERE clauses in queries, whether for updating or deleting data. The WHERE clause is used to filter data based on specific conditions that we define. Typically, the query used is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*cgMUxQJodPz_aBGbR5Qlkw.png" /></figure><p><strong>Descriptions:</strong></p><p>· column_name: the name of the column whose data will be accessed<br>· table_name: the name of the table whose data will be accessed<br>· condition: A specific condition used in data access<br>The basic structure of the condition here is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/1*Ti-AQZxF5ULa3oruDEA4Zw.png" /></figure><p>For a clearer understanding, let’s apply this to the courier table. I’ll write the query as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*8invt_bvfRLVDOZKkTZBUA.png" /></figure><p>Next, I will run that query and get the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*fSGIqiXERpBpu5ZCiU9PNg.png" /></figure><p>In the query above, I will display all columns in the courier table, provided that the value in the status column is “stay.” As shown in the courier table, there are only two people or couriers whose status is still “stay.”</p><p><strong>Filtering for Numerical Data</strong></p><p>Now I will add a new column — the salary column for each courier — and then fill in the salary for each courier. The updated courier table data is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*wdJpAqxu0_GR-XkekdUCzA.png" /></figure><p>I added a salary column with values ranging from 3,500,000 to 4,200,000. Now I’ll try to display the filtered data. For example, if I want to see couriers whose salary is less than or equal to 4 million, the filter query would be as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*O7JJ4qm5C-femqPdx0gAFw.png" /></figure><p>Based on the query above, I’m still using the SELECT statement because I want to display data from that table, and I’m also using the WHERE clause to specify the conditions I want. For this condition, I’m using a mathematical operator that means “less than or equal to.” The result is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*q2BkmcNqHj5arD2DoLAzKw.png" /></figure><p>As we can see, the value in the “Salary” column will automatically display values less than or equal to 4 million. This also applies to other conditions, such as less than, greater than, and so on.</p><p><strong>Using <em>OR </em>Operator</strong></p><p>The OR operator is used when selecting criteria or conditions to be met. So, if we write a query with two conditions connected by the OR operator, the result is:<br>1. Meets all criteria or conditions, or<br>2. Meets at least one criterion or condition</p><p>For queries, the general format is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*D7l3bHgoIlGXep0Rb_hoJA.png" /></figure><p><strong>Descriptions:</strong></p><p>· column_name: the name of the column whose data will be accessed<br>· table_name: the name of the table whose data will be accessed<br>· condition1: The first condition that will be used as a criterion for retrieving data<br>· condition2: The second condition that will be used as a criterion for retrieving data<br>For a clearer understanding, let’s apply this to the courier table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*rXBKW2Y5GiBhNZWK2pVjVA.png" /></figure><p>Based on that query, we will display data for couriers who are currently in the “delivering” status or whose salary is less than 4 million; the results are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*tL5HHJh0rGDqJ5PCdua0MA.png" /></figure><p>As you can see, I’ve displayed all the columns in the courier table based on the criteria specified in the query I wrote earlier.</p><p><strong>Using <em>AND </em>Operator</strong></p><p>The AND operator is essentially written the same way as the OR operator, but the underlying logic is different. The AND operator evaluates to true only if all defined conditions are met, since all conditions are mutually exclusive. For queries, it is generally used as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*y3WIXfbLwIcIzqSLOG6R6w.png" /></figure><p><strong>Descriptions:</strong></p><p>· column_name: the name of the column whose data will be accessed<br>· table_name: the name of the table whose data will be accessed<br>· condition1: The first condition that will be used as a criterion for retrieving data<br>· condition2: The second condition that will be used as a criterion for retrieving data<br>For a clearer understanding, let’s try this out on the courier table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*YRdCOnz2e4rnE2OQ1b5Fuw.png" /></figure><p>Based on that query, we will display data for couriers who currently have a “stay” status and earn a salary of 4 million; the results are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*ZtxJGCRi0bqDIwAapRhPYw.png" /></figure><p>Based on these results, since I used the AND operator, both conditions must be met. Since, according to the courier table, there is only one entry that meets the criteria of having a “stay” status and a salary of 4 million, only one row of data is displayed.</p><p><strong>Function in MYSQL</strong></p><p>A function is a subprogram used to execute a specific command based on the function we are operating on. Functions are extremely helpful when processing data, making the process easier and more accurate. Functions can perform data operations in a database, such as numerical calculations (sum, count, avg, etc.) or non-numerical operations, such as string manipulation.</p><p><strong><em>Scalar Function, Aggregate, Text</em></strong></p><p>Scalar functions in SQL are used to return a single value from a given value, while aggregate functions are used to perform calculations on a group of values and then return a single value.</p><ol><li><strong><em>Scalar Functions</em></strong></li></ol><p>In MySQL, you can view the complete list of scalar functions on the following page. There are many mathematical functions available in MySQL; we will try just a few of them, such as ABS(), CEILING(), FLOOR(), ROUND(), SQRT(), MOD(), and EXP().</p><p>· <strong>Math Scalar Function - ABS()</strong></p><p>The ABS() function is used to return the absolute value of the input. A typical query looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/195/1*W_OxkYcOVEDiYghBIgnCZA.png" /></figure><p>Or, to calculate the absolute value of a number, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*vlfDERkBmINAePeCX7d0ww.png" /></figure><p>When we find the absolute value of -3.5, the result is 3.5. Similarly, when we find the absolute value of a column, the values in that column will yield their absolute values.</p><p>· <strong>Math Scalar Function- CEILING()</strong></p><p>The CEILING() function is used to return the smallest integer that is greater than or equal to the input value, or the value rounded up to the nearest integer. A typical query looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*4xgLK0-tOkqhnoOQqjFGmw.png" /></figure><p>Alternatively, to query the ceiling value of a number, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/1*gOKvgaFR8wCPy-C4Ewlgjg.png" /></figure><p>When we encounter cases involving decimal values — whether a single value or values in a column — and want to round them up, we can use the CEILING function. In the example above, I will round the value 3.5, which will automatically result in 4.</p><p>· <strong>Math Scalar Function - FLOOR()</strong></p><p>The FLOOR() function is used to return the largest integer that is less than or equal to the input value, rounded down. A typical query looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/211/1*kXNwQzUCLhHqjNtLNJTmkQ.png" /></figure><p>Atau untuk membuat <em>query </em>nilai <em>floor </em>dari suatu nilai dapat kita gunakan sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*oG4I311VfXbQjg5ZheW0BQ.png" /></figure><p>When we encounter cases involving decimal values — whether a single value or values in a column — and want to round them down, we can use the FLOOR function. In the example above, I will round the value 21.38, which will automatically result in 21.</p><p>· <strong>Math Scalar Function - ROUND()</strong></p><p>The ROUND() function is used to return a decimal number rounded to a specific number of decimal places that we define. The general syntax is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/1*YPd6hiRjtU288_TI2Zug4w.png" /></figure><p>Or, to create a query for the rounded value of a number, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/185/1*1yHgrRFJyc6orsZ1ONo4dQ.png" /></figure><p>When we encounter cases involving decimal values with a large number of digits after the decimal point — whether for individual values or values in a column — and we want to round them to a specific number of decimal places to make them easier to read, we can use the round function. In the example above, I will round the value 25.67843 to two decimal places, so the result will automatically be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*l1Po4MaCJ9w--Z8loaXyLw.png" /></figure><p>· <strong>Math Scalar Function- SQRT()</strong></p><p>The SQRT() function is used to return the square root of an input value. The typical query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*B0KSUujp4XzehNJMRxHGYg.png" /></figure><p>Or, to calculate the square root of a value, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/1*nAc-POlRSKJwKXkL8szlGg.png" /></figure><p>When we encounter a case where a value needs to be squared, we can use the sqrt function. For example, I will find the square roots of the values in the “Salary” column using the query shown above, and the results are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/163/1*oIwE0DgtkUpIRPhDqYkB6w.png" /></figure><p>For other scalar functions, the syntax for writing the query is the same; you just need to adapt its use to your needs. Just try out each function for yourself.</p><p><strong>2. <em>Aggregate Functions</em></strong></p><p>In MySQL, you can view the complete list of aggregate functions on the following page. There are many mathematical functions available in MySQL; we will try just a few of them, such as AVG(), SUM(), COUNT(), MIN(), and MAX().</p><p>· <strong>Math Scalar Function - AVG()</strong></p><p>The AVG() function is used to calculate the average of a column. A typical query looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/195/1*iu0MiUS3JKBROJzcdnyvfQ.png" /></figure><p>Or, to query the AVG value of a column, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/1*VqA72CCgauSqKrUupfjDKg.png" /></figure><p>When we need to calculate the average of a column, we can use the AVERAGE function. Let’s try to calculate the average salary of the couriers in the couriers table using the query above. The result is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/1*x6KO4kkwVG3QlssIKITAxw.png" /></figure><p>The average salary for a courier is Rp. 3,910,000.</p><p>· <strong>Math Scalar Function - SUM()</strong></p><p>The SUM() function is used to calculate or sum a group of values (rows) in a column. The typical query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*XQb0kxUmQQQPztpEbKjgmg.png" /></figure><p>Or, to create a query for the SUM of a column, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*ozi_FQSZPGeJK3nze4NeCQ.png" /></figure><p>When we need to calculate the sum of values in a column, we can use the SUM function. Let’s try to find the total salary of the couriers in the couriers table using the query above. The result is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/106/1*sbKZE096cmHSc3LsRFy9Ow.png" /></figure><p>The total salary for all couriers is Rp. 19,550,000.</p><p>· <strong>Math Scalar Function - COUNT()</strong></p><p>The COUNT() function is used to count the number of rows in a column. The typical query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/1*jNU_TTXjmZH_myuIqtmRPQ.png" /></figure><p>Alternatively, to retrieve the COUNT value for a column, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*7GDplMG5Mxwd_FE3rjVHsA.png" /></figure><p>In the query above, we will find the number of rows in the first_name column, which is 5 rows.</p><p>· <strong>Math Scalar Function - MIN() and MAX()</strong></p><p>The MAX() function is used to return the maximum value in a column. The MIN() function is used to return the minimum value in a column. The typical query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*YGIVvrMPg_Tq4j3Hyi_dKQ.png" /><figcaption>Minimum Value</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/195/1*yuu_hOzUHuQbcGIjaKl_Ng.png" /><figcaption>Maximum Value</figcaption></figure><p>Alternatively, to query the MIN or MAX value of a column, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*L3VS4wX-2vExIECvJNjqzg.png" /></figure><p>In the query above, we will find the minimum and maximum values in the “salary” column, and the results are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*QCV-ad9faY6ckUKnuvhwzA.png" /><figcaption>Hasil Gaji Minimal</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*rHt7A4F3-WKQPRsBPocP7A.png" /><figcaption>Hasil Gaji Maksimal</figcaption></figure><p><strong>3. <em>Text Function</em></strong></p><p>In addition to performing functions on numerical data, we can also perform functions on text data. For a complete list of functions, please visit the following page. Text functions include CONCAT(), SUBSTR(), LENGTH(), REPLACE(), LOWER(), UPPER(), and others.</p><p>· <strong>String Function - CONCAT()</strong></p><p>The CONCAT() function is used to combine all arguments or inputs. The typical query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*3CA16RaE8BdoP7OiaePp3A.png" /></figure><p>Alternatively, to create a query that concatenates values from a table, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*KS0mUA5W5AH21Cuh6Iwksg.png" /></figure><p>In the query above, we will combine two columns — first_name and last_name — into a single column named nama_kurir. The result is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/1*3v0mzJHuSe-wP_kBhw8-Hg.png" /></figure><p>· <strong>String Function - LENGTH()</strong></p><p>The LENGTH() function is used to count the number of characters in a string or text. The typical query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/1*1mRdP-cN8F1A-4mS9b3qZA.png" /></figure><p>Or, to query the length of a table, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*ASVCmFgu9CHDiTEY_Z4d6A.png" /></figure><p>In the query above, we will return the number of characters in the first_name and last_name columns. The result is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*K3X4Hl7HUvzVkHnzk-IdqA.png" /></figure><p>· <strong>String Function - REPLACE()</strong></p><p>The REPLACE() function is used to replace characters in a string. The typical query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*fKo08C-6QAhrMxQ5Zwvnyw.png" /></figure><p>Alternatively, to create a query to replace values in a table, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*GTUQ9b32ZuX4OAnHoXslUg.png" /></figure><p>In the query above, we will change the value in the “status” column from ‘Delivering’ to “Delivering.” The result is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*k6XX2xlClp3Ry9zoMathng.png" /></figure><p>· <strong>String Function - LOWER() and UPPER()</strong></p><p>Sometimes we also need to change characters; for example, we might want to convert uppercase letters to lowercase using the LOWER() function. Conversely, we can convert lowercase letters to uppercase using the UPPER() function. The general query is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/1*lBdHyg4e8VYQcsmx-xtglQ.png" /></figure><p>Alternatively, to query the UPPER() value from a table, we can use the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*6iz8nwiAUSdnIgRycZ3qGA.png" /></figure><p>In the query above, we will convert the characters in the first_name column from lowercase to uppercase. The result is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*-EJR3Ak2997lZj85xyZvpQ.png" /></figure><p>The same applies when using the LOWER() function.<br>For more information, please visit the official MySQL website.</p><p>Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afacd0b963aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple Credit Approval Model menggunakan Decision Tree pada Microsoft Azure Machine Learning]]></title>
            <link>https://medium.com/@akmalabdillah/simple-credit-approval-model-menggunakan-decision-tree-pada-microsoft-azure-machine-learning-9c77dd9b189c?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c77dd9b189c</guid>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[credit]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Sat, 18 Dec 2021 04:45:52 GMT</pubDate>
            <atom:updated>2021-12-18T04:45:52.158Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*RY5YSyVls97NJQ6dqh6ZAg.jpeg" /></figure><p>Hallo. Assalamualaikum Wr. Wb.</p><p>Pada era sekarang, perkembangan ilmu IT sudah sangat pesat dan berkembang, terutama dengan banyaknya data yang sangatlah banyak. Kadang kita menemukan banyak data tetapi informasi yang didapat dengan cepat menemukan informasi pada sekumpulan data tersebut. Nah, sekarang sudah ada keilmuan Data Science dan Big Data. Yang mana dapat mengolah lautan data tersebut menjadi informasi yang sangat berguna. Di era big data ini, muncul sebuah metode yang bernama Machine Learning. Teknik Machine Learning ini adalah salah satu cabang yang menarik dari AI (Artificial Intelligence). Kenapa menarik? Karena Machine Learning adalah mesin yang dapat belajar layaknya manusia.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/0*Vs8TnkcJkNc2cVW0.jpg" /></figure><p>Salah satu teknologi yang tersedia untuk memudahkan dalam pembuatan Machine Learning adalah Microsoft Machine Learning atau Azure Machine Learning. Apa itu Azure Machine Learning? Azure Machine Learning yaitu layanan cloud untuk mempercepat maupun mengelola sistem atau siklus hidup dari pembelajaran mesin. Azure Machine Learning dimaksudkan untuk individu dan tim yang menerapkan MLOps dalam organisasi mereka untuk memasukkan model pembelajaran mesin ke dalam produksi dalam lingkungan produksi yang aman dan dapat diaudit. Azure juga sangat fleksibel bagi seorang Data Scientist karena dapat mendukung hampir semua sistem operasi, perangkat, dan juga bahasa pemrograman. Teman-teman dapat masuk dan registrasi ke Azure Machine Learning dengan mengklik <a href="https://studio.azureml.net/">Microsoft Machine Learning Studio (classic) (azureml.net)</a>. Setelah masuk, maka akan tertampil dashboard seperti berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Q3F5vz32JaRP3Et_J1cNQ.png" /></figure><p>Pada kesempatan kali ini, saya akan mengangkat case tentang Credit Approval, yang mana datanya saya ambil dari DQLab.</p><p>Selanjutnya kita kembali ke aplikasi Azure ML, dan memasukkan dataset yang akan kita gunakan. Selanjutnya klik tombol <em>new </em>pada pojok kiri bawah dan pilih datasets yang akan kita gunakan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RNU6gbUgHzcKHb3CCT1hew.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*685tv_5xC68JhpLBkX-TQA.png" /><figcaption>Dataset Berhasil Terupload</figcaption></figure><p><strong>Selanjutnya kita akan membuat sebuah experiment, </strong>dengan mengklik dashboard <em>experiment </em>pada bagian kiri dan memasukkan dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*_4vCp14SZH2ov27o8e43EQ.png" /></figure><p>Selanjutnya kita akan berada pada workspace dari <em>experiment</em>, dimana disisi kiri tersebut terdapat icon-icon atau disebut modul yang dapat didrag kedalam workspace.</p><p>Selanjutnya kita klik <em>experiment &gt; blank experiment</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s1IP6DfHJcFyBEZbmyIc-w.png" /><figcaption>Tampilan Awal Blank Experiment</figcaption></figure><p>Selanjutnya kita dapat merubah judul dari workspace tersebut sesuai dengan apa yang kita inginkan, seperti “Credit Approval Model”. Setelah itu, cari dan drag modul pada menu <em>Saved Datasets &gt; My Datasets &gt; “blank-data.csv” </em>kedalam workspace. Jika berhasil terinput, maka tampilannya akan berubah seperti:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lhmdYYtca1D2fi7uWoQLkg.png" /></figure><p><strong>Melihat Isi Data</strong></p><p>Bagaimana jika kita ingin melihat atau mereview data tersebut? Dapat kita lakukan dengan cara mengklik bulatan kecil pada data tersebut (Bulatan 1) selanjutnya klik <em>visualize</em>. Maka tampilannya sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oNHP2RECTxYtVFtAcCShbg.png" /><figcaption>Visualize Dataset</figcaption></figure><p>Untuk deskripsi datasetnya adalah sebagai berikut:</p><ul><li>Jumlah barisnya ada 600 baris</li><li>Jumlah kolomnya adalah 12 kolom dengan nama kolomnya adalah sebagai berikut:</li></ul><ol><li>id : Nomor unik yang digunakan untuk identifikasi masing-masing aplikasi pengajuan</li><li>age : Umur dari pengaju aplikasi kredit</li><li>sex : Jenis kelamin dari pengaju aplikasi kredit</li><li>region : Daerah dimana pengaju aplikasi kredit tinggal</li><li>income : Jumlah penghasilan dari pengaju aplikasi kredit</li><li>married : Status pernikahan pengaju aplikasi kredit</li><li>children : Jumlah anak yang dimiliki oleh pengaju aplikasi kredit</li><li>car : Apakah pengaju aplikasi kredit mempunyai mobil</li><li>save_acc : Apakah pengaju aplikasi kredit mempunyai rekening tabungan account</li><li>current_acct : Apakah pengaju aplikasi kredit mempunyai rekening tabungan yang sama saat ini</li><li>mortage : Apakah pengaju aplikasi kredit mempunyai kredit kepemilikan rumah saat ini</li><li>loan_approved : Status disetujui atau tidaknya aplikasi pengajuan kredit</li></ol><p><strong>Overview Isi Data</strong></p><ul><li>Header Data</li></ul><p>Untuk melihat semua header dari data kita, dapat diklik scroll di sebelah kanan. Semisal setelah kita lihat semua nama kolom yang ada dan kita akan memilih dan melihat satu kolom misal kolom income. Klik header kolom “income”, nantinya akan muncul di panel sebelah kanan distribusi dari nilai kita yang nantinya dapat kita analisis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HO7RN5F9tMY05f3QjvpZ1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KfgSw23e0ggpcTnikUqHEQ.png" /></figure><p>Terdapat 2 bagian pada overview data tersebut, yaitu statistics dan visualization dengan penjelasannya sebagai berikut:</p><ul><li><strong>Statistics </strong>berisi tentang informasi distribusi angka pada kolom tertentu. Seperti <em>mean, median, min-max, standard deviation, unique values, missing values, feature type. </em>Seperti pada kolom “income” untuk rata-ratanya yaitu 27524,03, mediannya yaitu 24952,3. Nilai terkecilnya adalah 5014,21 sedangkan nilai terbesarnya yaitu 63130,1. Untuk deviasi standarnya yaitu 12899,47. Terdapat 599 data yang berbeda pada kolom tersebut dengan tidak ada nilai yang hilang. Untuk tipe datanya adalah tipe data numerik.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*5eGZef9V1z8fgVBxKykuww.png" /></figure><ul><li><strong>Visualization </strong>berisi tentang karakteristik dari kolom tersebut yang berbentuk chart. Seperti chart yang berisikan masing-masing nilai baris pada kolom tersebut dengan jumlah frekuensinya. Sebagai contoh saya gunakan kolom “income” untuk visualisasinya. Sangat jelas terlihat income mana yang frekuensinya terbanyak dan income mana yang frekuensinya terkecil. Pada visualisasi histogram tersebut, kita juga dapat membandingkan histogram dari masing-masing kolom. Selain itu kita juga dapat membuat garis <em>cumulatif distribution </em>dan <em>probability density </em>pada kasus tersebut.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*J2ZAgHWnGnvZ7eg3pwA9PQ.png" /></figure><p><strong>Menambahkan Module Split Data</strong></p><p>Selanjutnya kita akan melakukan pembagian data menjadi 2, yaitu training data dan juga testing data untuk algoritma decision tree. Training data adalah data yang digunakan untuk melatih algoritma agar membentuk suatu model. Sedangkan testing data adalah data uji untuk melihat seberapa besar akurasi yang didapatkan. Pada workspace yang kita buat, sekarang tambahkan module “Split Data” dari kategori “Data Transformation” selanjutnya pilih “Sample Split” pada workspace kita.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*NMxWVoy2j-aiedNQajKgrA.png" /></figure><p>Dapat kita lihat terdapat 1 port input dan 2 port output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*hPFZMXpVAyN7J7pbt6YUxA.png" /></figure><p>Sekarang mari kita hubungkan kedua modul tersebut dengan cara mengklik bagian output port dari dataset “blank-data.csv”, lalu drag ke input port dari modul “Split Data” dan lepaskan seperti pada gambar berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*Rtsf-I6aUZBMoUnDG3Vbaw.png" /></figure><p>Selanjutnya klik modul “Split Data” dan pada panel kanan masukkan konfigurasi sebagai berikut:</p><ul><li>Fraction of rows in the first output datase dikonfigurasikan menjadi 0.8, hal ini berarti 80% data akan diberikan output port 1, sisa 20% ke output port 2.</li><li>Randomized split : False (tidak dicentang) yang berarti data dibagi secara berurutan, jadi tidak berdasarkan acakan (random)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*DkT6wW118QiOLUnreKODOw.png" /></figure><p>Selanjutnya kita save dulu pada dashboard bawah dan kita running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*KmmKOb0NaR8wE-Hcq9yj6w.png" /></figure><p>Tunggu hingga modul split data terdapat simbol centang hijau, yang mana berarti proses eksekusi telah selesai.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*QATx7gq_pTwo9XEQP7Fkbw.png" /></figure><p><strong>Melihat hasil pada Split Data</strong></p><p>Sekarang kita dapat melihat hasil eksekusi untuk modul “Split Data”, dengan cara klik pada port output 1 dan pilih visualize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*dSAuQPGFSAY380HkkFgx7A.png" /></figure><p>Maka akan didapat hasil sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*UKW863wb2yfvnz-GIzrFAA.png" /></figure><p>Jumlah baris adalah 480 baris, yang mana jumlah ini 80% dari keseluruhan data yaitu 600 baris. Nah, data ini akan dijadikan training data dalam proses berikutnya.</p><p>Sekarang mari kita coba klik port output 2 pada modul Split Data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*zMGp4rIpkmn6V6NU7AQtWQ.png" /></figure><p>Dapat kita lihat bahwa jumlah barisnya adalah 120 baris, dimana jumlah ini adalah 20% daru seluruh jumlah barisnya yaitu 600 baris. Porsi data ini akan kita jadikan testing data.</p><p><strong>Membuat Model Decision Tree</strong></p><p>Untuk membuat pemodelan Decision Tree, maka diperlukan 2 modul yang dibutuhkan dari kategori “Initialize Model” untuk pemilihan algoritma dan “Train Model” untuk pembentukan model. Selanjutnya untuk Decision Tree sendiri, masih banyak algoritma didalamnya yang dapat digunakan. Pada kesempatan kali ini, kita akan menggunakan modul “Two-Class Boosted Decision Tree”. Untuk langkahnya adalah sebagai berikut: Masukkan modul “Two-Class Boosted Decision Tree” dan “Train Model” ke dalam workspace. Selanjutnya hubungkan semua modul sehingga menjadi seperti berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*-uN3cKomsI7FxSWC8dSOCw.png" /></figure><p>Jika kita perhatikan, pada label “Train Model” terdapat tanda merah dikarenakan kita belum memasukkan kolom yang akan diprediksi (Label Column).</p><p>Selanjutnya klik pada modul “Train Model”, nantinya akan muncul di panel kanan “Launch column selector” dan jalankan selector tersebut.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T6csK8jl4-0C-YC0TIsT4w.png" /></figure><p>Setelah itu akan muncul kotak dialog “Select a Single Column” pilih kolom “loan_approved” dari bagian “Available Columns” untuk selanjutnya dimasukkan kedalam “Selected Columns”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*DBt4gvbLtl5heELsloF7CQ.png" /></figure><p>Selanjutnya klik tombol centang pada bagian kanan bawah dialog “Select a single column”.</p><p>Setelah itu kita beralih ke modul “Two-Class Boosted Decision Tree” dan rubah konfigurasinya menjadi “Number of trees constructed” ke nilai 1. Simpan dan jalankan experiment kita melalui tombol “Save” dan “Run” yang terletak pada bagian bawah workspace kita.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBH5VdGQotEDTUSZ6e6m0w.png" /></figure><p>Setelah sistem selesai me<em>running </em>program, klik output port pada modul “Train Model”, dan pilih visualize. Nantinya akan memunculkan satu decision tree yang berisikan informasi variabel dan nilai serta cabang keputusannya.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*APoxLyi6bZVD-ioFkxq4bA.png" /></figure><p>Kita dapat mengatur tampilan, entah itu zoom in atau zoom out tanpa takut penuruan resolusi. Selanjutnya klik opsi “Edges” pada bagian bawah, maka nantinya akan muncul kondisi True dan juga False sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*Ux4NPu0giFocT6Do8zi82Q.png" /></figure><p><strong>Melakukan Prediksi Menggunakan Testing Data</strong></p><p>Masukkan modul “Score Model” dan susun pada workspace dengan penempatan sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*fFsEiu7MwKatXFM37rUMfw.png" /></figure><p>Selanjutnya tinggal kita save dan juga jalankan seperti biasa, setelah selesai running program kita dapat melihat output port dari “Score Model” dan terakhir pilih visualize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*QOOO3dDNu3WNfu3HsEFSUg.png" /></figure><p>Berdasarkan hasil tersebut, dapat kita lihat bahwa kolom baru yaitu “Scored Labels” dimana kolom tersebut merupakan kolom prediksi dari model untuk kolom “loan_approved”. Berdasarkan 2 kolom tersebut dimana kolom Scored Labels merupakan adalah nilai prediksi dari kolom loan_approved. Tetapi terdapat beberapa baris dari 2 kolom itu yang memiliki perbedaan nilai.</p><p><strong>Melakukan Evaluasi pada Model</strong></p><p>Sekarang kita perlu untuk memasukkan modul “Evaluate Model” dan memasukkan ke workspace dengan komposisi sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*-LyzWdDN7HrE3vnMFSuuyw.png" /></figure><p>Selanjutnya tidak ada yang perlu diatur maupun dikonfigurasikan, kita hanya perlu menyimpan dan menjalankan program tersebut saja. Setelah selesai, klik output port dari modul “Evaluate Model” dan pilih visualize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*UW18lWjUKnGnGb3U6vLCZQ.png" /></figure><p>Berdasarkan gambar dari ROC Curve tersebut, dapat dilihat bahwa model curve model condong ke arah kiri atas yang mana hal ini dapat dikatakan bahwa model dianggap bagus.</p><p>Untuk melihat false positive rate dan juga tabel score bin kita dapat scroll saja kebawah.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WrJqUmFO_k1FAiIsLbuHCg.png" /></figure><p><strong>Membuat Web Service</strong></p><p>Agar model tersebut dapat digunakan, maka harus terdapat API (Application Programming Interface) terlebih dahulu dari model tersebut ke aplikasi. Salah satu mekanisme yang paling populer adalah web service. Nah, Azure Machine Learning menyediakan fitur tersebut dengan caranya yaitu klik “Set Up Web Service” dan pilih “Predictive Web Service”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*6pZBfjOqpHy6bj1Ga6Fv4g.png" /></figure><p>Setelah beberapa saat akan muncul tab baru yang bernama “Predictive Experiment” dengan tampilan susunan modul yang berbeda dari sebelumnya. Tambahan dari modul di tab baru ini adalah tambahan modul “Web Service Input” yang akan menggantikan “blank-data.csv” dan juga “Web Service Output”.</p><p><strong>Deploy Web Service</strong></p><p>Sekarang kita sudah siap untuk melakukan deploy web service untuk dapat digunakan lebih lanjut, kita dapat menjalankannya dengan me<em>running </em>predictive experiment tersebut sekali saja. Setelah selesai, klik tombol “Deploy Web Service” pada dashboard bagian bawah. Maka akan keluar output seperti berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BPxgxIKfXbS5MrIqPxIXqg.png" /></figure><p>Dapat dilihat dari gambar diatas bahwa terdapat detail API key dan link untuk menggunakan jenis servisnya, apakah akan melakukan satu persatu (request/response) atau dalam kelompok besar data (batch). Nah tetapi pada studi kasus ini kita akan menggunakan form test yang langsung disediakan oleh Azure Machine Learning. Klink link test (bukan tombol warna biru test) yang ada di request/response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oCIlBw2T0kREq666qP0pKA.png" /></figure><p>Maka tidak lama kita akan dibawa ke tab baru yang salah satu isinya adalah form analisis yang akan kita coba. Mari kita coba memasukkan data-data berdasarkan kolom sebelumnya. Setelah semua data sudah dimasukkan kita tinggal klik “Test request-response”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*E3AlUboZBtvIbo9dwE_fgg.png" /></figure><p>Maka dengan data seperti diatas, kemungkinan besar keputusan approvenya adalah tidak disetujui.</p><p><strong>Web Service pada Excel</strong></p><p>Untuk pengguna yang terbiasa dengan Excel, maka dari Azure sudah menyiapkan Web Service yang terhubung dengan Excel, sehingga akan mempermudah pengguna yang terbiasa dengan Excel. Pengguna tinggal memasukkan data-data yang akan dianalisis dan nantinya akan teranalisis secara otomatis.</p><p>Caranya adalah kita kembali ke halam Web Service, download file Excel untuk tipe API Request/Response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YI2yYvW4Je28QluEVejXmA.png" /></figure><p>Setelah kita selesai mengunduh file tersebut, kita buka filenya. Sebelumnya kita harus mengaktifkan makro untuk dapat membuka file tersebut.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cvFZDnMwaJvB2YyTKPA1zQ.png" /></figure><p>Kita tinggal memasukkan data yang ada atau memasukkan percobaan data untuk mengetahui hasil prediksi disetujui atau tidaknya credit berdasar data tersebut.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdD6oXBMdhxs4rzea40fDg.png" /></figure><p>Berdasarkan data yang saya masukkan tersebut, dapat dilihat bahwa prediksi hasil pembiayaannya adalah disetujui.</p><p>Berdasarkan studi kasus pada kesempatan kali ini, saya menyimpulkan bahwa Microsoft Azure Machine Learning sangat bermanfaat untuk perhitungan approval credit atau pembiayaan berdasarkan data yang sudah ada. Tetapi memang perlu training dulu dan juga testing. Tetapi saya rasa dengan bertambah banyaknya permintaan pembiayaan sekarang, dibutuhkan kecepatan analisis untuk menyelesaikan permasalahan tersebut.</p><p>Cukup sekian dari saya, jika ada salah ilmu, salah kalimat atau kata saya memohon maaf. Saya sangat terbuka untuk koreksi dari apa yang saya tuliskan.</p><p>Wassalamualaikum Wr. Wb.</p><p>========================================</p><p>Thanks to DQLab.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c77dd9b189c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Membuat Database Sederhana Menggunakan MySQL]]></title>
            <link>https://medium.com/@akmalabdillah/membuat-database-sederhana-menggunakan-mysql-64d3647c7f2d?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/64d3647c7f2d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Sun, 17 Oct 2021 08:12:13 GMT</pubDate>
            <atom:updated>2021-10-17T08:42:59.236Z</atom:updated>
            <content:encoded><![CDATA[<p>MySQL #1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pZcKjsWKYQ7_pD0V_FrOtg.jpeg" /></figure><p>Assalamualaikum Wr. Wb.</p><p>Pada kesempatan kali ini, saya akan menuliskan bagaimana membuat database sederhana menggunakan MySQL. <em>Database</em> atau <em>basis data</em> adalah kumpulan <em>data</em> yang disimpan secara <em>sistematis</em> di dalam komputer yang dapat diolah atau dimanipulasi menggunakan perangkat lunak (<em>program aplikasi</em>) untuk menghasilkan informasi. Pendefinisian <em>basis data</em> meliputi <em>spesifikasi</em> berupa <em>tipe</em> <em>data</em>, struktur <em>data</em> dan juga batasan-batasan <em>data</em> yang akan disimpan. <em>Basis</em> <em>data</em> merupakan aspek yang sangat penting dalam sistem informasi dimana <em>basis</em> <em>data</em> merupakan gudang penyimpanan <em>data</em> yang akan diolah lebih lanjut. <em>Basis</em> <em>data</em> menjadi penting karena dapat mengorganisasi <em>data</em>, menghindari duplikasi <em>data</em>, hubungan antar <em>data</em> yang tidak jelas dan juga <em>update</em> yang rumit.</p><p>Proses memasukkan dan mengambil <em>data</em> ke dan dari media penyimpanan data memerlukan perangkat lunak yang disebut dengan sistem manajemen <em>basis</em> <em>data</em> (<em>database</em> <em>management</em> <em>system</em> | <em>DBMS</em>). <em>DBMS</em> merupakan sistem perangkat lunak yang memungkinkan pengguna <em>database</em> (<em>database</em> <em>user</em>) untuk memelihara, mengontrol dan mengakses <em>data</em> secara praktis dan <em>efisien</em>. Dengan kata lain semua akses ke <em>basis</em> <em>data</em> akan ditangani oleh <em>DBMS</em>. Ada beberapa fungsi yang harus ditangani <em>DBMS</em> yaitu mengolah pendefinisian <em>data</em>, menangani permintaan pemakai untuk mengakses <em>data</em>, memeriksa <em>security </em>dan <em>integrty</em> <em>data</em> yang didefinisikan oleh <em>DBA</em> (<em>Database</em> <em>Administrator</em>), menangani kegagalan dalam pengaksesan data yang disebabkan oleh kerusakan sistem maupun <em>disk</em> dan menangani unjuk kerja semua fungsi secara <em>efisien</em>. Contoh dari DBMS adalah MySQL, PostgreSQL, MongoDB, Oracle dsb. Pada kesempatan kali ini saya akan menggunakan MySQL.</p><p>MySQL adalah <a href="https://www.merdeka.com/cari/?q=sistemmanajemen">sistem manajemen</a> database relasional yang berbasis SQL (Structured Query Language). MySQL bisa digunakan untuk berbagai tujuan, termasuk sebagai gudang data, e-commerce, dan aplikasi pencatatan. (<a href="https://www.merdeka.com/jabar/mysql-adalah-sistem-manajemen-berbasis-data-ketahui-cara-kerjanya-kln.html#:~:text=MySQL%20adalah%20sistem%20manajemen%20database,%2Dcommerce%2C%20dan%20aplikasi%20pencatatan.">Merdeka.com</a>)</p><p>Pertama-tama untuk membuat sebuah database menggunakan MySQL, kita harus mendownload di situs resmi dari MySQL, yaitu mysql.com. Kita dapat mengunduhnya sesuai dengan perangkat yang digunakan. Setelah itu kita dapat menginstall dan mengkonfigurasikannya. Setelah kita menginstal, kita dapat menggunakannya dengan menggunakan terminal maupun dengan perangkat lain yang sudah disediakan oleh MySQL, yaitu MySQL Workbench. MySQL Workbench dapat diunduh pada website resmi dari MySQL. Sebenarnya sama saja kegunaannya, tetapi mungkin lebih simpel menggunakan MySQL Workbench. Maka dari itu, pada kesempatan kali ini saya akan menggunakan MySQL Workbench dengan sesekali mengecek menggunakan terminal.</p><p>Langkah pertama untuk membuat database menggunakan MySQL terutama dengan menggunakan terminal adalah login terlebih dahulu sesuai dengan username dan juga password yang sudah dikonfigurasikan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*67ixVClvu5cY31GfVr1k4A.png" /><figcaption>Tampilan Awal MySQL Workbench</figcaption></figure><p>Jika teman-teman ingin membukanya menggunakan terminal, tinggal ketikkan <em>cmd </em>selanjutnya ketikkan <em>mysql -uroot -p </em>dan selanjutnya dapat memasukkan passwordnya saja.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*Sc2iLPALOUbuI0fbmvahjA.png" /><figcaption>Membuka MySQL menggunakan Terminal</figcaption></figure><p>Sekarang kita dapat membuat database dengan menggunakan <em>syntax;</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*knR678M2E_hu-GizfY14mQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*XD2d_giiQ4mCsBThLfS9mA.png" /><figcaption>Database</figcaption></figure><p>Pada kesempatan kali ini, saya membuat sebuah database pengiriman paket yang bernama safe_express. Untuk penamaan database tidak diperbolehkan menggunakan ‘spasi’, maka dari itu dapat menggunakan underscore atau simbol lain untuk menggantinya. <em>Syntax </em>untuk melihat database yang ada adalah <em>show databases; . </em>Jangan lupa untuk setiap akhiran dari perintah, kita harus menambahkan “;”. Untuk mengaktifkan <em>database </em>yang akan kita gunakan, kita dapat menggunakan perintah: “ <em>use safe_express;</em> ”</p><p>Selanjutnya, kita tinggal membuat tabel yang nantinya akan kita gunakan untuk me<em>manage </em>data yang ada. Kolom maupun garis pada tabel disini kondisional tergantung dari kebutuhan masing-masing user. Untuk <em>syntax</em>nya sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*6sM6H4cwQcegzcbFAjE3nw.png" /><figcaption>Membuat Tabel</figcaption></figure><p>Untuk mengecek apakah tabel kita masuk atau tidak, kita dapat melihatnya menggunakan perintah <em>show tables; .</em></p><p>Pada pembuatan tabel tersebut, saya menamainya sebagai tabel kurir. Tabel kurir tersebut berisikan beberapa kolom, yaitu kolom id, nama depan, nama belakang, nomor handphone, status kurir yang dimaksudkan untuk menampung data dari kurir sebuah perusahaan pengiriman paket. Pada kolom <strong>id </strong>saya menggunakan tipe data <em>integer </em>untuk nantinya data yang masuk adalah data numerik bulat. Pada kolom id juga saya cantumkan <em>auto increament </em>untuk mengurutkan id tersebut agar tidak menuliskan secara manual, dan saya juga memberi <em>primary key </em>pada kolom id yang digunakan sebagai identitas dari tiap baris yang ada berdasarkan id dari masing-masing. Untuk kolom <strong>first_name </strong>dan <strong>last_name </strong>saya menggunakan <em>varchar(100) </em>karena data nantinya berbentuk string yang batasnya adalah masing-masing 100 karakter. Keterangan <em>not null </em>tersebut agar kolom tersebut tidak boleh kosong. Pada kolom <strong>status</strong>, saya menggunakan tipe data <em>enum</em>, karena analogi pemilihannya adalah status kurir masih proses mengantar atau masih stay di kantor. Pada kolom <strong>no_hp </strong>saya menambahkan <em>unique constraint </em>agar data nomor handphone dari kurir tidak ada yang sama, jika pada nantinya ada kasus insert data nomor handphone sama, maka MySQL akan menolak data tersebut. Pada akhir pembuatan tabel, kita harus menentukan <em>storage </em>yang akan kita gunakan untuk menampung data-data yang nantinya akan dimasukkan kedalam tabel tersebut. Pada MySQL sebenarnya terdefault enginenya adalah InnoDB untuk versi MySQL 5.6 keatas. Untuk versi MySQL 5.6 kebawah masih menggunakan engine MyISAM. Perbedaannya adalah jika InnoDB implementasi pengunciannya adalah level baris atau <em>row-level lock</em> sedangkan jika MyISAM implementasi pengunciannya adalah level tabel atau <em>table-level lock</em>.</p><p>Jika teman-teman ingin melihat deskripsi dari tabel yang kita buat, dapat menggunakan perintah <em>desc nama_table; </em>atau pada kasus ini <em>desc kurir; </em>dan hasilnya adalah:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*vFor2ZTJ7gAWXGZ-KJnsuA.png" /><figcaption>Deskripsi Tabel</figcaption></figure><p>Jika teman-teman ingin melihat <em>query </em>yang telah digunakan untuk membuat sebuah tabel, dapat menggunakan perintah <em>show create table nama_table; </em>atau pada kasus ini adalah <em>show create table kurir;</em> nantinya akan keluar hasil sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*0TWr_3Tob4VjCH2-WvNuRg.png" /><figcaption>Show Create Table Kurir</figcaption></figure><p>Selanjutnya kita dapat meng<em>insert </em>data kedalam tabel tersebut. Untuk <em>insert </em>data kita dapat menggunakan perintah <em>insert into</em>. Untuk lebih jelasnya dapat dilihat sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*ecQe6sOe4X9SuWqJ05BtUg.png" /><figcaption>Insert Data</figcaption></figure><p>Pada gambar tersebut, saya memasukkan data berdasarkan nama kolom yang saya tuliskan pada perintah, sebenarnya penulisan nama kolomnya tidak berurutan juga tidak masalah, asalkan nantinya penulisan valuenya mengikuti penulisan nama kolom yang ada diatasnya. Teman-teman juga dapat memasukkan perbaris tidak perlu langsung beberapa baris seperti gambar diatas, asalkan nantinya simbol untuk akhir bukan koma (,) tapi titik koma (;). Untuk data <em>integer</em>, sebenarnya tidak perlu menggunakan tanda petik pada penulisan datanya.</p><p>Untuk teman-teman yang ingin melihat tampilan data pada tabelnya, dapat menggunakan perintah “ <em>select * from kurir; </em>“. Bintang tersebut menunjukkan semua kolom yang ada nantinya akan ditampilkan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*D3uE3RwRXFzDlcZJxv-sVQ.png" /><figcaption>Hasil Insert Data</figcaption></figure><p>Untuk penggunaan menggunakan terminal juga sama saja, tinggal kita ketikkan perintah yang kita inginkan dengan menggunakan logika MySQL. Sebagai contoh saya akan melihat tampilan data yang ada diatas menggunakan terminal, maka kita ketikkan “ <em>select * from kurir; </em>” di terminal juga, maka akan muncul hasil sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*4AiJv1i1fuog28ObqOJgZA.png" /><figcaption>Tampilan Data pada Terminal</figcaption></figure><p>Teman-teman juga dapat melihat berdasarkan beberapa kolom saja, misal kita hanya akan melihat nama depan, nama belakang dan statusnya saja. Kita dapat menuliskan perintah “ <em>select first_name, last_name, status from kurir;</em> ” maka hasilnya sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*C5axMS1G-zKLTFCOJNH9kw.png" /><figcaption>Hasil Insert Data 2</figcaption></figure><p>Ya, jadi seperti itu pembuatan database yang sederhana. Nantinya akan saya share lagi beberapa kasus pada database. Nantikan pada pembahasan selanjutnya. Jika ada kekurangan baik penulisan maupun ilmu yang ada, teman-teman dapat kontak saya langsung melalui beberapa media sosial yang ada. Sampai jumpa pada pembahasan selanjutnya. Terima kasih.</p><p>Wassalamualaikum Wr. Wb.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64d3647c7f2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Penentuan Harga Call Option Saham dengan Pemodelan Black-Scholes Option Pricing Model menggunakan…]]></title>
            <link>https://medium.com/@akmalabdillah/penentuan-harga-call-option-saham-dengan-pemodelan-black-scholes-option-pricing-model-menggunakan-a9c1342ed9b1?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9c1342ed9b1</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[pasar-modal]]></category>
            <category><![CDATA[saham]]></category>
            <category><![CDATA[black-scholes-model]]></category>
            <category><![CDATA[economics]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Sun, 29 Aug 2021 08:03:15 GMT</pubDate>
            <atom:updated>2021-08-29T08:03:15.534Z</atom:updated>
            <content:encoded><![CDATA[<h3>Penentuan Harga <em>Call Option Saham dengan Pemodelan Black-Scholes Option Pricing Model menggunakan Software R</em></h3><p>Saham merupakan instrumen keuangan yang lebih dikenal masyarakat daripada jenis instrumen keuangan lainnya. Saham menunjukkan bukti kepemilikan seorang atas perusahaan yang menjual sahamnya. Investasi dalam bentuk saham mempunyai peluang memiliki untung dalam bentuk <em>capital gain</em> dan <em>dividen</em>. <em>Capital gain</em> merupakan keuntungan yang diperoleh dari selisih antara harga penjualan dan harga beli saham. <em>Capital gain</em> terbentuk karena adanya perubahan nilai harga saham.</p><p>Opsi (<em>option</em>) adalah sekuritas yang memberikan hak bukan kewajiban kepada pembelinya untuk membeli atau menjual suatu komoditas (sekuritas atau asset riil) dengan harga tertentu pada saat atau sebelum tanggal jatuh tempo opsi tersebut (Arifin, Teori Keuangan &amp; Pasar Modal, 2005). Berdasarkan fungsinya, opsi terbagi menjadi 2 yaitu opsi beli (<em>call option</em>) dan opsi beli (<em>put option</em>). Opsi beli (<em>call option</em>) adalah hak untuk membeli sebuah asset dengan harga tertentu pada saat atau sebelum jatuh tempo yang telah disepakati bersama. Opsi jual (<em>put option</em>) adalah hak untuk menjual sebuah asset dengan harga tertentu pada saat atau sebelum jatuh tempo yang telah disepakati bersama. Sedangkan jika berdasarkan periode waktu pelaksanaannya, maka opsi dibagi menjadi 2 yaitu opsi tipe amerika (<em>American Option</em>) dan opsi tipe eropa (<em>European Option</em>). Opsi tipe amerika adalah opsi yang dapat dilaksanakan sewaktu-waktu sampai jatuh tempo berakhir. Sedangkan opsi tipe eropa adalah opsi yang dapat dilaksanakan pada saat jatuh tempo.</p><p>Harga opsi dapat dipengaruhi oleh beberapa faktor, seperti harga saham yang berubah-ubah, <em>exercise price, </em>waktu ke jatuh tempo, volatilitas, bunga bebas risiko maupun dividen, sehingga sulit untuk ditebak harganya agar keuntungan bagi pembeli opsi saham dapat maksimal. Maka dari itu, Fisher Black dan Myron Scholes membuat model untuk menyelesaikan permasalahan harga saham yang diberi nama <em>Black-Scholes Option Pricing Model</em>. Dasar pemikiran dari <em>Black-Scholes</em> yaitu investor dapat menciptakan portfolio bebas risiko dengan mengombinasikan opsi dengan saham yang diopsikan. Portfolio bebas risiko dapat dibentuk karena baik harga saham maupun harga opsi dipengaruhi oleh sumber ketidakpastian yang sama yaitu pergerakan harga saham (Arifin, Teori Keuangan &amp; Pasar Modal, 2005).</p><p>Dalam studi kasus kali ini, saya akan menggunakan contoh data saham AAPL dari tanggal 01–12–2020 sampai 28–02–2021 dengan salah satu contoh perdagangan opsi saham pada tanggal 18–01–2021 dan time to maturity datenya sampai 12–03–2021.</p><p>Langkah pertama yang dilakukan adalah pemanggilan data dari Yahoo! Finance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*b-VzlkGazD_pyael9i1guw.png" /><figcaption>Pemanggilan Data</figcaption></figure><p>Dari syntax tersebut saya menggunakan syntax <em>quantmod </em>untuk memanggil data saham dari Yahoo! Finance. Dengan ketentuan data dari AAPL dengan waktu dari 01–12–2020 sampai 28–02–2021.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*0ZQ6-id0tMU7FWrxZ_yksA.png" /><figcaption>6 Data Awal Saham AAPL</figcaption></figure><p>Selanjutnya untuk melihat plot pergerakan harga saham dan juga statistika deskriptifnya adalah sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*SSkBep7YQlDzF99xjRNnEQ.png" /><figcaption>Plot Harga Saham dan Statistika Deskriptif</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*tgNoXlszam8AsGmhIlN9pQ.png" /><figcaption>Plot Pergerakan Harga Saham</figcaption></figure><p>Dapat dilihat bahwa pergerakan harga saham <em>Apple Inc. </em>cenderung mengalami kenaikan dan juga penurunan. Selama bulan Desember 2020, harga saham dari <em>Apple Inc. </em>cenderung naik (<em>uptrend</em>). Selanjutnya dari tanggal 29 Desember 2020 sampai 19 Januari 2021 mengalami fluktuatif atau <em>sideways </em>dimana harga terkonsolidasi dalam <em>range </em>yang sempit. Selanjutnya setelah tanggal 19 Januari 2021, saham mengalami kenaikan signifikan ke titik tertinggi di angka 143,2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*xMeuMJlDb3_j3di_GbhTVw.png" /><figcaption>Statistika Deskriptif Saham AAPL</figcaption></figure><p>Berdasarkan output dari statistika deskriptif tersebut, dapat dilihat secara umum bahwa saham AAPL pernah berada pada titik terendah dalam 2 bulan ke poin 121 sedangkan titik tertingginya adalah 143,2. Untuk rata-rata harga sahamnya selama 2 bulan adalah 130,5. (Dalam US$)</p><p>Selanjutnya saya akan menghitung <em>return </em>saham dari AAPL. Pada kesempatan kali ini saya langsung menggunakan <em>package </em>dari R yaitu <em>periodReturn </em>dengan memilih <em>dailyReturn </em>dan tipenya arithmetic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*u-cjirF7F29tjOehpmnTIQ.png" /><figcaption>Sintaks <em>Return Saham</em></figcaption></figure><p>Hasil dari <em>return </em>sahamnya adalah sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/206/1*858yycaQWD55YWP7YbX07A.png" /><figcaption>6 Data Awal Return Saham</figcaption></figure><p>Untuk plot pergerakan <em>return </em>saham adalah sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*avsnVne6TxrV7_jMba7-9A.png" /><figcaption>Pergerakan <em>Return Saham</em></figcaption></figure><p>Berdasarkan gambar diatas dapat dilihat bahwa pergerakan <em>return </em>saham mengalami pergerakan yang berfluktuatif. Kadang bernilai positif kadang juga bernilai negatif. Jika <em>return </em>saham bernilai positif maka investor akan mengalami keuntungan sedangkan sebaliknya jika bernilai negatif maka investor mengalami kerugian.</p><p>Selanjutnya saya akan menghitung volatilitas dari data saham AAPL yang saya ambil tersebut. Ada beberapa langkah untuk mencari nilai volatilitas harga saham, yaitu:</p><ol><li><strong>Menghitung rata-rata <em>return </em>saham</strong></li></ol><p>Untuk perhitungan rata-rata <em>return </em>saham dalam R dengan menggunakan <em>syntax “meanaapl=mean(returnaapl)”</em>. Maka didapat nilai rata-rata <em>return </em>saham adalah <strong>0,0002085013.</strong></p><p><strong>2. Menghitung standar deviasi dari <em>return </em>saham</strong></p><p>Untuk perhitungan standar deviasi dari <em>return </em>saham, dapat menggunakan <em>syntax “sdaapl=sd(returnaapl)”</em>. Maka didapatkan nilai standar deviasinya adalah <strong>0,01884688.</strong></p><p><strong>3. Menghitung volatilitas saham</strong></p><p>Setelah memperoleh hasil standar deviasi dari <em>return </em>saham, maka dapat dimasukkan ke persamaan untuk perhitungan volatilitas saham. Rumus untuk perhitungan volatilitas saham adalah sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/107/1*55pruhIYko4zIGO5kSOONA.png" /><figcaption>Rumus Volatilitas Saham</figcaption></figure><p>Dengan nilai <em>tau</em>nya adalah 252 atau hari perdagangan. Maka perhitungan menggunakan <em>software </em>R adalah sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*dedjS6Y9VFuX1MfvpB6MQw.png" /><figcaption>Sintaks Volatilitas Saham</figcaption></figure><p>Dengan menggunakan sintaks tersebut, maka volatilitas sahamnya adalah <strong>0,299185</strong> atau sebesar <strong>29,92%</strong>. Ketika nilai volatilitas saham masih dibawah 40% maka perubahan harga sahamnya belum terlalu besar.</p><p>Selanjutnya tinggal masukkan beberapa nilai kedalam pemodelan <em>Black-Scholes Option Pricing Model</em>. Untuk perumusan <em>Black-Scholes OPM </em>adalah sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*B2Y3wkxprq4n-APOxwOhLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*bxS3QKgAZZyN7AIFV3TFMw.png" /></figure><p>Pada kesempatan kali ini, saya menggunakan suku bunga (r) harian adalah <strong>0,00122</strong>. Harga saham aset induknya (S0) adalah pada saat pembukaan <strong>127,14</strong>. Jangka waktunya (T) adalah <strong>0,1452 </strong>dalam tahunan. <em>Strike Price </em>(K) adalah <strong>118</strong>.</p><p>Pertama-tama saya harus mencari nilai <strong>d1 </strong>dan juga <strong>d2 </strong>dengan menggunakan sintaks sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*pwxpdx0k2CqSK_g0QNkvMQ.png" /><figcaption>Sintaks d1 dan d2</figcaption></figure><p>Hasil dari d1 adalah <strong>0,7220991</strong> dan nilai d2nya adalah <strong>0,8039652</strong>. Selanjutnya saya akan menghitung distribusi kumulatif dari d1 dan d2nya.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/163/1*kvl1VzfHnvghyXwdaE9F6Q.png" /><figcaption>Peluang Kumulatif Distribusi Normal</figcaption></figure><p>Hasil dari N(d1) adalah <strong>0,7648832 </strong>dan hasil N(d2) adalah <strong>0,7892915</strong>.</p><p>Selanjutnya ke tahapan akhir tinggal masukkan ke perhitungan <em>Black-Scholes OPM </em>nilai-nilai yang sudah didapat diatas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*vms5Orcb3XUWIgwyaJWVUQ.png" /><figcaption>Perhitungan <em>Black-Scholes OPM</em></figcaption></figure><p>Didapatkan dengan menggunakan perhitungan teoritis <em>Black-Scholes OPM </em><strong>4,127356 US$</strong>. Nah, tinggal investor membandingan dengan nilai di pasaran, jika nilai dipasaran lebih rendah dari nilai perhitungan tersebut, maka direkomendasikan untuk dibeli. Tetapi, jika harga di pasar lebih tinggi, maka investor harus mengkaji dan mempertimbangkan ulang lagi sebelum memutuskan untuk membeli. Entah melanjutkan dengan analisis fundamental maupun analisis yang lain.</p><p>========================================</p><p>Cukup Sekian dari Saya, Jika ada kesalahan penulisan maupun salah metode bisa kontak saya saja langsung.</p><p>Email: akmalmvic0312@gmail.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9c1342ed9b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Peramalan Jumlah ‘Produksi Ikan’ di Kota Pekalongan menggunakan Seasonal ARIMA]]></title>
            <link>https://medium.com/@akmalabdillah/peramalan-jumlah-produksi-ikan-di-kota-pekalongan-menggunakan-seasonal-arima-fff198bfd8a9?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/fff198bfd8a9</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[fishing]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[arima]]></category>
            <category><![CDATA[sarima]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Tue, 21 Jan 2020 17:27:28 GMT</pubDate>
            <atom:updated>2020-01-21T17:27:28.440Z</atom:updated>
            <content:encoded><![CDATA[<h3>بِسْمِ اللَّهِ الرَّحْمَنِ الرَّحِيم</h3><p><strong>Assalamualaikum Wr. Wb.</strong></p><p>Pada kesempatan kali ini, saya ingin menulis hasil penelitian saya di Kota Pekalongan tentang prediksi produksi ikan di Kota Pekalongan pada bulan Januari-Mei 2019.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*HI00a80_cPTn8zfJNV_BUQ.jpeg" /></figure><p>Perikanan adalah kegiatan manusia yang berhubungan dengan pengelolaan dan pemanfaatan sumberdaya hayati perairan. Sumberdaya hayati perairan tidak dibatasi secara tegas dan pada umumnya mencakup ikan, amfibi, dan berbagai avertebrata penghuni perairan dan wilayah yang berdekatan, serta lingkungannya. Di Indonesia, menurut UU RI №31/2004, sebagaimana telah diubah dengan UU RI №45/2009, kegiatan yang termasuk dalam perikanan dimulai dari praproduksi, produksi, pengolahan sampai dengan pemasaran, yang dilaksanakan dalam suatu sistem bisnis perikanan. Dengan demikian, perikanan dapat dianggap merupakan usaha agribisnis. (Wikipedia, 2018)</p><p><strong>Konsep dan Definisi</strong></p><p><strong>Perikanan tangkap</strong> adalah semua kegiatan yang memanfaatkan sumberdaya laut, berupa ikan, tanaman laut dan berbagai sumberdaya yang dapat diambil untuk dimanfaatkan.</p><p><strong>Alat tangkap perikanan</strong> adalah semua jenis peralatan yang digunakan dalam mengambil sumberdaya laut berupa ikan/binatang/tanaman dan sejenisnya, baik berupa alat tangkap manual maupun yang menggunakan mesin-mesin atau media lainnya.</p><p><strong>Produksi tangkap</strong> mencakup semua hasil tangkap ikan/binatang laut lainnya/tanaman laut yang diperoleh dari sumber perikanan laut.</p><p><strong>Volume produksi atau berat produksi</strong>, yang dimaksud adalah berat basah pada waktu hasil panen didaratkan, jadi hasil panen tesebut didaratkan.</p><p><strong>Nilai produksi</strong> yang dimaksud adalah nilai rupiah pada waktu hasil didaratkan, jadi harga yang digunakan adalah harga produsen.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Pada kesempatan kali ini, berdasarkan pola data dan karakteristiknya saya menggunakan metode <em>Seasonal Autoregressive Integrated Moving Average (SARIMA) </em>dikarenakan data mengandung musiman.</p><p>Dalam kesempatan ini saya melakukan peramalan menggunakan <em>software R Studio</em> yaitu metode <em>Seasonal ARIMA</em>. Untuk melakukan peramalan penulis pertama meng<em>input</em>kan data yang akan diteliti. Dengan menggunakan metode <em>SARIMA</em> kemudian melihat plot dari data. Selanjutnya penulis akan melakukan uji stasioneritas data. Selanjutnya jika belum juga stasioner, dilakukan <em>differencing</em>. Data yang digunakan peneliti yaitu <em>differencing </em>2 untuk mencapai stasioner. Selanjutnya penulis akan melakukan pemodelan. Setelah mendapatkan model maka peneliti kemudian dapat melakukan <em>forecasting</em> selama 5 bulan pada tahun 2019. Setelah mendapatkan hasil peramalan kemudian peneliti melakukan evaluasi <em>error</em> dengan melihat nilai MAD, MAPE , MSE dan RMSE.</p><p><strong>Analisis Deskriptif</strong></p><p>Analisis deskriptif pada kasus ini dilakukan untuk melihat gambaran umum pergerakan jumlah produksi ikan di Kota Pekalongan pada tahun 2013–2018. Pada analisis deskriptif ini peneliti menghitung nilai jumlah, rata-rata produksi ikan di Kota Pekalongan tahun 2013–2018.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*l6wpmXULbPW-9RZKXwg72g.png" /><figcaption>Jumlah Produksi dan Rata-rata (Produksi Perikanan Kota Pekalongan)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*-U4yHO9E03RufBo9_An7mA.png" /><figcaption>Grafik Rata-Rata Produksi Ikan Kota Pekalongan</figcaption></figure><p>Berdasarkan tabel dan grafik diatas dapat diketahui bahwa jumlah produksi ikan di Kota Pekalongan pada tahun 2013–2018 cenderung stagnan setiap tahunnya. Peningkatan paling tinggi pada tahun 2018 yaitu dengan jumlah penumpang 25.565 Ton, dengan rata-rata 2130,42 Ton.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*tO8gZhjAJuLaO7wyll9PQg.png" /><figcaption>Plot Data Asli</figcaption></figure><p>Berdasarkan gambar diatas, dapat penulis simpulkan bahwa data tersebut mengandung musiman. Karena pada awal tahun akan terjadi paceklik penangkapan ikan atau produksi ikan, sedangkan pada triwulan terakhir akan terjadi lonjakan produksi ikan.</p><p><strong>Cek Stasioner terhadap Variansi dan Mean</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*zin8XmLop50pwdHGEyNadQ.png" /><figcaption>Plot <em>ACF </em>dan <em>PACF </em>Awal</figcaption></figure><p>Berdasarkan plot dapat diketahui bahwa lag cenderung keluar, jadi dapat dikatakan data tidak stasioner. Namun, jika dilihat dengan visualisasi saja, mungkin kurang untuk membuktikannya. Maka, untuk melihat stasioneritas variansi dan <em>mean, </em>dapat dilakukan dengan melakukan uji ADF sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*OpcjZAGiG3QfqJ8x_wyjAw.png" /><figcaption><em>Augmented Dickey-Fuller Test</em></figcaption></figure><p>· Hipotesis</p><p>H0 : Data mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>H1 : Data tidak mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji</p><p><em>p-value </em>= 0,3058</p><p>· Keputusan</p><p>Gagal tolak H0, karena <em>p-value </em>= 0,3058 &gt; α = 0,05</p><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, data yang ada mendukung hipotesis nol yang artinya bahwa data jumlah produksi ikan Kota Pekalongan tidak stasioner dalam <em>mean</em>.</p><p>Selain dengan uji ADF, untuk menguji stasioneritas data bisa dengan uji KPSS, seperti berikut :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*0wbtTSTIdtPfGCCkA-H66g.png" /><figcaption>KPSS <em>Test</em></figcaption></figure><p>· Hipotesis</p><p>H0 : Data mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>H1 : Data tidak mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji</p><p><em>p-value </em>= 0,1</p><p>· Keputusan</p><p>Gagal tolak H0, karena <em>p-value </em>= 0,1 &gt; α = 0,05</p><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, data yang ada mendukung hipotesis nol yang artinya bahwa data jumlah produksi ikan Kota Pekalongan stasioner dalam <em>mean</em>.</p><p>Setelah pengujian stasioneritas data menggunakan uji <em>ADF </em>dan uji <em>KPSS </em>mempunyai hasil yang tidak konsisten, maka penulis melakukan pengujian stasioneritas kembali dengan cara <em>differencing </em>untuk musimannya terlebih dahulu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*IuscPUy0f5ePbCUtLCfFLA.png" /><figcaption>Plot Data Diferensi Musiman</figcaption></figure><p>Dari gambar diatas dengan visualisasi mungkin bisa dikatakan bahwa data yang ada sudah stasioner. Dengan demikian, praktikan melakukan pengujian stasioneritas dengan uji <em>ADF </em>dan uji <em>KPSS. </em>Diperoleh <em>output </em>sebagai berikut untuk uji <em>ADF</em> :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*Y0xnt4hpF4F7PyHM-_p26A.png" /><figcaption><em>Augmented Dickey-Fuller Test </em>Diferensi Musiman</figcaption></figure><p>· Hipotesis</p><p>H0 : Data mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>H1 : Data tidak mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji</p><p><em>p-value </em>= 0,5393</p><p>· Keputusan</p><p>Gagal tolak H0, karena <em>p-value </em>= 0,5393 &gt; α = 0,05</p><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, data yang ada mendukung hipotesis nol yang artinya bahwa data jumlah produksi ikan Kota Pekalongan tidak stasioner dalam <em>mean</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*l0yHbEbnbBsgS4Dzx12kog.png" /><figcaption>KPSS <em>Test</em> Diferensi Musiman</figcaption></figure><p>· Hipotesis</p><p>H0 : Data mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>H1 : Data tidak mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji</p><p><em>p-value </em>= 0,1</p><p>· Keputusan</p><p>Gagal tolak H0, karena <em>p-value </em>= 0,1 &gt; α = 0,05</p><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, data yang ada mendukung hipotesis nol yang artinya bahwa data jumlah produksi ikan Kota Pekalongan stasioner dalam <em>mean</em>.</p><p>Setelah pengujian stasioneritas data menggunakan uji <em>ADF </em>dan uji <em>KPSS </em>hasilnya tetap tidak konsisten, maka penulis melakukan pengujian stasioner kembali dengan cara diferensi untuk non musimannya.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/1*surLk3s3CPoJ2_FOX44d6Q.png" /><figcaption>Plot Data Diferensi Non Musiman</figcaption></figure><p>Dari gambar diatas, dengan visualisasi mungkin bisa dikatakan data yang ada sudah stasioner. Dengan demikian, penulis melakukan pengujian stasioneritas dengan uji ADF dan uji KPSS. Diperoleh <em>output</em> sebagai berikut untuk uji ADF:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*JR6vMNdkMtwh_O103-fscw.png" /><figcaption><em>Augmented Dickey-Fuller Test </em>Diferensi Non Musiman</figcaption></figure><p>· Hipotesis</p><p>H0 : Data mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>H1 : Data tidak mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji</p><p><em>p-value </em>= 0,01</p><p>· Keputusan</p><p>Gagal tolak H0, karena <em>p-value </em>= 0,01 &lt; α = 0,05</p><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, data yang ada mendukung hipotesis nol yang artinya bahwa data jumlah produksi ikan Kota Pekalongan stasioner dalam <em>mean</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*UuYLvy4hS29yPCpfNG6raQ.png" /><figcaption>KPSS <em>Test </em>Diferensi Non Musiman</figcaption></figure><p>· Hipotesis</p><p>H0 : Data mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>H1 : Data tidak mengandung <em>unit root </em>(Data tidak stasioner dalam <em>mean</em>)</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji</p><p><em>p-value </em>= 0,1</p><p>· Keputusan</p><p>Gagal tolak H0, karena <em>p-value </em>= 0,1 &gt; α = 0,05</p><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, data yang ada mendukung hipotesis nol yang artinya bahwa data jumlah produksi ikan Kota Pekalongan stasioner dalam <em>mean</em>.</p><p>Berdasarkan plot, uji ADF dan uji KPSS bahwa data yang ada sudah stasioner.</p><p><strong>Identifikasi Model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/1*ModOJmtWmDQrILrrPZFeVQ.png" /><figcaption>Plot ACF dan PACF Data Non Musiman</figcaption></figure><p>Berdasarkan plot ACF diatas, diketahui bahwa dari 4 garis vertikal awal terdapat 1 garis saja yang melewati batasan garis putus-putus horizontal dan di <em>lag</em> ke 2, sehingga didapatkan bahwa batas atas nilai MA(q) yang akan dilakukan <em>overfitting</em> yaitu 2 atau MA(2). Sedangkan untuk plot PACF, diketahui bahwa dari 4 garis vertikal awal terdapat 2 garis yang melewati batasan garis putus-putus hotizontal dan yang terakhir keluar adalah di <em>lag</em> ke 3, sehingga didapatkan bahwa batas atas nilai AR(p) yang akan dilakukan <em>overfitting</em> yaitu 3 atau AR(3).</p><p>Berdasarkan plot ACF yang sama, diketahui bahwa dari tidak ada garis vertikal berkelipatan 10 yang melewati batasan garis putus-putus horizontal, sehingga didapatkan bahwa batas atas nilai SMA(Q) yang akan dilakukan <em>overfitting</em> yaitu 0 atau SMA(0). Sedangkan untuk plot PACF, diketahui bahwa dari beberapa garis vertikal yang berkelipatan 10 tidak ada yang melewati batasan garis putus-putus horizontal, sehingga didapatkan bahwa batas atas nilai SAR(P) yang akan dilakukan <em>overfitting</em> yaitu 0 atau SAR(0). Sehingga didapatkan model utama yaitu <em>SARIMA(3,2,2)(0,2,0). </em>Selain model utama yang didapatkan, dapat diduga beberapa model yang mungkin disekitar model utama, oleh karena itu dilakukan <em>overfitting </em>pada model.</p><p><strong>Estimasi dan Overfitting Model</strong></p><p>Berikut adalah <em>output </em>untuk beberapa model yang mungkin disekitar model utama.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*UPIwZwwnNsNgPR2bQ86BOg.png" /><figcaption>Model Selain Model Utama</figcaption></figure><p>Berikut adalah analisis uji kecocokan untuk model yang telah dibuat.</p><p>· Hipotesis</p><p>H0 : Koefisien-koefisien tidak signifikan terhadap model.</p><p>H1 : Koefisien-koefisien signifikan terhadap model.</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji dan Keputusan</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*bbbYI1yM4wHFbN1iofZVVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*QOWS-qsT2gbWb-mk9UWMFg.png" /><figcaption>Overfitting Model</figcaption></figure><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, dapat disimpulkan bahwa SARIMA(4,2,0)(0,1,1)[12], SARIMA(4,2,0)(0,2,1)[12], dan SARIMA(3,2,0)(1,2,0)[12] tolak H0 sehingga model ini memiliki koefisien yang signifikan sementara model lainnya tidak.</p><p><strong>Diagnostic Check</strong></p><p>Setelah menemukan model ARIMA yang memiliki koefisien signifikan untuk semua variabel, kemudian penulis melakukan <em>diagnostic</em> <em>check</em> untuk model yang signifikan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*3Q2l45JTvq6WA0OHmJ8uVw.png" /><figcaption><em>Diagnostic Check </em>SARIMA(4,2,0)(0,1,1)[12]</figcaption></figure><p>Berdasarkan gambar diatas, dapat dilihat pada grafik ACF bahwa terdapat tidak lebih dari 1 lag yang keluar dari batas interval sehingga asumsi <em>white</em> <em>noise</em> terpenuhi. Kemudian nilai p-value pada <em>L-jung Box</em> juga diatas batas 5% yang berarti bahwa asumsi non-autokorelasi pada model 3 terpenuhi atau tidak ada autokorelasi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*JXmabOvIPfY_vdxg3xab0A.png" /><figcaption><em>Output Jarque Bera Test </em>SARIMA(4,2,0)(0,1,1)[12]</figcaption></figure><p>· Hipotesis</p><p>H0 : Data <em>residual </em>berdistribusi normal</p><p>H1 : Data <em>residual </em>tidak berdistribusi normal</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji</p><p><em>p-value </em>= 0,4592</p><p>· Keputusan</p><p>Gagal tolak H0, karena <em>p-value </em>= 0,4592 &gt; α = 0,05</p><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, data yang ada mendukung hipotesis nol yang artinya bahwa data <em>residual </em>model SARIMA(4,2,0)(0,1,1)[12] berdistribusi normal dan asumsi kenormalan sisaan terpenuhi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*hpAZw8pYORnhKpT2hzEV2A.png" /><figcaption><em>Diagnostic Check </em>SARIMA(4,2,0)(0,2,1)[12]</figcaption></figure><p>Berdasarkan gambar diatas, dapat dilihat pada grafik ACF bahwa terdapat tidak lebih dari 1 lag yang keluar dari batas interval sehingga asumsi <em>white</em> <em>noise</em> terpenuhi. Kemudian nilai p-value pada <em>L-jung Box</em> juga mayoritas batas 5% yang berarti bahwa asumsi non-autokorelasi pada model 4 terpenuhi atau tidak ada autokorelasi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*k3a6rdWK_eujgrFEcri8aA.png" /><figcaption><em>Output Jarque Bera Test </em>SARIMA(4,2,0)(0,2,1)[12]</figcaption></figure><p>· Hipotesis</p><p>H0 : Data <em>residual </em>berdistribusi normal</p><p>H1 : Data <em>residual </em>tidak berdistribusi normal</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji</p><p><em>p-value </em>= 0,1304</p><p>· Keputusan</p><p>Gagal tolak H0, karena <em>p-value </em>= 0,1304 &gt; α = 0,05</p><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, data yang ada mendukung hipotesis nol yang artinya bahwa data <em>residual </em>model SARIMA(4,2,0)(0,2,1)[12] berdistribusi normal dan asumsi kenormalan sisaan terpenuhi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*a0XJtxymrxz9pcoN0cHZDw.png" /><figcaption><em>Diagnostic Check </em>SARIMA(3,2,0)(1,2,0)[12]</figcaption></figure><p>Berdasarkan gambar diatas, dapat dilihat pada grafik ACF bahwa terdapat tidak lebih dari 1 lag yang keluar dari batas interval sehingga asumsi <em>white</em> <em>noise</em> terpenuhi. Kemudian nilai p-value pada <em>L-jung Box</em> dibawah batas 5% yang berarti bahwa asumsi non-autokorelasi pada model 5 tidak terpenuhi terpenuhi atau ada autokorelasi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*2ySzmkEUNj4-f2psdF8X9A.png" /><figcaption><em>Output Jarque Bera Test </em>SARIMA(3,2,0)(1,2,0)[12]</figcaption></figure><p>· Hipotesis</p><p>H0 : Data <em>residual </em>berdistribusi normal</p><p>H1 : Data <em>residual </em>tidak berdistribusi normal</p><p>· Tingkat Signifikansi</p><p>α = 0,05</p><p>· Daerah Kritis</p><p>Tolak H0 jika <em>p-value </em>&lt; α</p><p>· Statistik Uji</p><p><em>p-value </em>= 0,3352</p><p>· Keputusan</p><p>Gagal tolak H0, karena <em>p-value </em>= 0,3352 &gt; α = 0,05</p><p>· Kesimpulan</p><p>Dengan tingkat kepercayaan 95%, data yang ada mendukung hipotesis nol yang artinya bahwa data <em>residual </em>model SARIMA(3,2,0)(1,2,0)[12] berdistribusi normal dan asumsi kenormalan sisaan terpenuhi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*oxmigHgE_b8Dbztsn91Pmg.png" /><figcaption>Hasil Diagnostic Test</figcaption></figure><p><em>Keterangan: T = Terpenuhi; TT = Tidak Terpenuhi</em></p><p>Berikut adalah <em>output summary </em>untuk 2 model yang memenuhi semua <em>diagnostic test</em> :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*3CsCuU5-KbgtjN94QhWC2Q.png" /><figcaption><em>Output Summary </em>Model 3 dan Model 4</figcaption></figure><p>Dari hasil diagnostic test didapatkan 2 model SARIMA yang memenuhi asumsi white noise, asumsi autokorelasi dan asumsi normalitas, sehingga dari kedua model tersebut dicari model yang terbaik.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*jjCrKYlUnKuvij3209qZlg.png" /><figcaption>Perbandingan Nilai AIC dan MAPE</figcaption></figure><p>Berdasarkan nilai AIC dan MAPE terkecil, maka dipilihlah model SARIMA(4,2,0)(0,1,1)[12] sebagai model terbaik.</p><p><strong>Peramalan</strong></p><p>Berikut adalah hasil peramalan data produksi ikan Kota Pekalongan 5 periode kedepan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*FBnV2930mt-Wqdy4stZvRA.png" /><figcaption>Hasil Peramalan</figcaption></figure><p>Jadi, peramalan untuk data jumlah produksi ikan di Kota Pekalongan dari bulan Januari 2019 sampai bulan Mei 2019 secara berurut sebanyak 1316,73 Ton; 602,47 Ton; 912,09 Ton; 937,12 Ton; dan 588,44 Ton. Berdasarkan data-data sebelumnya, dapat praktikan lihat peramalan 5 bulan pertama di tahun 2019 ini menurun dari tahun 2018 yaitu secara berurutan sebanyak 1028 Ton; 554 Ton; 2121 Ton; 2625 Ton; 2281 Ton.</p><p>Berikut adalah nilai prediksinya:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*T-DhEq460ehNNig83aRBnA.png" /><figcaption>Nilai Prediksi</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*R1-5ip3eFsucjJDUa2lhJg.png" /><figcaption>Plot Prediksi</figcaption></figure><p><strong>Kesimpulan</strong></p><p>1. Berdasarkan hasil pergerakan pertumbuhan produksi ikan Kota Pekalongan tahun 2013–2018 maka dapat disimpulkan bahwa produksi ikan 2013–2018 secara umum mengalami peningkatan setiap tahunnya, walaupun pada tahun 2017 mengalami penurunan. Pada tahun 2018 adalah jumlah produksi tertinggi pada 7 tahun terakhir.</p><p>2. Model yang optimal dan penulis gunakan adalah model SARIMA(4,2,0)(0,1,1)[12] dengan nilai AIC = 97,84, dan nilai MAPE = 3,92.</p><p>3. Hasil peramalan untuk bulan Januari sampai Mei 2019 adalah secara berurut sebanyak 1316,73 Ton; 602,47 Ton; 912,09 Ton; 937,12 Ton; dan 588,44 Ton.</p><p>========================================</p><p>Please contact me if there are any questions.</p><p>Thank You! And thanks to BPS Kota Pekalongan!</p><p>Wassalamualaikum Wr. Wb.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fff198bfd8a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Object Counter]]></title>
            <link>https://medium.com/@akmalabdillah/object-counter-45f1335e889?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/45f1335e889</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Tue, 07 Jan 2020 03:48:50 GMT</pubDate>
            <atom:updated>2020-03-08T13:41:51.722Z</atom:updated>
            <content:encoded><![CDATA[<h3>Object Counter menggunakan Python</h3><p>Assalamualaikum Wr. Wb.</p><p>Hello Data Enthusiast!</p><p>Pada kesempatan kali ini, kita akan mempelajari tentang object counting atau penghitungan objek yang lewat berdasarkan video yang kita pilih.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fkryo9rVeh0gBnjc0aN2XA.png" /></figure><p>Pertama, kita harus mempunyai video sampel terlebih dahulu untuk dijadikan studi kasus. Pada kesempatan kali ini, saya menggunakan video dari youtube terkait video cctv di salah satu toko. Video tersebut lalu kita download dan kita masukkan pada folder “c:/counter” dengan nama “toko.mp4”.</p><p>Selanjutnya tinggal kita fokus terhadap pengkodingannya. Yang pertama kita dapat menuliskan koding untuk mendefinisikan pendeteksian objek yang nantinya akan kita seleksi memakai sistem dan juga pendefinisian modul “Cars” yang akan kita pakai. Untuk pengkodingannya sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*10lcmPOKsmycQJx2pMWA2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*tW1b_oSDcyrr-NiCdzm8cA.png" /></figure><p>Selanjutnya kita akan beralih ke pengkodingan terkait object counter kali ini. Yang pertama saya akan menuliskan inptan modul yang akan kita pakai.</p><pre>import numpy as np<br>import cv2<br>import Cars<br>import time</pre><p>Selanjutnya kita akan mendefinisikan import video dan juga spesifikasi video yang kita gunakan.</p><pre>#mendefinisikan input dan output<br>cnt_up   = 0<br>cnt_down = 0</pre><pre>#Membuka video<br>cap = cv2.VideoCapture(&#39;toko.mp4&#39;)</pre><pre>#Dimensi (properti) video<br>##cap.set(3,160) #Width<br>##cap.set(4,120) #Height</pre><pre>#Meletakkan properti video ke setiap frame di video<br>for i in range(19):<br>    print(i), cap.get(i)</pre><pre>w = cap.get(3)<br>h = cap.get(4)<br>frameArea = h*w<br>areaTH = frameArea/500<br>print (&#39;Area Threshold&#39;), areaTH</pre><p>Berdasarkan koding diatas, kita akan menggunakan video bernama “toko.mpv” yang sudah didownload sebelumnya dan menggunakan properti video landscape.</p><p>Selanjutnya kita akan membuat garis tracker untuk mendeteksi objek yang bergerak melewati garis tersebut.</p><pre>#Garis masuk dan keluar<br>line_up = int(1.45*(h/3)) #merah<br>line_down   = int(2.8*(h/4)) #biru</pre><pre>up_limit =   int(2.1*(h/5)) #putihatas<br>down_limit = int(2.4*(h/4)) #putihbawah</pre><pre>print (&quot;Red line y:&quot;),str(line_down)<br>print (&quot;Blue line y:&quot;), str(line_up)<br>line_down_color = (255,0,0)<br>line_up_color = (0,0,255)<br>pt1 =  [0, line_down];<br>pt2 =  [w, line_down];<br>pts_L1 = np.array([pt1,pt2], np.int32)<br>pts_L1 = pts_L1.reshape((-1,1,2))<br>pt3 =  [0, line_up];<br>pt4 =  [w, line_up];<br>pts_L2 = np.array([pt3,pt4], np.int32)<br>pts_L2 = pts_L2.reshape((-1,1,2))</pre><pre>pt5 =  [0, up_limit];<br>pt6 =  [w, up_limit];<br>pts_L3 = np.array([pt5,pt6], np.int32)<br>pts_L3 = pts_L3.reshape((-1,1,2))<br>pt7 =  [0, down_limit];<br>pt8 =  [w, down_limit];<br>pts_L4 = np.array([pt7,pt8], np.int32)<br>pts_L4 = pts_L4.reshape((-1,1,2))</pre><p>Berdasarkan koding diatas, saya akan membuat 4 tracker line untuk mendeteksi wajah. Jika ingin menggunakan video lain, tinggal kita sesuaikan saja spesifikasi dan posisi dari tracker line tersebut.</p><p>Selanjutnya kita akan membuat koding untuk membuat background dari bayangan object yang melintas dan juga pemfilteran morfologi dari object tersebut.</p><pre>#Background subtractor<br>fgbg = cv2.createBackgroundSubtractorMOG2(detectShadows = True)</pre><pre>#Elemen struktural untuk filter morfologi<br>kernelOp = np.ones((3,3),np.uint8)<br>kernelOp2 = np.ones((5,5),np.uint8)<br>kernelCl = np.ones((11,11),np.uint8)</pre><pre>#Variabel-variabel<br>font = cv2.FONT_HERSHEY_SIMPLEX<br>cars = []<br>max_p_age = 5<br>pid = 1</pre><p>Selanjutnya kita akan membuat koding terkait pengulangan pengolahan dari pendeteksian gambar, dikarenakan yang kita deteksi adalah gambar bergerak bukan gambar berhenti. Maka dari itu kita akan membuat sistem agar dapat terus menerus mendeteksi object yang kita hitung tersebut. Pada kesempatan kali ini menggunakan pengulangan <em>while</em>. Untuk materi terkait perulangan dapat kita lihat <a href="https://www.petanikode.com/python-perulangan/">disini</a>.</p><pre>while(cap.isOpened()):</pre><pre>#Membaca gambar dari aliran video<br>    ret, frame = cap.read()</pre><pre>for i in cars:<br>        i.age_one() #tandai setiap object sebagai satu object<br>#Tahap Pra-pengolahan    <br>    #Penerapan background subtractor<br>    fgmask = fgbg.apply(frame)<br>    fgmask2 = fgbg.apply(frame)</pre><pre>#Menggunakan metode binarisasi untuk menghilangkan bayangan (color ke grey)<br>    try:<br>        ret,imBin= cv2.threshold(fgmask,200,255,cv2.THRESH_BINARY)<br>        ret,imBin2 = cv2.threshold(fgmask2,200,255,cv2.THRESH_BINARY)<br>  <br>        #Proses Opening (erosi-&gt;dilasi) untuk menghilangkan noise<br>        mask = cv2.morphologyEx(imBin, cv2.MORPH_OPEN, kernelOp)<br>        mask2 = cv2.morphologyEx(imBin2, cv2.MORPH_OPEN, kernelOp)<br>  <br>        #Proses Closing (dilasi -&gt; erosi) untuk menghilangkan pixel ke putih<br>        mask =  cv2.morphologyEx(mask , cv2.MORPH_CLOSE, kernelCl)<br>        mask2 = cv2.morphologyEx(mask2, cv2.MORPH_CLOSE, kernelCl)<br>    except:<br>        print(&#39;EOF&#39;)<br>        print (&#39;UP:&#39;),cnt_up<br>        print (&#39;DOWN:&#39;),cnt_down<br>        break</pre><p>Selanjutnya kita akan membuat kontur dari objek tersebut:</p><pre>#Pembuatan Kontur    <br>     <br>    contours, hierarchy = cv2.findContours(mask2,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)<br>    for cnt in contours:<br>        area = cv2.contourArea(cnt)<br>        if area &gt; areaTH:<br>    <br>            #   TRACKING <br>            #Pengaturan untuk kondisi mobil yang jamak<br>   <br>            M = cv2.moments(cnt)<br>            cx = int(M[&#39;m10&#39;]/M[&#39;m00&#39;])<br>            cy = int(M[&#39;m01&#39;]/M[&#39;m00&#39;])<br>            x,y,w,h = cv2.boundingRect(cnt)</pre><pre>new = True<br>            if cy in range(up_limit,down_limit):<br>                for i in cars:<br>                    if abs(cx-i.getX()) &lt;= w and abs(cy-i.getY()) &lt;= h:<br>                        <br>      #object dekat dengan object yang telah terdeteksi sebelumnya<br>                        new = False<br>                        i.updateCoords(cx,cy)   <br>      #perbaharui koordinat dalam object dalam atur ulang usia (awal penandaan)<br>                        if i.going_UP(line_down,line_up) == True:<br>                            cnt_up += 1;<br>                            print (&quot;ID:&quot;),i.getId(),&#39;crossed going up at&#39;,time.strftime(&quot;%c&quot;)<br>                        elif i.going_DOWN(line_down,line_up) == True:<br>                            cnt_down += 1;<br>                            print (&quot;ID:&quot;),i.getId(),&#39;crossed going down at&#39;,time.strftime(&quot;%c&quot;)<br>                        break<br>                    if i.getState() == &#39;1&#39;:<br>                        if i.getDir() == &#39;down&#39; and i.getY() &gt; down_limit:<br>                            i.setDone()<br>                        elif i.getDir() == &#39;up&#39; and i.getY() &lt; up_limit:<br>                            i.setDone()<br>                    if i.timedOut():<br>      #hapus object (mobil) dari daftar<br>                        index = cars.index(i)<br>                        cars.pop(index)<br>                        del i     #hapus dari memori<br>                if new == True:<br>                    p = Cars.MyCars(pid,cx,cy, max_p_age)<br>                    cars.append(p)<br>                    pid += 1     </pre><p>Selanjutnya untuk trayektori gambar:</p><pre>for i in cars:<br>        cv2.putText(frame, str(i.getId()),(i.getX(),i.getY()),font,0.3,i.getRGB(),1,cv2.LINE_AA)</pre><pre>str_up = &#39;Keluar: &#39;+ str(cnt_up)<br>    str_down = &#39;Masuk: &#39;+ str(cnt_down)<br>    frame = cv2.polylines(frame,[pts_L1],False,line_down_color,thickness=2)<br>    frame = cv2.polylines(frame,[pts_L2],False,line_up_color,thickness=2)<br>    frame = cv2.polylines(frame,[pts_L3],False,(255,255,255),thickness=1)<br>    frame = cv2.polylines(frame,[pts_L4],False,(255,255,255),thickness=1)<br>    cv2.putText(frame, str_up ,(15,40),font,0.5,(200,200,200),2,cv2.LINE_AA)<br>    cv2.putText(frame, str_up ,(15,40),font,0.5,(0,200,0),1,cv2.LINE_AA)<br>    cv2.putText(frame, str_down ,(15,60),font,0.5,(200,200,200),2,cv2.LINE_AA)<br>    cv2.putText(frame, str_down ,(15,60),font,0.5,(0,0,200),1,cv2.LINE_AA)</pre><pre>cv2.imshow(&#39;Frame&#39;,frame)<br>    #cv2.imshow(&#39;Mask&#39;,mask)    <br>    <br>    #klik ESC untuk keluar<br>    k = cv2.waitKey(30) &amp; 0xff<br>    if k == 27:<br>        break<br>#Akhir untuk while(cap.isOpened())</pre><p>Terakhir adalah pembersihan layar ketika kita menyelesaikan penghitungan object tersebut.</p><pre>#bersihkan layar<br>cap.release()<br>cv2.destroyAllWindows()</pre><p>Selanjutnya kita simpan keseluruhan file tersebut bernama “counterpeople.py” kedalam folder (“c:/counter”) bersama file “Cars.py”. Selanjutnya kita running program dengan menggunakan command prompt.</p><pre>c:/counter&gt; python counterpeople.py</pre><p>Maka hasilnya seperti ini:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j4IQRPAYQBYfzpw1xXUvfg.png" /></figure><p>Maka sistem sudah otomatis menghitung objek yang pada kesempatan kali ini kita menggunakan objek orang berjalan. Sebenarnya sistem ini lebih kompatibel terhadap objek yang berbentuk seperti mobil dikarenakan modul yang kita pake lebih baik untuk objek mobil. Tetapi cukup baik jika digunakan untuk objek manusia.</p><p>Terima kasih, semoga bermanfaat!</p><p>Wassalamualaikum Wr. Wb.</p><p>========================================</p><p>Thanks to:</p><p>Bapak RB. Fajriya Hakim</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45f1335e889" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Face Recognition 4]]></title>
            <link>https://medium.com/@akmalabdillah/face-recognition-4-c2a83e5177c8?source=rss-4a0a5dd3696f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2a83e5177c8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[face-recognition]]></category>
            <dc:creator><![CDATA[Akmal Abdillah]]></dc:creator>
            <pubDate>Mon, 06 Jan 2020 05:00:47 GMT</pubDate>
            <atom:updated>2020-01-06T05:00:47.816Z</atom:updated>
            <content:encoded><![CDATA[<p>Bismillahirrahmanirrahim</p><p>Assalamualaikum Wr. Wb.</p><p>Untuk face recognition ke-4 atau yang terakhir ini, kita akan menyambungkan data yang kita masukkan kedalam database terhadap visual penangkapan wajah kita.</p><p>Untuk yang pertama, kita siapkan software database. Pada kesempatan kali ini kita menggunakan software database <em>SQLiteStudio</em>. Software tersebut dapat kita download <a href="https://sqlitestudio.pl/index.rvt?act=download">disini</a>. Maka akan muncul jendela browser seperti ini:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*MnHoARTQy_rWq4syHjCyhQ.png" /></figure><p>Setelah itu bisa kita download (sesuaikan dengan laptop anda) dan install software tersebut. Ekstrak file download tersebut di folder “c:/capture”. Setelah instalasi selesai, buka software tersebut, tampilan awalnya sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zc2f92SXREfVw-KndoQGuQ.png" /></figure><p>Buat database baru, dengan cara <em>database -&gt; add a database</em> atau bisa gunakan pintasan keyboard <em>ctrl + O</em>. Kemudian akan muncul jendela baru, kita masukkan sesuai dengan gambar dibawah saja.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*fNoEkZTbLyF3QgHbZ5LIqw.png" /></figure><p>Kemudian dapat kita klik tombol “+” atau “tambah” berwarna merah putih disamping kolom file untuk menyimpan database tersebut. Penyimpanan file database tersebut kita tuju di folder “c:/capture”, sehingga ada file datawajah.db di folder capture. Selanjutnya klik OK saja dan akan nampak database baru di sebelah sisi kiri.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/1*DfU5W7HjNfhsFpokIe4-zw.png" /></figure><p>Selanjutnya kita akan membuat tabel didalam database tersebut dengan cara klik dua kali database “datawajah” sehingga muncul 2 menu yaitu tables dan views, kita pilih tables dan cari ikon creat a table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0iP0yC20RI4NlZRC7Ut0CQ.png" /></figure><p>Sehingga setelah kita mengklik ikon tersebut akan muncul jendela seperti dibawah ini:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*ZLQs1CJLOFHmOM7npyQdFA.png" /></figure><p>Kita beri nama tabel tersebut sebagai “orang”, selanjutnya setelah kita menulis nama tabel, kita klik ikon centang diatasnya atau ikon “commit structure changes”. Selanjutnya kita isi struktur tabel atau kolom tersebut seperti ini:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8m-Zdm-a8VMMYTZ4_OY3dQ.png" /></figure><p>Berdasarkan gambar diatas, saya menggunakan <em>primary key</em> untuk kolom ID dan menggunakan <em>Not Null</em> untuk kolom ID dan Nama.</p><p>Beralih ke menu di sebelahnya, yaitu data. Berdasarkan koding dan juga rekam wajah dari awal, maka kita memasukkan data tersebut kedalam tabel. Sehingga tampilan datanya sebagai berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*vy-WhXEnpwWsh3r243tXVA.png" /></figure><p>Setelah semua dirasa cukup. maka struktur dan data sudah masuk kedalam database <em>datawajah.db</em>. Selanjutnya kita kembali ke koding terakhir kita untuk menyambungkan ke database sql dengan pengenalan wajah.</p><pre>import cv2,os, time<br>import numpy as np<br>from PIL import Image<br>import pickle<br>import sqlite3<br>camera=0<br>faceDetect=cv2.CascadeClassifier(&#39;haarcascade_frontalface_default.xml&#39;);<br>video=cv2.VideoCapture(camera,cv2.CAP_DSHOW)<br>a=0<br>recognizer=cv2.face.LBPHFaceRecognizer_create();<br>recognizer.read(&quot;c://capture/training/training.xml&quot;)<br>id=0<br>fontface=cv2.FONT_HERSHEY_SIMPLEX<br>fontscale=1<br>fontcolor=(0,255,255)<br>path=&#39;DataSet&#39;<br>def getProfile(id):<br> conn=sqlite3.connect(&quot;datawajah.db&quot;)<br> cmd=&quot;SELECT * FROM orang WHERE id=&quot;+str(id)<br> cursor=conn.execute(cmd)<br> profile=None<br> for row in cursor:<br>  profile=row<br> conn.close()<br> return profile<br>while(True):<br> check, frame=video.read();<br> gray=cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)<br> faces=faceDetect.detectMultiScale(gray,1.3,5);<br> for(x,y,w,h) in faces:<br>  a=a+1<br>  cv2.rectangle(frame,(x,y),(x+w,y+h),(0,255,0),2)<br>  id,conf=recognizer.predict(gray[y:y+h,x:x+w])<br>  profile=getProfile(id)<br>  if(profile!=None):<br>   cv2.putText(frame,str(profile[1]),(y,h+x-180),fontface,fontscale,fontcolor)<br>   cv2.putText(frame,str(profile[2]),(y,h+x-150),fontface,fontscale,fontcolor)<br>   cv2.putText(frame,str(profile[3]),(y,h+x-120),fontface,fontscale,fontcolor)<br>   #cv2.putText(frame,str(profile[4]),(x,y+h+100),fontface,fontscale,fontcolor);<br> cv2.imshow(&quot;wajah&quot;,frame);<br> if(cv2.waitKey(1)==ord(&#39;q&#39;)):<br>  break<br> print(a)<br>cam.release()<br>cv2.destroyAllWindows()</pre><p>Kemudian simpan file tersebut dengan nama “<em>tangkapwajahlengkap.py</em>” didalam folder “c:/capture”. Selanjutnya bisa kita running program tersebut dengan command prompt seperti biasa.</p><pre>c:/capture/&gt; python tangkapwajahlengkap.py</pre><p>Hasilnya akan tampak seperti berikut:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fr2b8g3gGcNdxIfARE4nog.png" /></figure><p>Alhamdulillah berhasil gaes!!</p><p>Terlihat pada gambar tersebut sudah tertera identitas pemilik wajah, yaitu nama, umur dan gendernya.</p><p>Selamat mencoba gaes!!</p><p>Wassalamualaikum Wr. Wb.</p><p>========================================</p><p>Thanks to:</p><p>Bapak RB. Fajriya Hakim.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2a83e5177c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>