<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abdulquadri Ayodeji Oshoare on Medium]]></title>
        <description><![CDATA[Stories by Abdulquadri Ayodeji Oshoare on Medium]]></description>
        <link>https://medium.com/@abdulquadri-oshoare?source=rss-480dff2fffee------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*ySLqQAKJtkSzXekH2HfzbQ.jpeg</url>
            <title>Stories by Abdulquadri Ayodeji Oshoare on Medium</title>
            <link>https://medium.com/@abdulquadri-oshoare?source=rss-480dff2fffee------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:05:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abdulquadri-oshoare/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Change Data Capture (CDC): MySQL to BigQuery using Binary Logs Replication]]></title>
            <link>https://abdulquadri-oshoare.medium.com/mysql-to-bigquery-via-change-data-capture-cdc-using-binary-logs-replication-5cea24a124e4?source=rss-480dff2fffee------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cea24a124e4</guid>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Abdulquadri Ayodeji Oshoare]]></dc:creator>
            <pubDate>Mon, 07 Aug 2023 14:06:18 GMT</pubDate>
            <atom:updated>2023-08-07T14:16:01.487Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*J58D58zALNkmNKhAdaZaSg.png" /><figcaption>Architecture Diagram</figcaption></figure><h3>Introduction</h3><p>This article explains how Binary Log Replication can be used to perform CDC when moving data from MySQL to BigQuery.</p><p>The following tools will be used in this project:</p><blockquote><strong>Source: </strong>MySQL Database (Locally or on Cloud)<br><strong>Destination:</strong> BigQuery <br><strong>Orchestrator: </strong>Airflow <br><strong>Container: </strong>Docker and Docker Compose installed on the computer<br><strong>Dataset: </strong>Ny_taxi yellow_trips_data</blockquote><h3>What is Change Data Capture (CDC)?</h3><p>CDC is a method of capturing database events. The events can be one or all of the following (create, insert, update, delete). CDC is an efficient method for replicating databases because it ensures that data replication is near real-time.</p><p>MySQL like many other relational databases handles CDC differently. <br>It stores database events in logs (Binlog) that can be pulled to ingest large volumes of data.</p><p>Link to my git repository for the full code.</p><p><a href="https://github.com/quadriano31/Airflow-MySQL-to-BigQuery">GitHub - quadriano31/Airflow-MySQL-to-BigQuery: Using Binlog replication to perform CDC</a></p><h4>Getting Started</h4><pre># Project Structure<br>root/<br>├──airflow_image<br>│   └── Dockerfile<br>│   └── requirements.txt<br>├── dags/<br>│   └── package/<br>│      └── config.py<br>│   └── full_load.py<br>│   └── incremental_load_cdc.py<br>├── data/<br>│   └── yellow_tripdata_2021-01.csv.gz<br>├── logs/<br>├── .env<br>├── plugins<br>├── load_data_to_db.py<br>├── load_incremental_to_db.py<br>├── my.cnf<br>└── docker-compose.yml</pre><p>Create the files and folder following the project structure above.<br>The following files are important for Binary log replication.</p><p>MySQL uses a configuration file called my.cnf that stores various settings and parameters for the database server. It is used to control the behavior and performance of the database server.<br>Contents of my.cnf</p><pre>[mysqld]<br>server-id   = 1<br>log_bin    = bin.log<br>expire_logs_days = 10<br>max_binlog_size  = 100M<br>binlog-format    = row #Very important if you want to receive write, update, and delete row events</pre><p>Content of requirements.txt</p><pre>google-auth<br>google-cloud-bigquery<br>google-cloud-bigquery-storage<br>pandas<br>pandas-gbq<br>mysql-replication <br>protobuf <br>pymysql<br>mysql-connector-python</pre><p>Contents of .env</p><pre>AIRFLOW_UID=1000<br>AIRFLOW_GID=0</pre><p>Contents of Dockerfile</p><pre># Base image<br>FROM apache/airflow:2.6.3<br># User airflow<br>USER airflow<br># Upgrade pip<br>RUN pip install --upgrade pip<br># Copy requirements.txt<br>COPY requirements.txt .<br># Install python packages<br>RUN pip install -r requirements.txt</pre><p>Contents of docker-compose file</p><pre>version: &#39;3&#39;<br>x-airflow-common:<br>  &amp;airflow-common<br>  build: airflow_image/<br>  environment:<br>    &amp;airflow-common-env<br>    LD_PRELOAD: /usr/lib/x86_64-linux-gnu/libstdc++.so.6<br>    AIRFLOW__CORE__EXECUTOR: CeleryExecutor<br>    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow<br>    # For backward compatibility, with Airflow &lt;2.3<br>    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow<br>    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow<br>    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0<br>    AIRFLOW__CORE__FERNET_KEY: &#39;&#39;<br>    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: &#39;true&#39;<br>    AIRFLOW__CORE__LOAD_EXAMPLES: &#39;false&#39;<br>    AIRFLOW__API__AUTH_BACKENDS: &#39;airflow.api.auth.backend.basic_auth&#39;<br>    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}<br>  volumes:<br>    - ./dags:/opt/airflow/dags<br>    - ./logs:/opt/airflow/logs<br>    - ./plugins:/opt/airflow/plugins<br>  user: &quot;${AIRFLOW_UID:-50000}:0&quot;<br>  depends_on:<br>    &amp;airflow-common-depends-on<br>    redis:<br>      condition: service_healthy<br>    postgres:<br>      condition: service_healthy<br>services:<br>  postgres:<br>    image: postgres:13<br>    environment:<br>      POSTGRES_USER: airflow<br>      POSTGRES_PASSWORD: airflow<br>      POSTGRES_DB: airflow<br>    volumes:<br>      - postgres-db-volume:/var/lib/postgresql/data<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;pg_isready&quot;, &quot;-U&quot;, &quot;airflow&quot;]<br>      interval: 5s<br>      retries: 5<br>    restart: always<br>  redis:<br>    image: redis:latest<br>    expose:<br>      - 6379<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;redis-cli&quot;, &quot;ping&quot;]<br>      interval: 5s<br>      timeout: 30s<br>      retries: 50<br>    restart: always<br>  airflow-webserver:<br>    &lt;&lt;: *airflow-common<br>    command: webserver<br>    ports:<br>      - 8080:8080<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;--fail&quot;, &quot;http://localhost:8080/health&quot;]<br>      interval: 10s<br>      timeout: 10s<br>      retries: 5<br>    restart: always<br>    depends_on:<br>      &lt;&lt;: *airflow-common-depends-on<br>      airflow-init:<br>        condition: service_completed_successfully<br>      mysqldb:<br>        condition: service_healthy<br>  airflow-scheduler:<br>    &lt;&lt;: *airflow-common<br>    command: scheduler<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &#39;airflow jobs check --job-type SchedulerJob --hostname &quot;$${HOSTNAME}&quot;&#39;]<br>      interval: 10s<br>      timeout: 10s<br>      retries: 5<br>    restart: always<br>    depends_on:<br>      &lt;&lt;: *airflow-common-depends-on<br>      airflow-init:<br>        condition: service_completed_successfully<br>  airflow-worker:<br>    &lt;&lt;: *airflow-common<br>    command: celery worker<br>    healthcheck:<br>      test:<br>        - &quot;CMD-SHELL&quot;<br>        - &#39;celery --app airflow.executors.celery_executor.app inspect ping -d &quot;celery@$${HOSTNAME}&quot;&#39;<br>      interval: 10s<br>      timeout: 10s<br>      retries: 5<br>    environment:<br>      &lt;&lt;: *airflow-common-env<br>      # Required to handle warm shutdown of the celery workers properly<br>      # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation<br>      DUMB_INIT_SETSID: &quot;0&quot;<br>    restart: always<br>    depends_on:<br>      &lt;&lt;: *airflow-common-depends-on<br>      airflow-init:<br>        condition: service_completed_successfully<br>  airflow-triggerer:<br>    &lt;&lt;: *airflow-common<br>    command: triggerer<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &#39;airflow jobs check --job-type TriggererJob --hostname &quot;$${HOSTNAME}&quot;&#39;]<br>      interval: 10s<br>      timeout: 10s<br>      retries: 5<br>    restart: always<br>    depends_on:<br>      &lt;&lt;: *airflow-common-depends-on<br>      airflow-init:<br>        condition: service_completed_successfully<br>  airflow-init:<br>    &lt;&lt;: *airflow-common<br>    entrypoint: /bin/bash<br>    # yamllint disable rule:line-length<br>    command:<br>      - -c<br>      - |<br>        function ver() {<br>          printf &quot;%04d%04d%04d%04d&quot; $${1//./ }<br>        }<br>        airflow_version=$$(gosu airflow airflow version)<br>        airflow_version_comparable=$$(ver $${airflow_version})<br>        min_airflow_version=2.2.0<br>        min_airflow_version_comparable=$$(ver $${min_airflow_version})<br>        if (( airflow_version_comparable &lt; min_airflow_version_comparable )); then<br>          echo<br>          echo -e &quot;\033[1;31mERROR!!!: Too old Airflow version $${airflow_version}!\e[0m&quot;<br>          echo &quot;The minimum Airflow version supported: $${min_airflow_version}. Only use this or higher!&quot;<br>          echo<br>          exit 1<br>        fi<br>        if [[ -z &quot;${AIRFLOW_UID}&quot; ]]; then<br>          echo<br>          echo -e &quot;\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m&quot;<br>          echo &quot;If you are on Linux, you SHOULD follow the instructions below to set &quot;<br>          echo &quot;AIRFLOW_UID environment variable, otherwise files will be owned by root.&quot;<br>          echo &quot;For other operating systems you can get rid of the warning with manually created .env file:&quot;<br>          echo &quot;    See: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#setting-the-right-airflow-user&quot;<br>          echo<br>        fi<br>        one_meg=1048576<br>        mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))<br>        cpus_available=$$(grep -cE &#39;cpu[0-9]+&#39; /proc/stat)<br>        disk_available=$$(df / | tail -1 | awk &#39;{print $$4}&#39;)<br>        warning_resources=&quot;false&quot;<br>        if (( mem_available &lt; 4000 )) ; then<br>          echo<br>          echo -e &quot;\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m&quot;<br>          echo &quot;At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))&quot;<br>          echo<br>          warning_resources=&quot;true&quot;<br>        fi<br>        if (( cpus_available &lt; 2 )); then<br>          echo<br>          echo -e &quot;\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m&quot;<br>          echo &quot;At least 2 CPUs recommended. You have $${cpus_available}&quot;<br>          echo<br>          warning_resources=&quot;true&quot;<br>        fi<br>        if (( disk_available &lt; one_meg * 10 )); then<br>          echo<br>          echo -e &quot;\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m&quot;<br>          echo &quot;At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))&quot;<br>          echo<br>          warning_resources=&quot;true&quot;<br>        fi<br>        if [[ $${warning_resources} == &quot;true&quot; ]]; then<br>          echo<br>          echo -e &quot;\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m&quot;<br>          echo &quot;Please follow the instructions to increase amount of resources available:&quot;<br>          echo &quot;   https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin&quot;<br>          echo<br>        fi<br>        mkdir -p /sources/logs /sources/dags /sources/plugins<br>        chown -R &quot;${AIRFLOW_UID}:0&quot; /sources/{logs,dags,plugins}<br>        exec /entrypoint airflow version<br>    # yamllint enable rule:line-length<br>    environment:<br>      &lt;&lt;: *airflow-common-env<br>      _AIRFLOW_DB_UPGRADE: &#39;true&#39;<br>      _AIRFLOW_WWW_USER_CREATE: &#39;true&#39;<br>      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}<br>      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}<br>    user: &quot;0:0&quot;<br>    volumes:<br>      - .:/sources<br>  airflow-cli:<br>    &lt;&lt;: *airflow-common<br>    profiles:<br>      - debug<br>    environment:<br>      &lt;&lt;: *airflow-common-env<br>      CONNECTION_CHECK_MAX_COUNT: &quot;0&quot;<br>    # Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252<br>    command:<br>      - bash<br>      - -c<br>      - airflow<br>  flower:<br>    &lt;&lt;: *airflow-common<br>    command: celery flower<br>    ports:<br>      - 5555:5555<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;--fail&quot;, &quot;http://localhost:5555/&quot;]<br>      interval: 10s<br>      timeout: 10s<br>      retries: 5<br>    restart: always<br>    depends_on:<br>      &lt;&lt;: *airflow-common-depends-on<br>      airflow-init:<br>        condition: service_completed_successfully<br>  mysqldb:<br>    image: mysql:5.7<br>    environment:<br>      MYSQL_ALLOW_EMPTY_PASSWORD: &quot;yes&quot;<br>      MYSQL_DATABASE: trips_db<br>    container_name: mysql_server<br>    volumes:<br>      - &quot;./mysql_data:/var/lib/mysql:rw&quot;<br>      - &quot;./my.cnf:/etc/mysql/conf.d/my.cnf&quot;<br>      - ./logs/mysql:/var/log/mysql<br>    ports:<br>      - 3306:3306<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;mysqladmin&quot; ,&quot;ping&quot;, &quot;-h&quot;, &quot;localhost&quot;]<br>      interval: 10s<br>      timeout: 20s<br>      retries: 10<br>    restart: always<br><br>networks:<br>  default:<br>    name: airflow-network<br>volumes:<br>  postgres-db-volume:</pre><p>In the root of the container run docker compose up and wait some minutes for the container to be provisioned.</p><p>Run “docker container ps” to confirm that all containers are running and healthy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WdbbWelPV29joIYi1a3R4g.png" /><figcaption>docker container running</figcaption></figure><p>To login to airflow, go to localhost:8080 on your browser.</p><blockquote>username: airflow<br>password: airflow</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cg-3GG7vFqmlwM4mKWQZNg.png" /><figcaption>airflow login</figcaption></figure><h4>Load Data into MySQL data</h4><p>To load data into the MySQL database, let’s create a Python script called load_full_data.py and paste the following codes.</p><pre>import pandas as pd <br>import pymysql <br>from sqlalchemy import create_engine<br>import MySQLdb<br>host = &quot;localhost&quot;<br>user = &quot;root&quot;<br>password = &quot;&quot;<br>db = &quot;trips_db&quot;<br><br># Construct the connection URL with an empty password<br>connection_url = f&quot;mysql+mysqldb://{user}:{password}@{host}/{db}&quot;<br># Create an SQLAlchemy engine and connect to the database<br>engine = create_engine(connection_url)<br>conn = engine.connect()<br>df = pd.read_csv(&quot;./data/yellow_tripdata_2021-01.csv.gz&quot;,nrows=1000,compression=&#39;gzip&#39;)<br>df.tpep_pickup_datetime = pd.to_datetime(df.tpep_pickup_datetime)<br>df.tpep_dropoff_datetime = pd.to_datetime(df.tpep_dropoff_datetime)<br>df.to_sql(name=&quot;yellow_tripdata&quot;,con=conn,index=False)</pre><p>The script connects to MySQL using create_engine and loads the first 1000 data into the trips_db database and yellow_tripdata table.</p><p>To confirm that the data has been successfully loaded, enter the MySQL terminal on docker by running:</p><blockquote>docker exec -it &lt;mysql-container-id&gt; /bin/bash</blockquote><p>log into the database by running “<strong>mysql -uroot</strong>”, after login, type “<strong>use trips_db</strong>” and run the SQL command below. You should get the result in the image below.</p><pre>SELECT table_name<br>FROM information_schema.tables<br>WHERE table_schema = &#39;trips_db&#39; AND table_name = &#39;yellow_tripdata&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*p1GXbv5uXqtnsCOi1HOOZA.png" /><figcaption>testing DB</figcaption></figure><h4>Turn Binary Log Replication ON</h4><p>In order for Binary log replication to work, you need to provide read access to the log folder.</p><p>Enter the terminal of the MySQL container and run the following bash commands to provide permission on the folder.</p><blockquote>chmod 644 /etc/mysql/*</blockquote><p>Restart the MySQL container after running the command above. To confirm that Binary log replication has been enabled, enter the MySQL terminal again and run the commands below.</p><pre>mysql -uroot<br>set @@global.show_compatibility_56=ON;<br>select variable_value as &quot;BINARY LOGGING STATUS (log_bin) :: &quot; <br>from information_schema.global_variables where variable_name=&#39;log_bin&#39;;</pre><p>You should get the result below after running the last command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*Clrkr05yg2_9H0wA6BtOfQ.png" /><figcaption>Binary replication</figcaption></figure><h4>Connect Airflow to MySQL</h4><p>For airflow to communicate with MySQL DB, we need to create a connection using the airflow network that was created using the docker-compose file. Run the commands below to get the network gateway.</p><blockquote>docker network inspect airflow-network</blockquote><p>Copy the network gateway as shown below. This will be used to create the MySQL connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*C2g_qnttw7U5pd1RW2DQcw.png" /><figcaption>docker network</figcaption></figure><p>Go to the airflow UI &gt;&gt; Admin &gt;&gt; connections on localhost:8080.<br>Create a new MySQL connection and add the following details below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*D6aSWCZmd2jzknOBU-gwuQ.png" /><figcaption>airflow connection page</figcaption></figure><p>Make sure to test the connection and ensure that it connects successfully.</p><h4>Set up BigQuery and Service account</h4><p>In order to connect to BigQuery, we need to set up a GCP project, create a BigQuery dataset, and set up a service account.</p><p>If you don’t have a GCP account, you can create one using the link below and get $300 cloud credits that can be used for 90 days.</p><p><a href="https://cloud.google.com/">Cloud Computing Services | Google Cloud</a></p><p>In the Google Cloud console, search for BigQuery. Create the BigQuery dataset with the name “trips_db”.</p><p>To set up a service account, Navigate to IAM-Admin &gt; service account in the Google Cloud console.</p><p>Create the service account with the desired name and assign the BigQuery Admin role to the service account.</p><p>Download the &lt;service-account-name&gt;.json and place it under the dags folder.</p><blockquote>Note: Always add service accounts to .gitignore</blockquote><h4>Set up Airflow Dags</h4><p>Now that we’ve successfully created airflow containers, it’s time to create our first dag and move data from the MySQL database to BigQuery.</p><p>Under the dags/package folder, create a config.py file. <br>This will be used to store our variables.</p><pre>class CONFIG():<br>    PROJECT_ID = &quot;marine-fusion-394105&quot;<br>    TABLE_NAME = &quot;yellow_tripdata&quot;<br>    INCREMENTAL_TABLE_NAME = &quot;yellow_tripdata_incremental&quot;<br>    DATASET_REF = &quot;trips_db&quot;<br>    DB = &quot;trips_db&quot;</pre><blockquote>Note: in a production environment kindly use airflow variable</blockquote><p>Create a full_load.py file under the dags folder and copy the codes below into it.</p><pre>import time<br>from datetime import datetime,timedelta<br>import os<br>from airflow.models import DAG<br>from airflow.operators.python_operator import PythonOperator<br>from airflow.operators.bash_operator import BashOperator<br>from airflow.providers.mysql.hooks.mysql import MySqlHook<br>from google.cloud import bigquery<br>from google.oauth2 import service_account<br># Import other packages<br>from packages.config import CONFIG<br># Set Google Cloud credentials file path<br>os.environ[&#39;GOOGLE_APPLICATION_CREDENTIALS&#39;] = &#39;/opt/airflow/dags/keys.json&#39;<br><br># Define default_args dictionary for the DAG<br>default_args = {<br>    &#39;owner&#39;: &#39;airflow&#39;,<br>    &#39;depends_on_past&#39;: False,<br>    &#39;start_date&#39;: datetime(2023, 2, 15),<br>    &#39;email_on_failure&#39;: False,<br>    &#39;email_on_retry&#39;: False,<br>    &#39;retries&#39;: 1,<br>    &#39;retry_delay&#39;: timedelta(minutes=1),<br>}<br># Instantiate the DAG<br>dag = DAG(<br>    &#39;full_copy&#39;,<br>    default_args=default_args,<br>    description=&#39;DAG to extract data from MySQL and load it into Google BigQuery&#39;,<br>    schedule_interval=&quot;0 23 * * 0&quot;,<br>    tags=[&#39;load_gcp&#39;],<br>    catchup=False,<br>)<br># Define the SQL extract task<br>def check_table_exist():<br>    try:<br>        hook = MySqlHook(mysql_conn_id=&quot;mysql_conn_id&quot;, schema=CONFIG.DB)<br>        query = f&quot;&quot;&quot;<br>            SELECT table_name<br>            FROM information_schema.tables<br>            WHERE table_schema = &#39;{CONFIG.DB}&#39; AND table_name = &#39;{CONFIG.TABLE_NAME}&#39;<br>        &quot;&quot;&quot;<br>        records = hook.get_records(sql=query)<br>        print(records)<br>        tbl_name = records[0]<br>        return tbl_name<br>    except Exception as e:<br>        print(&quot;Data extract error: &quot; + str(e))<br>check_table_exist_task = PythonOperator(<br>    task_id=&#39;sql_extract&#39;,<br>    python_callable=check_table_exist,<br>    dag=dag,<br>)<br>#check bigquery table exist <br>def create_bigquery_table():<br>    try:<br>        client = bigquery.Client()<br>        dataset_ref = client.dataset(CONFIG.DATASET_REF)<br>        table_ref = dataset_ref.table(CONFIG.TABLE_NAME)<br>        schema = []  # Define your schema here<br>        <br>        table = bigquery.Table(table_ref, schema=schema)<br>        table = client.create_table(table, exists_ok=True)  # Create if not exists<br>        print(f&quot;BigQuery table {CONFIG.TABLE_NAME} created or already exists.&quot;)<br>    except Exception as e:<br>        print(&quot;BigQuery table creation error: &quot; + str(e))<br>create_bigquery_table_task = PythonOperator(<br>    task_id=&#39;create_bigquery_table&#39;,<br>    python_callable=create_bigquery_table,<br>    dag=dag,<br>)<br># Define the GCP load task<br>def gcp_load(tbl_name):<br>    try:<br>        client = bigquery.Client()<br>        job_config = bigquery.job.LoadJobConfig()<br>        project_id = CONFIG.PROJECT_ID<br>        dataset_ref = CONFIG.DATASET_REF<br>        TABLE = CONFIG.TABLE_NAME<br>        table_id = f&quot;{project_id}.{dataset_ref}.{TABLE}&quot;<br>        rows_imported = 0<br>        job_config = bigquery.job.LoadJobConfig()<br>        # Set write_disposition parameter as WRITE_APPEND for appending to the table<br>        job_config.write_disposition = bigquery.WriteDisposition.WRITE_TRUNCATE<br>        hook = MySqlHook(mysql_conn_id=&quot;mysql_conn_id&quot;, schema=CONFIG.DB)<br>        for table_name in tbl_name:<br>            # print(table_name)<br>            query = f&#39;SELECT * FROM {table_name}&#39;<br>            df = hook.get_pandas_df(query)<br>            print(f&#39;Importing rows {rows_imported} to {rows_imported + len(df)}... for table {table_name}&#39;)<br>            <br>            # Then proceed with the data loading<br>            job = client.load_table_from_dataframe(df, table_id, job_config=job_config)<br>            job.result()  <br>            table = client.get_table(table_id)  # Make an API request.<br>            print(<br>                f&quot;Loaded {table.num_rows} rows and {len(table.schema)} columns to {table_id}&quot;<br>            )<br>    except Exception as e:<br>        print(&quot;Data load error: &quot; + str(e))<br>gcp_load_task = PythonOperator(<br>    task_id=&#39;gcp_load&#39;,<br>    python_callable=gcp_load,<br>    op_args=[check_table_exist_task.output],<br>    dag=dag,<br>)<br>check_table_exist_task &gt;&gt; create_bigquery_table_task &gt;&gt; gcp_load_task</pre><p>The first dag has 3 tasks. The first task checks the MySQL db if the table we want to pull from exits. The second task checks if The BigQuery table exists and creates it if it doesn’t exist. Then last task loads the data from MySQL to BigQuery.<br>Let’s see what this looks like in the airflow UI and run it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jSqk70bbr08UNhql-oRKew.png" /><figcaption>full_copy_dag</figcaption></figure><p>Cool, looks like the first dag has loaded data into the BigQuery table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nfGhCdrRUF2X1pDTj0sFAQ.png" /><figcaption>BigQuery Full load</figcaption></figure><h4>Incremental dag using CDC</h4><p>Let’s create the second dag which will load incremental data to BigQuery.<br>Create an incremental_load_cdc.py file and copy the content below.</p><pre>import time<br>from datetime import datetime, timedelta, date<br>import pytz <br>import os<br>import json<br>from airflow.models import DAG<br>from airflow.operators.python_operator import PythonOperator<br>import mysql.connector<br>from mysql.connector import MySQLConnection<br>from mysql.connector import cursor<br>from pymysqlreplication import BinLogStreamReader<br>from pymysqlreplication.row_event import (<br>    DeleteRowsEvent,<br>    UpdateRowsEvent,<br>    WriteRowsEvent<br>)<br>from airflow.providers.mysql.hooks.mysql import MySqlHook<br>from google.cloud import bigquery<br>from google.oauth2 import service_account<br># Import other packages<br>from packages.config import CONFIG<br># Set Google Cloud credentials file path<br>os.environ[&#39;GOOGLE_APPLICATION_CREDENTIALS&#39;] = &#39;/opt/airflow/dags/keys.json&#39;<br># Define default_args dictionary for the incremental DAG<br>default_args_incremental = {<br>    &#39;owner&#39;: &#39;airflow&#39;,<br>    &#39;depends_on_past&#39;: False,<br>    &#39;start_date&#39;: datetime(2023, 7, 15), <br>    &#39;email_on_failure&#39;: False,<br>    &#39;email_on_retry&#39;: False,<br>    &#39;retries&#39;: 1,<br>    &#39;retry_delay&#39;: timedelta(minutes=1),<br>}<br># Instantiate the incremental DAG<br>dag_incremental = DAG(<br>    &#39;gcp_extract_and_load_incremental&#39;,<br>    default_args=default_args_incremental,<br>    description=&#39;Incremental DAG to extract data from MySQL and load it into Google BigQuery&#39;,<br>    schedule_interval=&quot;*/5 * * * *&quot;,  # Schedule it every 5 minutes<br>    catchup=False,<br>    tags=[&#39;load_gcp&#39;, &#39;incremental&#39;],<br>)<br># Define the SQL extract task<br>def check_table_exist():<br>    try:<br>        hook = MySqlHook(mysql_conn_id=&quot;mysql_conn_id&quot;, schema=CONFIG.DB)<br>        query = f&quot;&quot;&quot;<br>            SELECT table_name<br>            FROM information_schema.tables<br>            WHERE table_schema = &#39;{CONFIG.DB}&#39; AND table_name = &#39;{CONFIG.TABLE_NAME}&#39;<br>        &quot;&quot;&quot;<br>        records = hook.get_records(sql=query)<br>        print(records)<br>        tbl_name = records[0][0]<br>        if tbl_name == CONFIG.TABLE_NAME:<br>            return True<br>        else:<br>            return False<br>    except Exception as e:<br>        print(&quot;Data extract error: &quot; + str(e))<br>check_table_exist_task = PythonOperator(<br>    task_id=&#39;check_table_exist&#39;,<br>    python_callable=check_table_exist,<br>    dag=dag_incremental,<br>)<br># Function to extract incremental data from MySQL using Binary Log Reader<br>def extract_incremental_data(**kwargs):<br>    task_instance = kwargs[&#39;task_instance&#39;]<br>    mysql_check_result = task_instance.xcom_pull(task_ids=&#39;check_table_exist&#39;)<br>    <br>    if mysql_check_result:<br>        try:<br>            # Connect to MySQL database using the Airflow MySQL Hook<br>            mysql_hook = MySqlHook(mysql_conn_id=&quot;mysql_conn_id&quot;)<br>            conn: MySQLConnection = mysql_hook.get_conn()<br>            cursor: cursor = conn.cursor()<br>            ###################################<br>            # Retrieve the connection details as a dictionary<br>            connection_details = mysql_hook.get_connection(conn_id=&quot;mysql_conn_id&quot;)<br>            # Access individual elements (host, port, user, passwd) from the dictionary<br>            host = connection_details.host<br>            port = connection_details.port<br>            user = connection_details.login<br>            passwd = connection_details.password<br>            # Initialize a list to store the modified rows<br>            modified_rows = []<br><br>            mysql_settings = {&#39;host&#39;: host, &#39;port&#39;: port, &#39;user&#39;: user, &#39;passwd&#39;: &#39;&#39;}<br>            # Create the BinLogStreamReader<br>            stream = BinLogStreamReader(<br>                connection_settings=mysql_settings,<br>                server_id=100,<br>                only_events=[DeleteRowsEvent, WriteRowsEvent, UpdateRowsEvent],<br>                #blocking=True,<br>                #resume_stream=True,<br>                #log_file=&#39;mysql-bin.000001&#39;,<br>                #log_pos=binlog_pos,<br>            )<br>            # Read the Binary Log and identify modified rows<br>            for binlogevent in stream:<br>                for row in binlogevent.rows:<br>                    if isinstance(binlogevent, DeleteRowsEvent):<br>                        modified_rows.append({key: str(value) if isinstance(value, date) else value for key, value in row[&quot;values&quot;].items()})<br>                    elif isinstance(binlogevent, UpdateRowsEvent):<br>                        modified_rows.append({key: str(value) if isinstance(value, date) else value for key, value in row[&quot;after_values&quot;].items()})<br>                    elif isinstance(binlogevent, WriteRowsEvent):<br>                        modified_rows.append({key: str(value) if isinstance(value, date) else value for key, value in row[&quot;values&quot;].items()})<br>            #close stream <br>            stream.close()<br>            cursor.close()<br>            conn.close()<br>            return modified_rows<br>        except Exception as e:<br>            print(&quot;Data extract error: &quot; + str(e))<br>            return []<br>extract_incremental_data_task = PythonOperator(<br>    task_id=&#39;extract_incremental_data&#39;,<br>    python_callable=extract_incremental_data,<br>    provide_context=True,<br>    dag=dag_incremental,<br>)<br>#check bigquery table exist <br>def create_bigquery_table():<br>    try:<br>        client = bigquery.Client()<br>        dataset_ref = client.dataset(CONFIG.DATASET_REF)<br>        table_ref = dataset_ref.table(CONFIG.INCREMENTAL_TABLE_NAME)<br>        <br>        table = bigquery.Table(table_ref)<br>        table = client.create_table(table, exists_ok=True)  # Create if not exists<br>        print(f&quot;BigQuery table {CONFIG.INCREMENTAL_TABLE_NAME} created or already exists.&quot;)<br>    except Exception as e:<br>        print(&quot;BigQuery table creation error: &quot; + str(e))<br>create_bigquery_table_task = PythonOperator(<br>    task_id=&#39;create_bigquery_table&#39;,<br>    python_callable=create_bigquery_table,<br>    dag=dag_incremental,<br>)<br><br># Modify the GCP load task to use WRITE_APPEND to add only the changes<br>def gcp_load_incremental(**kwargs):<br>        <br>    try:<br>        ti = kwargs[&#39;ti&#39;]<br>        data = ti.xcom_pull(task_ids=&#39;extract_incremental_data&#39;)<br>        if not data:<br>            print(&quot;No data changes found.&quot;)<br>            return<br>        client = bigquery.Client()<br>        job_config = bigquery.job.LoadJobConfig()<br>        project_id = CONFIG.PROJECT_ID<br>        dataset_ref = CONFIG.DATASET_REF<br>        TABLE = CONFIG.INCREMENTAL_TABLE_NAME<br>        table_id = f&quot;{project_id}.{dataset_ref}.{TABLE}&quot;<br>        job_config = bigquery.job.LoadJobConfig()<br>        # Set write_disposition parameter as WRITE_APPEND for appending only the changes<br>        job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND<br>        print(f&quot;Importing {len(data)} rows to {table_id}...&quot;)<br>        # Proceed with the data loading using the modified data<br>        job = client.load_table_from_json(data, table_id, job_config=job_config)<br>        job.result()<br>        print(f&quot;Loaded {len(data)} rows to {table_id}&quot;)<br>    except Exception as e:<br>        print(&quot;Data load error: &quot; + str(e))<br>gcp_load_incremental_task = PythonOperator(<br>    task_id=&#39;gcp_load_incremental&#39;,<br>    python_callable=gcp_load_incremental,<br>    provide_context=True,<br>    dag=dag_incremental,<br>)<br># Set task dependencies for the incremental DAG<br>check_table_exist_task &gt;&gt; extract_incremental_data_task &gt;&gt; create_bigquery_table_task &gt;&gt; gcp_load_incremental_task</pre><p>The incremental dag has four tasks. The first task checks if the Table to extract data exist. The second task, extract database changes from the Binary log. The third task checks if the BigQuery table exists and creates it. The final task loads the events extracted from the log to BigQuery.</p><p>The events we are extracting from the logs are the DeleteRowsEvent,<br>the UpdateRowsEvent, and the WriteRowsEvent.</p><p>Before we run the incremental dag file, let’s simulate events to the MySQL database.</p><p>I’ve created a load_incremental_to_db.py scripts that loads data every five minutes to the MySQL database.</p><pre>import pandas as pd<br>import pymysql<br>from sqlalchemy import create_engine<br>import MySQLdb<br>import time<br>host = &quot;localhost&quot;<br>user = &quot;root&quot;<br>password = &quot;&quot;<br>db = &quot;trips_db&quot;<br># Construct the connection URL with an empty password<br>connection_url = f&quot;mysql+mysqldb://{user}:{password}@{host}/{db}&quot;<br># Create an SQLAlchemy engine and connect to the database<br>engine = create_engine(connection_url)<br>conn = engine.connect()<br>chunk_size = 1000  # Number of rows to load per iteration<br>total_iterations = 100  # Number of iterations<br>for iteration in range(total_iterations):<br>    start_row = iteration * chunk_size<br>    df = pd.read_csv(&quot;./data/yellow_tripdata_2021-01.csv.gz&quot;, <br>                     skiprows=range(1, start_row + 1),  # Skip to the next batch of data<br>                     nrows=chunk_size, <br>                     compression=&#39;gzip&#39;)<br>    df.tpep_pickup_datetime = pd.to_datetime(df.tpep_pickup_datetime)<br>    df.tpep_dropoff_datetime = pd.to_datetime(df.tpep_dropoff_datetime)<br>    # Load the batch into the database<br>    df.to_sql(name=&quot;yellow_tripdata&quot;, con=conn, index=False, if_exists=&#39;append&#39;)<br>    print(f&quot;loading {iteration} batch&quot;)<br>    # Sleep for 5 minutes<br>    time.sleep(30)  # 300 seconds = 5 minutes</pre><p>Run the load_incremental_to_db.py on your local computer and start the incremental dags.</p><p>Let’s see how the incremental dag looks like in the airflow UI and run it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-bQahV2yPsPyn4_vSwkXw.png" /><figcaption>airflow incremental dags</figcaption></figure><p>Cool, it worked as we can see in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYK_esB1-RilwQl0fczWKg.png" /></figure><p>For the purpose of this tutorial, the full and incremental dags loaded data into separate tables. In a real-world scenario, the destination depends on your business.</p><h4>Conclusion</h4><p>That’s it guys, we have successfully used CDC to replicate data from MySQL to BigQuery. <br>You should have your full and incremental dags running by now.<br>I hope you found this tutorial interesting. Please share and remember to comment with your suggestions or feedback.</p><p>Don’t forget to clap and <strong>follow me</strong> on <a href="http://www.linkedin.com/in/abdulquadri-ayodeji/">LinkedIn</a> for posts on Data Engineering.</p><p>Cheers!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cea24a124e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Machine learning model as a Chatbot (Part 2)]]></title>
            <link>https://blog.chatbotslife.com/deploying-a-machine-learning-model-as-a-chatbot-part-2-20038a9b39ef?source=rss-480dff2fffee------2</link>
            <guid isPermaLink="false">https://medium.com/p/20038a9b39ef</guid>
            <category><![CDATA[analytics-vidhya]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[chatbot-development]]></category>
            <dc:creator><![CDATA[Abdulquadri Ayodeji Oshoare]]></dc:creator>
            <pubDate>Wed, 03 Nov 2021 14:24:31 GMT</pubDate>
            <atom:updated>2021-11-05T15:18:11.337Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="AI chatbots" src="https://cdn-images-1.medium.com/proxy/1*bQX-b9P-n7mx7UfvUS-T2w.jpeg" /></figure><p>Welcome to the concluding part of our Machine learning integrated chatbot series. I&#39;d recommend you read the first part using this <a href="https://abdulquadri-oshoare.medium.com/deploying-a-machine-learning-model-as-a-chatbot-part-1-d5dc9e8fa4ba">link</a> you haven&#39;t.</p><p>In Part 1 of this series, we covered model training and integrating the Dialogflow chatbot into Telegram.</p><p>We will continue building the backend using the Flask web framework.</p><h3>Flask</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*urwwVELM8B8ttR0a.png" /><figcaption>© <a href="https://palletsprojects.com/p/flask/">https://palletsprojects.com/p/flask/</a></figcaption></figure><p>Flask is a python web framework used to build a lightweight, scalable webhook service to communicate with our chatbot. When the chatbot gets input from the user, it communicates to the machine learning model using Flask to make a prediction.</p><h3>Create the flask webhook</h3><p>Before creating our Flask webhook, we need to create a virtual environment to install Flask and its dependencies.</p><pre>#Navigate to a folder on your computer </pre><pre>cd abdulquadri/oshoare/desktop/loanbot</pre><pre>#install the virtual environment </pre><pre>pip install venv myflaskbot</pre><pre>#myflaskbot is my virtual environment name</pre><figure><a href="https://chatbotslife.com/chatbot-conference-online-2021-7161960c2817"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lwlUzb2E7V5ZAsT4.png" /></a></figure><h4><strong>Install the Dependencies</strong></h4><p>Using the requirement.txt file found on my Github page, install the dependencies.</p><pre>pip install -r requirement.txt</pre><p><strong>Create the app.py file</strong></p><pre>#import required libraries<br>import numpy as np<br>from flask import Flask, request, make_response,render_template,jsonify<br>import json<br>import pickle<br>from flask_cors import cross_origin</pre><pre>#instantiate flask<br>app = Flask(__name__)<br>model = pickle.load(open(&#39;rfc.pkl&#39;, &#39;rb&#39;))</pre><pre><a href="http://twitter.com/app">@app</a>.route(&#39;/&#39;)<br>def hello():<br>    return render_template(&#39;home.html&#39;)</pre><pre>if __name__ == &#39;__main__&#39;:<br>    app.run()</pre><p>After creating the app.py file, create an optional folder called template. A simple HTML file with the tag H1 &quot;Hello World&quot; can be added to see the output in the local server.</p><h4>Run the app.py</h4><p>On the terminal, run python app.py</p><pre>python app.py </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZvwHctgg_fw2u2o7sEw6Mw.png" /></figure><p>Verify that Flask has started on port 5000, as shown in the image above.</p><h4>Create the predict &amp; Webhook method</h4><p>Flask uses the predict &amp; webhook method to return JSON information of the prediction to the Dialogflow chatbot.</p><p>Would you please copy the code below and paste it into your app.py file?</p><pre><a href="http://twitter.com/app">@app</a>.route(&#39;/predict&#39;, methods=[&#39;POST&#39;])<br>def predict():<br>     json_ = request.json<br>     query = (json_)<br>     prediction = model.predict(query)<br>     return jsonify({&#39;prediction&#39;: list(prediction)})</pre><pre># geting and sending response to dialogflow<br><a href="http://twitter.com/app">@app</a>.route(&#39;/webhook/&#39;, methods=[&#39;GET&#39;,&#39;POST&#39;])<br><a href="http://twitter.com/cross_origin">@cross_origin</a>()<br>def webhook():</pre><pre>req = request.get_json(silent=True, force=True)</pre><pre>print(&quot;req&quot;)</pre><pre>res = mlprediction(req)</pre><pre>res = json.dumps(res, indent=4)<br>    #print(res)<br>    r = make_response(res)<br>    r.headers[&#39;Content-Type&#39;] = &#39;application/json&#39;<br>    return r</pre><pre># processing the request from dialogflow<br>def mlprediction(req):</pre><pre>#sessionID=req.get(&#39;responseId&#39;)<br>    result = req.get(&quot;queryResult&quot;)<br>    #print(result)</pre><pre>parameters = result.get(&quot;parameters&quot;)<br>    ApplicantIncome=parameters.get(&quot;Income&quot;)<br>    CoapplicantIncome = parameters.get(&quot;CoapplicantIncome&quot;)<br>    LoanAmount=parameters.get(&quot;LoanAmount&quot;)<br>    Credit_History=parameters.get(&quot;Credit_History&quot;)</pre><pre>if str.lower(Credit_History) == &#39;yes&#39;:<br>        Credit_History = int(1)<br>    elif str.lower(Credit_History) == &#39;no&#39;:<br>        Credit_History = int(0)<br>    else:<br>        return {<br>            &quot;fulfillmentText&quot;: &quot;Error please start again and enter the correct information&quot;<br>            <br>        }</pre><pre>try:<br>        int_features = [ApplicantIncome, CoapplicantIncome, LoanAmount,Credit_History]<br>        final_features = [np.array(int_features)]<br>    <br>    except ValueError:<br>        return {<br>            &quot;fulfillmentText&quot;: &quot;Incorrect information supplied&quot;<br>        }</pre><pre>print(final_features)</pre><pre>intent = result.get(&quot;intent&quot;).get(&#39;displayName&#39;)<br>    <br>    if (intent==&#39;Default Welcome Intent - yes&#39;):<br>        prediction = model.predict(final_features)<br>        print(prediction)<br>    <br>     <br>        if(prediction==&#39;Y&#39;):<br>            status = &#39;Congratulations you are eligible for a loan 😀&#39;<br>        else:<br>            status = &#39;We are sorry you are not eligible for a loan at the moment&#39;</pre><pre>fulfillmentText= status<br>        <br>        print(fulfillmentText)<br>        print(prediction)<br>        return {<br>            &quot;fulfillmentText&quot;: fulfillmentText<br>        }</pre><pre>if __name__ == &#39;__main__&#39;:<br>    app.run()</pre><h3>Trending Bot Articles:</h3><blockquote><a href="https://chatbotslife.com/how-conversational-ai-can-automate-customer-service-ce1e6e330902">1. How Conversational AI can Automate Customer Service</a></blockquote><blockquote><a href="https://chatbotslife.com/automated-vs-live-chats-what-will-the-future-of-customer-service-look-like-69c90432c8f4">2. Automated vs Live Chats: What will the Future of Customer Service Look Like?</a></blockquote><blockquote><a href="https://chatbotslife.com/chatbots-as-medical-assistants-in-covid-19-pandemic-3d2731b85128">3. Chatbots As Medical Assistants In COVID-19 Pandemic</a></blockquote><blockquote><a href="https://chatbotslife.com/chatbot-vs-intelligent-virtual-assistant-whats-the-difference-why-care-180aebde9358">4. Chatbot Vs. Intelligent Virtual Assistant — What’s the difference &amp; Why Care?</a></blockquote><h3>Test the webhook using a Webtunnel</h3><p>To test the webhook, we are going to use ngrok. ngrok is a web tunnel tool that exposes local servers behind NATs and firewalls to the public internet over secure tunnels. ngrok makes it easy to test a live application without deploying it to a cloud service.</p><p>Procced to <a href="https://ngrok.com/download">https://ngrok.com/download</a> to download ngrok. extract the .zip folder and run ngrok.exe HTTP 5000 to expose port 5000 on your computer to the internet</p><pre>#expose http 5000 to the internet</pre><pre>ngrok.exe HTTP 5000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*eQUt3jb4StqMEe2aNM_wfg.png" /></figure><p>Make sure to copy the URL <a href="https://0931-105-112-38-173.ngrok.io/">https://0931-105-112-38-173.ngrok.io</a> and keep it handy. Dialogflow will use the URL to communicate with the flask app.</p><p>Test that the URL is exposed to the internet by pasting <a href="https://0931-105-112-38-173.ngrok.io/">https://0931-105-112-38-173.ngrok.io</a> in your browser. You should get the same welcome page as the one on the local server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DQNywZBNRj_ND6GZ_GIwOA.png" /></figure><h4>Connecting the backend and the frontend</h4><p>Now that the Telegram bot and the flask app are live, we connect our bot to the backend.</p><p>Remember the URL field we left blank in part 1? Kindly go to the Dialogflow console and click on the fulfilment tab. We will add the URL gotten from ngrok to the webhook URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*AoReFDdtZ8h7t7v1Fo1NUA.png" /></figure><p>Click save and test your chatbot with Telegram.</p><h3>Conclusion</h3><p>Hooray!, we have successfully built our first Machine learning-enabled chatbot.</p><p>You can interact with the chatbot using the links below</p><p>Link to the chatbot hosted on a website: <a href="https://loan-predictor-bot.herokuapp.com/">https://loan-predictor-bot.herokuapp.com/</a></p><p>Link to the Telegram bot: <a href="https://t.me/LoanPredictionBot">https://t.me/LoanPredictionBot</a></p><p>I hope you found this tutorial interesting. Please share and remember to comment with your suggestions or feedback.</p><p>Don&#39;t forget to clap and <strong>follow me</strong> on <a href="http://www.linkedin.com/in/abdulquadri-ayodeji/">LinkedIn</a> for posts on Data Science and AI.</p><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*ZxaGcKC3UQEydQIX" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F3e59ef%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F3e59ef&amp;image=http%3A%2F%2Fapi.screenshotlayer.com%2Fapi%2Fcapture%3Faccess_key%3Dfe59908dad3baab69ffab249a2224b03%26viewport%3D1024x612%26width%3D1000%26url%3Dhttps%253A%252F%252Fupscri.be%252F3e59ef%253Fscreenshot&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href">https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href</a></iframe><figure><a href="https://chatbotslife.com/bot-communities-mastermind-group-d2dae9876709#.53x0py6ou"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*6XUspT9JOSq0w0Fi35HIaA.png" /></a></figure><figure><a href="https://m.me/ChatbotsLife"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*c1LDMH5vbnIz9rmAka8Hwg.png" /></a></figure><figure><a href="https://chatbotslife.com/chatbot-development-hire-top-ai-chatbot-developers-c8b45ef7f207"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*D0Jf3dI6ZThtqcfwDYY7mg.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20038a9b39ef" width="1" height="1" alt=""><hr><p><a href="https://blog.chatbotslife.com/deploying-a-machine-learning-model-as-a-chatbot-part-2-20038a9b39ef">Deploying a Machine learning model as a Chatbot (Part 2)</a> was originally published in <a href="https://blog.chatbotslife.com/">Chatbots Life</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Machine learning model as a Chatbot (Part 1)]]></title>
            <link>https://blog.chatbotslife.com/deploying-a-machine-learning-model-as-a-chatbot-part-1-d5dc9e8fa4ba?source=rss-480dff2fffee------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5dc9e8fa4ba</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[chatbot-design]]></category>
            <category><![CDATA[analytics-vidhya]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Abdulquadri Ayodeji Oshoare]]></dc:creator>
            <pubDate>Wed, 03 Nov 2021 14:22:59 GMT</pubDate>
            <atom:updated>2021-11-05T15:18:03.178Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="AI chatbots" src="https://cdn-images-1.medium.com/max/1000/1*bQX-b9P-n7mx7UfvUS-T2w.jpeg" /><figcaption>© mygreatlearning.com/</figcaption></figure><blockquote>“What use is a machine learning model if you don’t deploy to production.”</blockquote><blockquote>- Anonymous</blockquote><p>Are you tired of waiting for a software developer to integrate your machine learning model into a website for testing after training your model for days?</p><p>Are you bored of the usual model deployment to a website?</p><p>Are you looking for a fun way to deploy your machine learning model?</p><p>Worry no more because, with your python skills, we can take an unusual approach and deploy the machine learning model as a chatbot instead.</p><p>Chatbot has been around the corner and is becoming increasingly popular post-COVID-19. What&#39;s more, chatbots are easy to access, easy to build, and can be integrated on almost any platform.</p><p>You can check out my previous article on how chatbots can turbocharge your business <a href="https://abdulquadri-oshoare.medium.com/how-chatbots-can-turbo-charge-your-business-7e9ef29823d8">here</a>.</p><p>Grab a seat while I walk you through this beautiful adventure.</p><figure><a href="https://chatbotslife.com/chatbot-conference-online-2021-7161960c2817"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lwlUzb2E7V5ZAsT4.png" /></a></figure><h3>The Flow</h3><p>This project would be divided into two sections, the frontend and the backend.</p><h4>* The frontend</h4><p>The frontend would consist of our chatbot, which would be built with Dialogflow and integrated with Telegram</p><h4>* The backend</h4><p>The backend Would consist of the ML model deployed as a flask API hosted on a server.</p><h3>The Dataset.</h3><p>The Dataset we are going to use is the Loan prediction dataset. The loan prediction dataset is a unique dataset that contains 12 columns. The data was gathered to predict if a customer is eligible for a loan.</p><p>The Dataset is publicly available on Kaggle and can be accessed using this <a href="https://www.kaggle.com/altruistdelhite04/loan-prediction-problem-dataset/discussion">link</a>.</p><h3>Cleaning and creating &amp; saving the machine learning models</h3><p>Let&#39;s Start with the bottom-up approach and build a simple Machine learning model. For this project, we would not be concerned with achieving high accuracy on our models. We only need enough accuracy so that our chatbot can make decent predictions on the fly.</p><h4>Loading and Analyzing the Data</h4><p>We start by importing the necessary libraries like Pandas, NumPy, Matplotlib, and seaborn. We then use pandas read_csv() to read the data.</p><pre>#import neccessary library</pre><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns</pre><pre>#loading the dataset</pre><pre>df_train = pd.read_csv(&quot;train.csv&quot;)<br>df_test = pd.read_csv(&#39;test.csv&#39;)</pre><h4>Data preparation and cleaning</h4><p>Checking the summary of the Dataset, we can see that we have a few missing data. We will take care of the missing data by filling it with the mean and mode. <br><strong>Please note:</strong> that there are more advanced techniques to fill missing data; for the sake of this project, we will prioritize simplicity over accuracy.</p><pre>df_train.info()</pre><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 614 entries, 0 to 613<br>Data columns (total 13 columns):<br> #   Column             Non-Null Count  Dtype  <br>---  ------             --------------  -----  <br> 0   Loan_ID            614 non-null    object <br> 1   Gender             601 non-null    object <br> 2   Married            611 non-null    object <br> 3   Dependents         599 non-null    object <br> 4   Education          614 non-null    object <br> 5   Self_Employed      582 non-null    object <br> 6   ApplicantIncome    614 non-null    int64  <br> 7   CoapplicantIncome  614 non-null    float64<br> 8   LoanAmount         592 non-null    float64<br> 9   Loan_Amount_Term   600 non-null    float64<br> 10  Credit_History     564 non-null    float64<br> 11  Property_Area      614 non-null    object <br> 12  Loan_Status        614 non-null    object <br>dtypes: float64(4), int64(1), object(8)<br>memory usage: 62.5+ KB</pre><h4>Filling missing Data</h4><pre>#Imputing Missing values with mean for continuous variable<br>df_train[&#39;LoanAmount&#39;].fillna(df_train[&#39;LoanAmount&#39;].mean(), inplace=True)</pre><pre>#Imputing Missing values with mode for categorical variables<br>df_train[&#39;Gender&#39;].fillna(df_train[&#39;Gender&#39;].mode()[0], inplace=True)<br>df_train[&#39;Married&#39;].fillna(df_train[&#39;Married&#39;].mode()[0], inplace=True)<br>df_train[&#39;Dependents&#39;].fillna(df_train[&#39;Dependents&#39;].mode()[0], inplace=True)<br>df_train[&#39;Loan_Amount_Term&#39;].fillna(df_train[&#39;Loan_Amount_Term&#39;].mode()[0], inplace=True)<br>df_train[&#39;Credit_History&#39;].fillna(df_train[&#39;Credit_History&#39;].mode()[0], inplace=True)<br>df_train[&#39;Self_Employed&#39;].fillna(df_train[&#39;Self_Employed&#39;].mode()[0], inplace=True)</pre><h4>Feature engineering</h4><p>Converting categorical variables into dummy/indicator variables.</p><pre>cat_cols = [&#39;Gender&#39;,&#39;Married&#39;,&#39;Education&#39;,&#39;Self_Employed&#39;,&#39;Property_Area&#39;]</pre><pre>train_df = pd.get_dummies(df_train,columns=cat_cols,drop_first=True)</pre><h4>Training the Model</h4><pre><strong>#split the data into train and test for training</strong></pre><pre>from sklearn.model_selection import train_test_split</pre><pre>X = train_df.drop(columns=&#39;Loan_Status&#39;)<br>y = train_df[&#39;Loan_Status&#39;]</pre><pre>X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.3,random_state=42)</pre><pre><strong>#import the randomForest algorithm to train the datasets</strong></pre><pre>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.feature_selection import SelectFromModel<br>from sklearn.metrics import accuracy_score</pre><pre>sel = SelectFromModel(RandomForestClassifier(n_estimators=100,random_state=0,n_jobs=-1))<br>sel.fit(X_train,y_train)<br>sel.get_support()</pre><pre>X_train_rfc = sel.transform(X_train)<br>X_test_rfc =sel.transform(X_test)</pre><pre><strong>#Using Recursive Feature Eliminination</strong></pre><pre>def run_randomForest(X_train,X_test,y_train,y_test):<br>    rfc = RandomForestClassifier(n_estimators=100,random_state=0,n_jobs=-1)<br>    rfc.fit(X_train,y_train)<br>    y_pred = rfc.predict(X_test)<br>    print(&quot;Accuracy: &quot;, accuracy_score(y_test,y_pred))<br>    print(&#39;\n&#39;)<br>    print(&quot;Confusion Matrix&quot;)<br>    print(&#39;\n&#39;)<br>    print(confusion_matrix(y_test,y_pred))</pre><pre><strong>#Save our model as pickle so it can be reused to make predictions</strong><br>import pickle<br>with open(r&#39;rfc.pkl&#39;,&#39;wb&#39;) as model_pkl:<br>    pickle.dump(rfc, model_pkl, protocol=2)</pre><p>After preparing the Dataset, I used recursive feature elimination to select features with the highest predicting factor. I found that using four features, we got an accuracy of 76%, while 11 features got an accuracy of 78%.</p><p>You can find the whole notebook containing the code on GitHub using this <a href="https://github.com/quadriano31/Loan-predictor-chatbot/">link</a>.</p><h3>Building the Frontend</h3><p>The front end would consist of the Dialogflow bot and telegram integration.</p><h3>What is Dialogflow?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*sumrRf5_cNWdGOKEj6fzcA.png" /></figure><p>Dialogflow is a natural language understanding platform designed by Google that makes it easy to design and build enterprise-grade chatbots.</p><p>Dialogflow makes creating chatbots easy, and It uses NLU Natural language understanding on pre-trained models to understand Users&#39; intent with little training data. One of the reasons I choose Dialogflow is its robustness and its easy Integration with another third-party app.</p><h3>Trending Bot Articles:</h3><blockquote><a href="https://chatbotslife.com/how-conversational-ai-can-automate-customer-service-ce1e6e330902">1. How Conversational AI can Automate Customer Service</a></blockquote><blockquote><a href="https://chatbotslife.com/automated-vs-live-chats-what-will-the-future-of-customer-service-look-like-69c90432c8f4">2. Automated vs Live Chats: What will the Future of Customer Service Look Like?</a></blockquote><blockquote><a href="https://chatbotslife.com/chatbots-as-medical-assistants-in-covid-19-pandemic-3d2731b85128">3. Chatbots As Medical Assistants In COVID-19 Pandemic</a></blockquote><blockquote><a href="https://chatbotslife.com/chatbot-vs-intelligent-virtual-assistant-whats-the-difference-why-care-180aebde9358">4. Chatbot Vs. Intelligent Virtual Assistant — What’s the difference &amp; Why Care?</a></blockquote><h3>Creating the chatbots using Dialogflow</h3><p>We will follow the steps below in creating our chatbot</p><ul><li>Create the chatbot</li><li>Create intents</li><li>Fulfilment</li><li>Integration</li></ul><h4>Create the chatbot</h4><p>To create our chatbot, kindly go to <a href="https://dialogflow.cloud.google.com/">https://dialogflow.cloud.google.com/</a> to create an account. After completing the account, click on create an agent to create the chatbot, as shown in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*OEoVH8Kfx2NqdDPu1G-dNg.png" /></figure><h4>Create Intents</h4><p>When creating a chatbot, Dialogflow presents you with two default intents; welcome intent and fallback intent.</p><p>Intents are the actions users want to execute. Intents represent something a user might ask for. Intents help the chatbot agent determine what the users want. e.g. One Pizza, please (Intent: Order Pizza).</p><h4>Welcome Intent</h4><p>The welcome intent is triggered when a user interacts with the bot; it is usually a greeting, as its name suggests. e.g. Hello, I&#39;m the loanPredictionBot. Would you like to make a prediction?</p><p>Click on the plus sign on the intent tab to Modify the welcome intent and add the response below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*QXVepmLfBA1s095tVYzR5g.png" /></figure><p>After adding the response, kindly click save and allow the chatbot to train.</p><p>After it has finished training, click on the default intent and create a follow-up intent.</p><h4>Follow-up Intent</h4><p>Follow-up intents are used to guide the user into making a prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*XB-tK2s5t1UzT6aHbSazYA.png" /><figcaption>follow-up intent</figcaption></figure><p>After the welcome intent is triggered, the user would be asked to make a Prediction; if the user clicks yes, the yes-follow up intent is triggered, and the user is asked for parameters to make the prediction. If the user selects no, the No-follow up intent is triggered.</p><h4>Follow-up intent</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*Y63IJkNl1IkJjcD7_iktGg.png" /></figure><p>The yes follow-up intent is where the real magic happens. This is where the user inputs details to be used for making predictions.</p><p>Kindly add the parameters(features) that we selected using recursive feature elimination. These are the features that the user would be prompted for when he tries to make a prediction.</p><p>Dialogflow uses entities to extract information from the intent. Use the default sys.number entity so that the chatbot would only accept numbers. Don&#39;t forget to click the save button after adding the parameters.</p><p>Using the same technique, let&#39;s create the No follow up intent.</p><p>The No follow-up intent would be triggered when the user clicks on No.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*dXrlglq-wpTf0bGNlWTCyw.png" /></figure><p>The fallback intents would be triggered when the Dialogflow agent cannot understand the user&#39;s input. It would be populated to return a response to guide users on how to use the bot.</p><p>For now, the fallback intent would be configured as described below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*O0xTR28A1lsIgNDZu-IXAw.png" /></figure><h4>Fulfillments</h4><p>Fulfilment is a feature in Dialogflow that allows the agent to communicate with external applications using a webhook. With fulfilment, you can connect the chatbot to a database, map API or backend service.</p><p>To use fulfilment, we must turn it on in the Dialogflow console, go to the fulfilment tab and turn on webhook. Leave the URL field empty for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*srFNsVmTXgIUG5GaB5ZWBQ.png" /></figure><p>Goto the default yes follow-up intent and turn on fulfilments. This will allow the yes follow-up intent to send input received from users to the backend flask service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*_JOuhF8H-A8V6kWGCq8lvQ.png" /></figure><h4>Testing the Agent</h4><p>Using the Dialogflow console, test the chatbot by typing &quot;hello&quot;. This would trigger the Welcome intent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*VqGZXO-HC3EdHg_U9qJ30Q.png" /></figure><h3>Telegram Integration</h3><p>To Integrate with Telegram, We need to create a Telegram bot.</p><p>Kindly go to Telegram and search for botFather. Using the on-screen guide, create a bot and copy the access token. We will use the access token to link Dialogflow with the telegram bot.</p><p>Kindly go to Dialogflow and click on the Integration&gt;&gt; Telegram.</p><p>Paste the access token as shown in the image below and click start.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*vW-URiwndtmjfMB-m83GyA.png" /></figure><p>Go back to the Telegram bot and interact. You should get the same response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*y8zjA7fkbR2PqKp2vc6o5g.png" /></figure><h3>Conclusion</h3><p>We have come to the end of the first part of creating a Machine learning model as a chatbot. In part two of this series (<a href="https://abdulquadri-oshoare.medium.com/deploying-a-machine-learning-model-as-a-chatbot-part-2-20038a9b39ef">link here</a>), we will deploy the machine learning model as a Flask API and link it with our chatbot.</p><p>I hope you found this tutorial interesting. Please share and remember to comment with your suggestions or feedback.</p><p>Don&#39;t forget to clap and <strong>follow me</strong> on <a href="http://www.linkedin.com/in/abdulquadri-ayodeji/">LinkedIn</a> for posts on Data Science and AI.</p><p>Cheers!!!</p><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*ZxaGcKC3UQEydQIX" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F3e59ef%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F3e59ef&amp;image=http%3A%2F%2Fapi.screenshotlayer.com%2Fapi%2Fcapture%3Faccess_key%3Dfe59908dad3baab69ffab249a2224b03%26viewport%3D1024x612%26width%3D1000%26url%3Dhttps%253A%252F%252Fupscri.be%252F3e59ef%253Fscreenshot&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href">https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href</a></iframe><figure><a href="https://chatbotslife.com/bot-communities-mastermind-group-d2dae9876709#.53x0py6ou"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*6XUspT9JOSq0w0Fi35HIaA.png" /></a></figure><figure><a href="https://m.me/ChatbotsLife"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*c1LDMH5vbnIz9rmAka8Hwg.png" /></a></figure><figure><a href="https://chatbotslife.com/chatbot-development-hire-top-ai-chatbot-developers-c8b45ef7f207"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*D0Jf3dI6ZThtqcfwDYY7mg.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5dc9e8fa4ba" width="1" height="1" alt=""><hr><p><a href="https://blog.chatbotslife.com/deploying-a-machine-learning-model-as-a-chatbot-part-1-d5dc9e8fa4ba">Deploying a Machine learning model as a Chatbot (Part 1)</a> was originally published in <a href="https://blog.chatbotslife.com/">Chatbots Life</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Chatbots can Turbo Charge your Business]]></title>
            <link>https://abdulquadri-oshoare.medium.com/how-chatbots-can-turbo-charge-your-business-7e9ef29823d8?source=rss-480dff2fffee------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e9ef29823d8</guid>
            <category><![CDATA[chatbots-for-banking]]></category>
            <category><![CDATA[analytics-vidhya]]></category>
            <category><![CDATA[chatbot-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Abdulquadri Ayodeji Oshoare]]></dc:creator>
            <pubDate>Sat, 16 Oct 2021 19:32:53 GMT</pubDate>
            <atom:updated>2021-10-16T19:32:53.126Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*XmQOe7FykD8u_PItftnVyg.jpeg" /><figcaption><strong>© </strong><a href="https://articles.bplans.com/"><strong>articles.bplans.com</strong></a></figcaption></figure><p>Have you ever wondered how your business can respond to customers&#39; complaints and resolve issues in the late part of the night?</p><p>Are you constantly worried about your business not having enough customer support strength?</p><p>Does your company have a peak sales period, e.g., Black Friday and Cyber Monday, and you do not want to hire additional staff to cater to the workload?</p><p>What if I told you a single chatbot could solve all your business needs.</p><p>Chatbots are opening up a new era of business-customer interaction. Small and large businesses have been using chatbots to scale and market their products efficiently.</p><p>Also, research shows that, unlike email, which has an open rate of 22%, SMS text messaging, such as WhatsApp, has a<strong> 98% </strong>open rate. As a business owner, the intelligent thing is to take your business to where your customers are.</p><p>And How do you reach your customers on social media faster? You guessed right, Chatbots.</p><blockquote>“You can use chatbots to automate the business processes from taking customers&#39; orders to suggesting personalized products to notifying the store that the product is out of stock.”</blockquote><h3><strong>What is a chatbot?</strong></h3><p>A chatbot is a conversational agent trained to understand human language.</p><p>According to <a href="https://www.expert.ai/blog/chatbot/">experts.ai</a>, a chatbot is an artificial intelligence (AI) software that can simulate a conversation (or a chat) with a user in natural language through messaging applications, websites, mobile apps, or telephone.</p><blockquote><a href="https://www.humable.io/glossary/chatbot/?lang=en">“Chatbot</a> comes from the term <strong>“chatterbot”</strong> and is the mix of two words: <strong>“chat”</strong> (to converse) and <strong>“bot”</strong> (robot).”</blockquote><h3>Types of Chatbots with Example</h3><p>There are different types of chatbots</p><ul><li>AI bots</li><li>Rule-based bots</li></ul><h4>Rule-based bot</h4><p>Rule-based bots conditions or set of rules defined. Rule-based bots are the simplest form to build and can be trained and deployed in minutes.</p><p>The conversation is limited because they cannot understand the context of the conversation and usually require human intervention when they fail.</p><p>Examples where rule-based bots are used, are FAQs and a ticket ordering system.</p><pre>You: Hello <br>Bot: Hi!, I&#39;m the movie ticket ordering bot. Kindly press 1 to order a ticket.<br>You: 1<br>Bot: Select a Category!. Press 1 for Now playing or 2 for New release.<br>You: 3<br>Bot: Incorrect command I would be transferring you to a human agent to assist.</pre><h4>AI-based bots</h4><p>AI bots are a more advanced type of bot. They use advanced Machine learning techniques like Natural language Understanding to respond to customers&#39; queries.</p><p>AI bots require a long training time and are the most successful type of bots.</p><p>Using advanced AI techniques such as natural language understanding, a chatbot can understand the context of the conversation even with untrained data. Many AI bots have self-learning ability meaning they can learn from its mistake and update their knowledge base. Also, AI bots can be trained using an Advanced algorithm to understand multiple languages like French, Spanish and Portuguese.</p><p>Examples of platforms where you can build your AI bots are:</p><ul><li>Dialogflow</li><li>IBM Watson</li><li>Amazon Alexa.</li></ul><h3>Advantage of integrating a Chatbot in your business</h3><p>Let&#39;s go through the benefits of incorporating a chatbot into your business.</p><blockquote>“According to <a href="https://www.ibm.com/blogs/watson/2017/10/how-chatbots-reduce-customer-service-costs-by-30-percent/">IBM</a>, every year, 265 billion customer service calls are made, and businesses spend $1.3 trillion on servicing them. AI chatbots can serve 80% of them without human involvement.”</blockquote><h4><strong>#24/7 Support</strong></h4><p>With the use of chatbots, companies can offer 24/7 support all year round with zero downtime. The chatbot can easily understand customers&#39; queries and respond based on the training data. There is no need to worry</p><h4>#Handle repetitive tasks that are boring to humans.</h4><p>A repetitive task like updating the price list of a product at 5 AM daily or frequently verifying customers&#39; serial numbers can prove tiresome to humans and also impact job satisfaction.</p><p>A bot can handle such tasks efficiently and free up time for a human to work on more creative tasks.</p><h4><strong>#Reduced training cost</strong></h4><p>A well-trained chatbot only needs to be deployed once and would handle hundreds or thousands of requests daily.</p><h4>#Personalized response</h4><p>After interacting with customers, chatbots can segment them into groups based on their responses and send personalized messages to them.</p><h4><strong>#Easy integration</strong></h4><p>Chatbots can be deployed in minutes and integrated into several messaging and voice channels which include Whatsapp, Facebook, telegram, slack, Genesys, Avaya e.t.c</p><h4>#Better anticipation of customer needs</h4><p>Chatbots can be used to answer repetitive customers complaint that would be tiring for humans. Most times, customers need quick answers and do not want to wait for a human. With a chatbot that can quickly achieve. A well-trained chatbot should understand the customers&#39; queries and should be able to hand them off to a human agent if need be.</p><p>Customers can use them to check account balances, pay bills, recharge their mobile phone, resolve complaints, make query schedule appointments e.t.c</p><blockquote>“69% of consumers prefer to use chatbots for the speed at which they can communicate with a brand. (<a href="https://www.salesforce.com/blog/2018/01/why-consumers-prefer-chatbots.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.htm.html">Salesforce State of Chatbots Report 2018</a>)”</blockquote><h3><strong>Use cases of chatbots in a real-life scenario</strong></h3><p>Let&#39;s walk through sectors that have successfully applied chatbots to their business.</p><h3>Banking Sector</h3><p>The Financial sector has enjoyed more benefits from the adoption of chatbots than any other sector.</p><p>Banks can now place services like rewards notifications, promos, nearest bank locations at the tip of the customer&#39;s finger.</p><p>Chatbots can quickly provide insights on spending, market-specific news, and opportunities to use credit card rewards. At the same time, continually test and analyze the clients&#39; level of interest-based on the response.</p><p>Furthermore, bots can provide balance analysis and forecast, cost and budget management, notification of bill payments, sends reward notifications, and other tips. Additionally, this AI-driven financial advisor allows customers to define and manage how much they withdraw from an ATM and spend on shopping every day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zBCJCyzvbIm7z2ppELTBBQ.jpeg" /></figure><p>The United Bank of Africa Leo chatbot is an excellent example of a chatbot in the banking sector. Leo can help make quick transactions, block or freeze accounts in case of suspected fraudulent activities, print mini-statements, help locate the nearest branch, resolve customer complaints, and even weather forecasts.</p><p>Leo is also available on WhatsApp and Facebook messenger.</p><h3>Retail Sector</h3><p>One of the areas where chatbot shines the most is the retail sector. Here a bot can be used for various things but mainly for marketing, optimizing supports, and improving sales.</p><p>The chatbot will boost sales by suggesting products to customers, showing codes and giveaways, and allowing customers to check out easy checkouts.</p><p>Business owners can use chatbots to automate the business processes from taking customers&#39; orders to suggesting personalized products to notifying the store that the product is out of stock.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*TxZxJREXTS6NdX5SlF6AVQ.png" /></figure><p>You can find the Original Coast Clothing bot on Facebook messenger using the <a href="https://www.messenger.com/t/542998526103632">link</a>.</p><h3>Transportation Sector</h3><p>The Chatbots can quickly help customers book rides, rewards promos, report lost and found items. Report missing items quickly—fast access to customer support.</p><p>A chatbot can also provide step-by-step direction and give an estimate of the fare. With chatbots, users can get public transit directions on-demand, 24/7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*n49Bto69r6psDJ46SXswRA.png" /></figure><p>Lara is an Artificial Intelligence chatbot that provides directions via public transportation with step-by-step instructions, fare estimations, and alternative means of transport around Lagos.</p><p>Lara has been dubbed the google map for street transportation.</p><p>It gives the direction-finding best available route using Google Maps and also provides an estimate of the fare.</p><p>If you stay in Lagos, you can check out <a href="http://lara.ng/">Lara</a>. You know when AI will be of help 😉.</p><h3>Conclusion</h3><p>Chatbots have taken customer support to the next level with benefits such as easy access to customers, improved customer satisfaction, reduced cost, greater reach, and customer satisfaction.</p><p>Businesses that adopt chatbots have a competitive advantage over those that rely on traditional marketing and providing Support. Significantly In this Covid-19 era where communication speed is of great importance, chatbots can help companies retain customers and stay one step ahead of their competitors.</p><p>If you find this post interesting 😍, let&#39;s see how many times you can hit the clap button in a second.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e9ef29823d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NATURAL LANGUAGE PROCESSING (NLP) Basics with spaCy (Part 1)]]></title>
            <link>https://abdulquadri-oshoare.medium.com/natural-language-processing-nlp-basics-with-spacy-part-1-1355f3feb26a?source=rss-480dff2fffee------2</link>
            <guid isPermaLink="false">https://medium.com/p/1355f3feb26a</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[analytics-vidhya]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Abdulquadri Ayodeji Oshoare]]></dc:creator>
            <pubDate>Mon, 02 Mar 2020 22:00:53 GMT</pubDate>
            <atom:updated>2021-05-16T17:30:30.681Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*WQ9QAHZXYQQ4_zhxMnPW0w.png" /><figcaption><strong>© </strong><a href="https://www.learntek.org/blog/natural-language-processing-applications/">https://www.learntek.org/blog/natural-language-processing-applications</a></figcaption></figure><p>In this article you’ll learn about the basics of Natural Language Processing(NLP) such as Tokenization, Lemmatization, Stopwords, Part of speech (POS)tagging, and more, using the spaCy library.</p><p>Text data as we all know can be a rich source of information if it can be efficiently mined and processed.</p><p>According to Statistics from the link below 👇</p><p><a href="https://blog.microfocus.com/how-much-data-is-created-on-the-internet-each-day/">How Much Data is Created on the Internet Each Day?</a></p><blockquote>In the year 2019, 26 billion texts were sent each day by 27 million people in the US.</blockquote><p>You wouldn’t want to start a journey without knowing your destination? 😎let’s talk a bit about what NLP and its uses before move into the spaCy Library.</p><h3>What is NLP?</h3><p>NLP is an area of computer science and artificial intelligence, concerned with the interactions between computers and humans (natural languages), in particular how to program computers to process and analyze large amounts of natural language data.</p><blockquote>NLP can simply be said to be the ability of a computer to read and understand human languages.</blockquote><h3>Uses of NLP</h3><ul><li>Classifying emails as legitimate or illegitimate</li><li>Sentiment Analysis</li><li>Movie reviews</li><li>Understanding text commands e.g ‘hey google, play this song’</li></ul><p>Now that we know what NLP is, and what it is used for, let dive into the spaCY library.</p><h3>Why spaCY?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*mpb2p1dEklKJXwpaHZGfdQ.png" /></figure><p>spaCy is an open-source NLP library for python developed by explosion.ai in 2015. it was designed to effectively handle NLP tasks efficiently. spaCY is faster and much easier to use, however, it doesn’t include pre-created models for sentiment analysis.</p><p>The figure below shows the speed comparison of spaCY to other NLP libraries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*41Fc3WxEGzDqBF-xmD5tdA.jpeg" /><figcaption>figure 1. Speed comparison of spaCY to other NLP library</figcaption></figure><h3>Getting Started</h3><blockquote>pip install -U spacy</blockquote><blockquote>To install the English language model</blockquote><blockquote>python -m spacy download en_core_web_sm</blockquote><p>It should take a while, so you can grab a cup of coffee ☕️ or go do some other work e.t.c while the English model is installing. when it has been successfully installed, it would show “linking successful”.</p><p>let’s go ahead and import spaCy.</p><pre>#import spacy and load the english language model<br>import spacy<br>nlp = spacy.load(&quot;en_core_web_sm&quot;)</pre><p>Note: the English language model must be loaded alongside spaCy to allow spaCy to perform Language processing in English.</p><h3>Tokenization</h3><p>is the process of breaking down a text into components or pieces. these pieces are called tokens. when performing tokenization we should note the following: prefix, suffix, infix, and exceptions. we don’t have to worry too much about that because spaCy has a built-in tokenizer that does the job for you.</p><p>let’s get our hands dirty with some codes</p><p><strong>Code:</strong></p><pre>&quot;&quot;&quot;A doc is a sequence of tokens, a doc can be an article, a newspaper or a tweet<br>&#39;u&#39; is a unicode string, it is the industry standard for handling text.&quot;&quot;&quot;</pre><pre>doc = nlp(u&quot;NLP isn&#39;t the same as Neuro-linguistic programming.&quot;)</pre><pre>for token in doc:<br>    print(f&quot;{token.text}&quot;)</pre><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*TqtXn9OTuZwz2OIrWfw28A.png" /><figcaption>figure 2</figcaption></figure><p>As we can see, spaCy is smart enough to distinguish between the prefix, infix, and suffix.</p><h3>Stemming</h3><p>Stemming is the process of reducing words to their root forms (stem) such as mapping a group of words to the same stem even if the stem itself is not a valid word in the Language.</p><p>The most common algorithm for stemming English is Porter’s algorithm which was developed in 1980.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*1d-QL180FkkB53ty4dVbQA.png" /><figcaption>figure 3. rules used by porter’s stemmer <a href="https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html">copyright</a>.</figcaption></figure><p>Although spaCy doesn’t have a library for stemming, it uses a better approach called lemmatization to reduce words effectively to their root forms.</p><h3>Lemmatization</h3><p>is the method of reducing a word to its root or base form (lemma). unlike stemming which follows an algorithm to chop off the words, lemmatization takes it a step further by looking at the surrounding text to determine the given word part of speech.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*Z5H9xNknIpqAnwFX8tuicQ.png" /><figcaption>figure 4. lemmatization</figcaption></figure><p>the lemma of running is run, while the lemma of eating is eat. again spaCy does this under the hood, so you don’t need to worry.</p><p><strong>Note:</strong> the difference between “lemma” and “lemma_” in the figure above. “lemma_” returns the lemma while “lemma” returns the hash values of the lemma.</p><p>If you’re confused 😵and want to know more differences between lemmatization and stemming click <a href="https://www.datacamp.com/community/tutorials/stemming-lemmatization-python">here</a></p><h3>Part-of-Speech tagging</h3><p>part-of-speech tagging or POS tagging is the marking of a word in a text or document to a particular part of speech(noun, verb, adjective) based on its definition and context. for additional knowledge on POS, tagging click <a href="https://en.wikipedia.org/wiki/Part-of-speech_tagging">here</a>.</p><p>spaCy can parse and tag a given text or document. spaCy uses a statistical model, which enables it to make predictions of which tag or label most likely applies in this context.</p><p><strong>Code:</strong></p><pre>import spacy</pre><pre>nlp = spacy.load(&quot;en_core_web_sm&quot;)<br>doc2 = nlp(&quot;Apple is looking at buying U.K. startup for $1 billion&quot;)</pre><pre>for token in doc:<br>    print(f&quot;{token.text:10}  {token.pos_:10} {token.tag_:10} {spacy.explain(token.pos_):15} {spacy.explain(token.tag_)}&quot;)</pre><pre>#Note the use of underscore</pre><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*ElOvUi31trsX6azi-C22Eg.png" /><figcaption>figure 5. Parts of speech tagging</figcaption></figure><ul><li><strong>Text:</strong> The original word text.</li><li><strong>POS:</strong> The simple part-of-speech tag.</li><li><strong>Tag:</strong> The detailed part-of-speech tag.</li></ul><h3>Stop words</h3><p>commonly used words such as ‘I’, ‘you’, ‘anyone’, appear so often in a document and as such cannot be tagged as nouns, verbs or a modifier.</p><p>spaCy has a built-in list of over 300 English stop words that can be used to filter out unnecessary words in a document. spaCy also allows a user to add custom stopwords, not in the spaCy’s built-in list.</p><p><strong>Code:</strong></p><pre>#To get the list of spaCY built-in stop words <br>print(nlp.Defaults.stop_words)</pre><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*iMHTENGiMFF1UOmlxyxM4Q.png" /><figcaption>figure 6 Stopwords</figcaption></figure><p><strong>Note</strong>: caution should be taken when removing stop words from a document, as shown in the figure above. if the sentence itself is a stopword it would lose its meaning.</p><h3>Visualizing Parts-of-speech and Named Entities</h3><p>spaCy comes with a built-in dependency visualizer called displacy, which can be used to visualize the syntactic dependency (relationships) between tokens and the entities contained in a text.</p><p>displacy can be viewed either in Jupyter notebook</p><p><strong>Code:</strong></p><pre>from spacy import displacy</pre><pre>doc4 = &quot;The Bus arrives by noon&quot;</pre><pre>displacy.render(doc4,style=&#39;dep&#39;)</pre><h4>Output:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*mgjKY2LWCHOyhefXruqStw.png" /><figcaption>figure 7. Visualizing syntactic dependency using displacy</figcaption></figure><p>We can play around by changing the style argument from ‘dep’ to ‘ent’ to visualize Named Entity using displacy.</p><p><strong>Code:</strong></p><pre>doc5 = nlp(&quot;Almost four months after promising $1 million of his own money to veterans&#39; causes, Donald Trump moved to fufil that pledge Monday evening - promising the entire sum to a single charity as he came under intense media scrutiny.&quot;)</pre><pre>displacy.render(doc5,style=&#39;ent&#39;)</pre><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*cZGhP4A9Cc572KnvSYkmzA.png" /><figcaption>figure 8. Visualizing Named Entities using displacy</figcaption></figure><h3>Conclusion</h3><p>We have come to the end of the first part in the NLP basics using the spaCy library in the next part we would go over Named Entity Recognition(NER) and Sentence segmentation.</p><p>I hope you found this tutorial interesting. Please share and remember to comment with your suggestions or feedback.</p><p>Don’t forget to <strong>follow me</strong> for posts on Data Science and AI.</p><p>Cheers!!!</p><p><strong>Reference</strong></p><p>[1] Jose Portilla, <a href="https://www.udemy.com/course/nlp-natural-language-processing-with-python/">Natural Language Processing with Python</a></p><p>[2] https://<a href="https://spacy.io/">SpaCy.io</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1355f3feb26a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Absenteeism at the workplace, a categorical approach]]></title>
            <link>https://abdulquadri-oshoare.medium.com/predicting-absenteeism-at-the-workplace-a-categorical-approach-105dd5515f47?source=rss-480dff2fffee------2</link>
            <guid isPermaLink="false">https://medium.com/p/105dd5515f47</guid>
            <category><![CDATA[multiclass-classification]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[random-forest-classifiers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[Abdulquadri Ayodeji Oshoare]]></dc:creator>
            <pubDate>Mon, 26 Aug 2019 18:25:47 GMT</pubDate>
            <atom:updated>2019-08-26T19:44:10.153Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S7NCFnjpGIZa78U17ccgCg.png" /><figcaption><strong>© kaggle.com</strong></figcaption></figure><p>In this post, I am going to create a machine learning model on the popular Absenteeism at work dataset. The Absenteeism at work dataset has been used by many organizations seeking to find the main reason behind employees’ absence, to reduce expenses and increase productivity in order to meet customer satisfaction.</p><h3><strong>Dataset</strong></h3><p>The dataset used is the popular “<a href="https://archive.ics.uci.edu/ml/datasets/Absenteeism+at+work">Absenteeism at work</a>” dataset. the dataset was created with records of absenteeism at work from July 2007 to July 2010 at a courier company in Brazil. it has also been used in academic research at the Universidade Nove de Julho — Postgraduate Program in Informatics and Knowledge Management.</p><h3><strong>Getting the data</strong></h3><p>We start by importing the necessary libraries, pandas, numpy, matplotlib, and seaborn. we then use pandas read_csv() to read the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*pJI9V-ZwPqGsi5zv5iJppw.png" /></figure><h3><strong>Data exploration/analysis</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*fqmMHaDP0fjJ8Du01ald-Q.png" /></figure><p>The dataset has 740 examples and 20 features + the target variable(Absenteeism time in hours).</p><h4><strong>A bit of feature engineering</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*nQEsSkzyAcEr-MVcokCvBA.png" /><figcaption>feature engineering</figcaption></figure><p>We can get 8 more features from feature engineering which are Age_Category, smoke_cat, absenteeism category, Disciplinary cat, drink_cat, Education_cat, transportation_category and distance category.</p><h3><strong>Data Visualization</strong></h3><p>Now that we’ve created new features, let’s see how they affect workplace absence.</p><p>First lets see how employees age affect absenteeism .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*OVkkiIzf2Wut2A9t0fAQgQ.png" /></figure><p>We can see from the plot that young and old employee has the highest numbers of hours absent.</p><p>Next, we can see how employees’ educational background affects absenteeism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*1hYuK1C9xZFoJkQhskT3Vg.png" /></figure><p>As educational qualification increases, so does the hours of absenteeism decreases.</p><p>This might be because the employee with higher qualifications receives more pay and benefits including health insurance. Who knows?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*5XEWOEygbD6TbkIJX731gw.png" /></figure><p>Employees who smoke tend to have more hours of absence than those who don’t.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*iBCxP9qGPVFpFBZSe4dZOQ.png" /></figure><p>So far, disciplinary actions have the highest effect on hours of the absence. When disciplinary measures are taken employees tends to starts to be punctual.</p><h3><strong>Building the Machine Learning Models</strong></h3><p>We split our data into train data and test data in the ratio 70:30 taking our target to be the absenteeism category and choosing features that could be gotten from an employee during recruitment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*Z2-u5RcEFejnPYqoIGFfsg.png" /><figcaption>train test split</figcaption></figure><p>We&#39;ll train our data with the following machine learning algorithm:</p><ul><li>Naïve bayes</li><li>Random forest</li><li>Logistic regression</li><li>Support vector machine.</li></ul><p>Note: Check target column to see if it’s balanced before you continue to the next step</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*SYwq_FGTB6U5QpgVXxcScg.png" /><figcaption>countplot showing data imbalance</figcaption></figure><p>From the Countplot above we can see that our target column is imbalanced, If we train the model without fixing this problem, the model will be completely biased.</p><p>Therefore we would apply a technique called oversampling,. the most common oversampling technique is SMOTE (Synthetic Minority Over-sampling Technique). In simple terms, it looks at the feature space for the minority class data points and considers its k nearest neighbors. for more information on SMOTE click <a href="https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*Oi8YFuvr98g_4QvY91TTOw.png" /></figure><p>Okay! Looks like SMOTE has increased the samples of minority classes. lets move on with the training.</p><p><strong>Naive Bayes classifier</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*ZOZ3S9s1BMBlodyTdPz62g.png" /><figcaption>naive bayes classifier</figcaption></figure><p><strong>Logistic Regression model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*l11BnFvbaqh_C9n-6qW6rQ.png" /><figcaption>Logistic Regression classifier</figcaption></figure><p><strong>Support vector machine model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*AOCN2VZBp55j-4f-CGvubA.png" /><figcaption>svm</figcaption></figure><p><strong>Random Forest model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*QEV0B0RiEwQTTfBs5C3cKA.png" /><figcaption>Random forest classifier</figcaption></figure><p>As we can see, the Random Forest classifier goes on the first place, with an accuracy of 0.89</p><p>Random Forest is a supervised learning algorithm, it builds multiple decision trees and merges them together to get a more accurate and stable prediction. One big advantage of random forest is, that it can be used for both classification and regression problems</p><h3><strong>Conclusion</strong></h3><p>In Conclusion, it has shown that absenteeism can be avoided before it occurs, or the prediction values can be used as an effective tool for crisis management. Also, the hidden causes of absenteeism could be used by an organization to set additional requirement for a new job. therefore the percentage of absenteeism could be reduced.</p><p>This is the publication of the project assignment of AI Wednesday cohort, organized by Data Science Nigeria.</p><p>Special thanks to everyone who contributed in making the project successful <a href="https://medium.com/u/480dff2fffee">Abdul Quadri</a>, <a href="https://medium.com/u/25bbbd0c5cfc">Ojo Olawale</a>, <a href="https://medium.com/u/8ecfc0735e06">Ogheneovo Idoghor</a> <a href="https://medium.com/u/c502bef321d9">Otulagun Daniel Oluwatosin</a>, <a href="https://medium.com/u/55dfdedb6ea5">Haneefah Abdul-Rahman Lekki</a></p><p>A special thanks to Data Science Nigeria for creating the platform to learn Data Science and Artificial Intelligence by creating world-class learning.</p><p>And also to our Instructors <a href="https://medium.com/u/fa421a9ff93d">Daniel Ajisafe</a> and the entire AI Wednesday Team, we say a big thank you.</p><p>Click <a href="https://github.com/quadriano31/DSN-Project.git">here</a> for the link to the notebook. If you find this post interesting 😍, let’s see how many times you can hit the clap button in a second.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=105dd5515f47" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>