<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Otto Godswill Phronexus on Medium]]></title>
        <description><![CDATA[Stories by Otto Godswill Phronexus on Medium]]></description>
        <link>https://medium.com/@godswillchidubam?source=rss-1d7367176c9c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5teoQtO4aO9wdLt48-PnOg.jpeg</url>
            <title>Stories by Otto Godswill Phronexus on Medium</title>
            <link>https://medium.com/@godswillchidubam?source=rss-1d7367176c9c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:58:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@godswillchidubam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Election Result Data Pipeline: Detection of Outlier Score via Proximity]]></title>
            <link>https://medium.com/@godswillchidubam/election-result-data-pipeline-detection-of-outlier-score-via-proximity-f1ff83f66abf?source=rss-1d7367176c9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1ff83f66abf</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[geospatial-data]]></category>
            <dc:creator><![CDATA[Otto Godswill Phronexus]]></dc:creator>
            <pubDate>Thu, 04 Jul 2024 22:11:16 GMT</pubDate>
            <atom:updated>2024-08-23T18:49:33.570Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi Data Rangers, there is a new mission. void queen has sent her minions to cause havoc. The details of the mission are in the pamphlet below. Guides have also been provided.</p><p>Task</p><blockquote>Allegations of vote manipulation and irregularities in the nigerian 2023 elections have been widespread, prompting a thorough investigation into the matter.<br>Your mission, should you choose to accept it, is to help us uncover potential voting irregularities and ensure the transparency of the election results. You will achieve this by identifying outlier polling units where the voting results deviate significantly from neighbouring units, indicating potential influences or rigging.</blockquote><p>Tips</p><blockquote>You can use geocoding techniques such as Google Maps Geocoding API or OpenCage Geocoding API, and add the geospatial data to the dataset.<br> You can use geodesic distance to calculate the distance between each polling unit and all other units, and set a predefined radius (e.g., 1km).</blockquote><p>Further information</p><blockquote>A folder containing csv files for each state is available each file has the following columns : “ State LGA Ward PU-Code PU-Name Accredited_Voters Registered_Voters Results_Found Transcription_Count Result_Sheet_Stamped Result_Sheet_Corrected Result_Sheet_Invalid Result_Sheet_Unclear Result_Sheet_Unsigned APC LP PDP NNPP Results_File”. Information regarding the longitude and latitude is not available.</blockquote><p>Mission Accepted!</p><p>Welcome, for the above problem, I have developed a more “data engineering” centered approach.</p><p>First, a plan is needed to deal with the issue. Here is my workflow and how I attempted the problem.</p><h3>Workflow</h3><ul><li>Extract the data</li><li>Geocode the locations ( given by using the PU name, LGA, and State)</li><li>Transform the data ready for Mongodb</li><li>Load to a Mongo DB server</li><li>Pull the needed data for the analysis</li><li>Analyze</li><li>Report</li></ul><p>Group the workflow into two major Phases.</p><h3><strong>Phase 1: ETL</strong></h3><p>There are multiple flat files so a database is needed. Though, I have used only one of these files.</p><p>The database of choice is the Mongo DB so first, let&#39;s prepare the storage location.</p><p>I created a new database on the atlas cluster named election and created two users and roles for this task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QKEWGDGpRFkrbR0F1TqpTQ.png" /><figcaption>Roles</figcaption></figure><p>I have created one role for a user inserting the data into the database and another for the user working on the analysis. (—users can also be made using the command line).</p><p>Next, I got my URI connection string and saved it in an environment variable, editing it with each user&#39;s username and password.</p><p>To extract geolocation, I will use the geopy module.</p><p>next, I created a script to load data from CSV and extract it.</p><ul><li>Import necessary libraries</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*e7yKvDoU2kYjNUCkEjDW_w.png" /><figcaption>Import Modules</figcaption></figure><p>Read in the data and create the address field using the state, lga, ward and PU name together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*oiNOsE4OnLMdntZ1z-coLg.png" /><figcaption>Get addresses</figcaption></figure><p>Using ArcGIS, create a function and apply it to the address column. The function is used to extract the longitude and latitude information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*OdZ6PdigKYOCixbDVpkh2A.png" /><figcaption>geopy geocode</figcaption></figure><p>The function took a while roughly 30 minutes so I felt it was best to save the pandas file (it was not necessary)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*D4mkfVYNIO_HzLOe0IGd_Q.png" /><figcaption>Write to feather</figcaption></figure><p>Next, I created a function to store the lat and longitude in MongoDB recognizable formats for geographical data, this is so I can optimize it with indexes when saved on MongoDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*9h-J36mNkUQ_goKKNGtF0g.png" /><figcaption>Change to Geographical object mongodb</figcaption></figure><p>the data was now ready to be uploaded.</p><p>Initiated MongoDB client</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*52o1fzIEzhScOx8W-x65cw.png" /></figure><p>Filtered the data to remove those without coordinates data — ( upload those with coordinates data with the field “location” and those without coordinates data without the field “location”.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*eLNiZzJWsFQPo38i49BzgQ.png" /><figcaption>With location field</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*l3tt4vhh7hApOAWDPnvECA.png" /><figcaption>Without location field</figcaption></figure><p>there were no null values, so all fields were inserted with location data.</p><p>create the index on the location field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*G416Pl0UK_-dVKn8ELDvQA.png" /></figure><p>Then close the connection.</p><blockquote>client.close()</blockquote><h3>Phase 2: Analysis</h3><p>The aim is to get those polling units in a particular state (RIVERS) where the number of voters for each party ( of the 4 major parties available in the dataset) significantly differs from its neighbors within specified radius.</p><p>To do this, I created a 3 stage pipeline.</p><ol><li>filter stage: To retrieve data from the state alone that has location field. I used the match stage of the agregation pipeline.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*ESIhdkU3hnqFS3BettXRLQ.png" /><figcaption>stage 1 query</figcaption></figure><p>2. Extract Neighbours data: I used a self-join (left outer join on self) to get all the neighbors within a specified range (1km). More precisely in MongoDB, I used the lookup stage and then an inner pipeline where I used the geoNear stage to find for each pooling unit, all pooling units a specified distance around the specific pooling unit excluding the current pooling unit ( excluding the pooling is important so if it is an outlier it does not affect the mean and std values of the others) then project required value instead of the whole document being joined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*vsZwH4AJ1Rp6D0YFEzm4xQ.png" /><figcaption>stage 2 query</figcaption></figure><p>3. Return required values: I used a projection stage to return the needed values such as mean, standard deviation and counts. mean and standard deviation of voters in polling units surrounding each polling unit is needed to calculate the z score of each pooling unit. Generally z-scores above 3(or lesser than -3) can be considered outliers. This threshold can be adjusted. The count of how many neighbours is also needed for clarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*pCYg1w0h7jPKlTLNAtbEFw.png" /><figcaption>Stage 3: Projection Stage</figcaption></figure><p>I exported the query to Python language and copied it and continued my analysis with Python.</p><ul><li>Import necessary libraries, load the Mongo client, and run the copied query.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*rc92S0bQ59uN4D_Dhp8pSw.png" /></figure><ul><li>Load the cursor returned to a dataframe</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*abxF5LiahTF-eo4q-b9txQ.png" /></figure><ul><li>Close the connection.</li></ul><blockquote>client.close()</blockquote><ul><li>Calculate the Z score of each party</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*hgJWEZOCqPdc9oO_1vBitA.png" /><figcaption>create Z — score</figcaption></figure><ul><li>I created threshold values for the Z-score (above which is an outlier), and count ( below which we can&#39;t say it is an outlier ) and defined a color scale that will later be used for plotting.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*vPE_owJrLO39vyj3y3M31w.png" /></figure><ul><li>I then calculated Top 5 outliers for each party based on the Z_score.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*DwiPyRygGsPo8RIoTGJw_g.png" /></figure><ul><li>I Recorded important parameters like the mean score and count of the outliers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*ENNUGJxc0g7RjohWdTaYmQ.png" /></figure><ul><li>I made a geographical plot of the locations, included the land and water bodies just because they might help give insights as to the distribution especially in a place like Rivers State.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*pFI-U3E20gSW1oOWkvwM9w.png" /><figcaption>Code for geographical plot</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*4hztKsI8bTsDINKapFDBkA.png" /></figure><ul><li>I Repeated the process for the other parties.</li><li>Compare across parties.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*rUrHsH1MepuPC4zlEq56sg.png" /></figure><ul><li>The average values as well</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*rVHdqfjDwfu-is7L_jTDEw.png" /></figure><ul><li>A detailed report of findings is available on my github <a href="https://github.com/wayne-chi/Election_ETL_Pipeline_1/blob/main/Election_presentation.pdf">Here</a>.</li></ul><p>My Findings</p><ul><li>There were significant deviations in around 40 polling units per party with NNPP having the largest numbers of outliers.</li></ul><p>Mission Completed!</p><p>Here is a <a href="https://github.com/wayne-chi/Election_ETL_Pipeline_1/tree/main">link to github</a> project for this work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1ff83f66abf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating User Creation with Bash Scripting — A Basic DevOps Project]]></title>
            <link>https://medium.com/@godswillchidubam/automating-user-creation-with-bash-scripting-a-basic-devops-project-2dc881d13c9f?source=rss-1d7367176c9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dc881d13c9f</guid>
            <category><![CDATA[devops-practice]]></category>
            <category><![CDATA[sysops]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Otto Godswill Phronexus]]></dc:creator>
            <pubDate>Wed, 03 Jul 2024 00:29:18 GMT</pubDate>
            <atom:updated>2024-07-03T00:29:18.995Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automating User Creation with Bash Scripting — A Basic DevOps Project</h3><p>I sought out beginner-friendly projects to improve my skills in systems administration. I got this task from <a href="https://hng.tech/internship">HNG</a>.</p><p>It is Project Time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*10osOScnccKVENaY.jpg" /><figcaption>Gotten from Dreamstime.com</figcaption></figure><p>This post explains how to do this task. I will also explain possible hurdles and how to go about resolving it.</p><h3>TASK</h3><blockquote>Your company has employed many new developers. As a SysOps engineer, write a bash script called create_users.sh that reads a text file containing the employee’s usernames and group names, where each line is formatted as user;groups.</blockquote><blockquote>The script should create users and groups as specified, set up home directories with appropriate permissions and ownership, generate random passwords for the users, and log all actions to /var/log/user_management.log. Additionally, store the generated passwords securely in /var/secure/user_passwords.txt.</blockquote><blockquote>Ensure error handling for scenarios like existing users and provide clear documentation and comments within the script.</blockquote><p>Extra Requirement</p><blockquote>Each User must have a personal group with the same group name as the username, this group name will not be written in the text file.<br> A user can have multiple groups, each group delimited by comma “,”<br> Usernames and user groups are separated by semicolon “;”- Ignore whitespace<br>e.g.<br> light; sudo,dev,www-data<br> idimma; sudo<br> mayowa; dev,www-data<br>For the first line, light is username and groups are sudo, dev, www-data</blockquote><p>Before we continue to solve this, here are some system requirements.</p><p>you would need to work with a Linus/Mac OS, or any other OS built on the UNIX system. If you are a Windows user, consider using containers (like docker) I have also provided links to some PaaS that are easy to setup.</p><h3>Solution</h3><p>Given the problem the first thing to do is to set up test files. In this case this would mean both a sample input file and a bash file to test if your main script works.</p><h4>Test Files</h4><p>create a test.txt file in your work environment and input the test conditions. This would be used for the final test</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*63r4Z092Qdc4eCit4hOpNg.png" /><figcaption>test file</figcaption></figure><p>Also create a user.txt file which you could use to test along as you write the scripts for the bash file. Just think of it as an alternative testing file.</p><p>for me, I build scripts this way and start by editing this other test file to meet the requirements one after the other till I have checked all my boxes then final test will be done with the test.txt file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*l58AZeklFYwg2PzpscEMvQ.png" /><figcaption>user.txt</figcaption></figure><p>The next thing to do is to build a file that checks or assist to check if our code is working (the test file).</p><h4>Building the test file</h4><p>As with bash scripts start with the hash bang (#!) 😊 (lol always excites me haha)</p><blockquote>#!/bin/bash</blockquote><p>the /bin/bash is the path and depends on the shell you are using . if you don&#39;t know that you can type “which bash” and then use the address returned. (well, if you have been using linux you probably already know that, but if you are doing this from a virtual machine / container for the first time, you might want to check that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*iA7brzoq22qwmh2frngTbg.png" /><figcaption>which bash command.</figcaption></figure><p>what we want to achieve with this test, is to quickly check if groups and users are inserted successfully. To do that two commands would be of great use</p><blockquote>“getent group” alternatively “cat /etc/group “to see group</blockquote><blockquote>“getent passwd” to see users</blockquote><p>in the context of getent passwd, applications are also referred to as users, to filter these out select those within the login UID range for human users.</p><p>this can be found by running the following two commands</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*SOu90ihTnD3rbHg73cpUXg.png" /><figcaption>UID_MIN/UID_MAX</figcaption></figure><p>I have added the below lines of code to make testing faster by taking in options. I have options v for verbosity, u to search if a certain user has been added -g to search for a certain group -l to display the log file information -p to display the passwords</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*kZiIvPugC_UHAJAVOz8Vjw.png" /><figcaption>optional arguments</figcaption></figure><p>these are not necessary but i like things with options so ya, and could help a lot with debugging and after testing.</p><p>so for verbosity</p><p>the getent psswd is used to get all users (human users and application) then it is filtered by those in the UID range for humans (1000 to 60000)</p><p>cat is used to read a file and its content</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*gnIj7uk0YRY74pfocqa7-Q.png" /><figcaption>verbosity</figcaption></figure><p>Then in similar way other options are set</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*7ozEHB5aqKXoZfsfa9Cmdw.png" /><figcaption>Other optional argument.</figcaption></figure><p>Grep is use to search for a phrase/text within a text/file.</p><p>Now the test file is ready, the main script can be written</p><h4>Main Script</h4><p>The first step here is to initialize and ensure that a file is passed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*U4pE6Tu8yP6HzFYdLhAEQQ.png" /><figcaption>step 1</figcaption></figure><p>the $1 refers to the first argument. -z option used to check if an argument was passed. (something like “” check if it is an empty string)</p><p>next, loop through each line of the input document using the while loop ( do .. while)</p><p>for each loop read the line of the input file, set the IFS (internal file separator) to “;” to demarcate users and groups, and always check if the username has been (first value before separator) do this should the file not end with a newline character. save the values to variables username, groups</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*pG_REz7gP7vXj_UWkvbs5A.png" /><figcaption>looping and reading each line</figcaption></figure><blockquote>username=$(echo “$username” | tr -s ‘[:space:]’ | xargs)</blockquote><blockquote>groups=$(echo “$groups” | tr -s ‘[:space:]’| xargs)</blockquote><p>for each variable remove trailing whitespace using the tr -s command and execute the command using the xargs.</p><p>check if the user already exist using the id function , if it does exist, skip this user and log the information, we wont want to have conflicting usernames and necessarily do not update the old user with the new users privileges, so log the information and work on it after the bulk add operation. If the user doesn&#39;t exist go ahead and create the user. Also useradd also creates a group &amp; folder for each user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*FCyk6BaDGcGRdrlllWjXmA.png" /><figcaption>Creating user</figcaption></figure><p>the group information gotten is now a string of values separated by comma and optionally spaces, using the internal field separator separate the values into an array, and for specific values using the tr and xargs like before clean trailing spaces for each value. create the group if it doesn&#39;t exist.</p><p>modify the user and add them to each group in the array.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*0klXiuTxjQRhmTLuMPGGKQ.png" /><figcaption>Create group and assign user</figcaption></figure><p>generate random password with the openssl command and save in password variable</p><blockquote># Generate a random password</blockquote><blockquote>password=$(openssl rand -base64 12)</blockquote><p>pass the username and password to the chpassswd command to set the password and log with a success information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*k-HoZm-YatDkHDIUQg_wxQ.png" /><figcaption>generate and save password</figcaption></figure><p>save the script as required create_user.sh</p><p>Then from the command line you can run the command</p><blockquote>sudo bash create_user.sh user.txt</blockquote><p>you have to run with su privilege to be able to create a user and access some of the directory.</p><p>if you run that for the first time you might get an error like this one</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*wljg_l0PI5-C7u3KnV_m_g.png" /></figure><p>what you should do is to go and create the needed files.</p><p>you can cd into the /var and create the directories if any doesn&#39;t exist then in each of the directories create the needed files</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*Bfis-aG57DDC6A-dodQnnA.png" /><figcaption>debug</figcaption></figure><p>when you are done creating the files you can cd back to your project folder ( where your scripts are) and run the command again</p><blockquote>sudo bash create_user.sh user.txt</blockquote><p>the code finished running without any messages, you can add an echo “success message” to indicate when it is done.</p><p>use the bash script to check if it successfully created rooney as a user</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*PqBQQZp-SHotolwLTLceCA.png" /><figcaption>test 1</figcaption></figure><p>test setting the verbosity to 0 and searching for only user ‘rooney’</p><p>next text for the groups using any sample group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*dl9e4vrq0YZL8KwDkfCx0g.png" /></figure><p>Finally, the moment of truth, test against the input text cases.</p><p>check for user sudo (not existing) and group ( dev)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*n1bxcZhFJXgx43vf3vFIaQ.png" /><figcaption>Result -1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*RhW91hi4haQG9w1uQKCyCg.png" /></figure><p>So Hooray !!! we have successfully automated a batch creation of user/role.</p><h3>Possible Hurdles</h3><p>I have tried to simplify so there aren’t any challenges with the work however for Windows users, if you encounter challenge below are some platforms that provide PaaS for you to practice on .</p><ul><li>Gitpod — I optimized it for gitpod <a href="https://github.com/wayne-chi/sample_gitpod_bashide">here</a></li><li>GitHub — Here is the <a href="https://github.com/wayne-chi/sample_gitpod_bashide">link </a>to the github portfolio</li><li>theia —</li><li>Amazon EC2, GCP, Azure Cloud Computing etc.</li></ul><p>If you are looking for projects to build your skills I would recommend <a href="https://hng.tech/internship">HNG Internship</a>. If you are looking to connect with top professionals in your tech discipline I recommend <a href="https://hng.tech/premium">HNG_Premium</a>.</p><p>If you are an employer or looking for someone to solve your tech needs then you can Hire a tech professional <a href="https://hng.tech/hire">Here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dc881d13c9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MATLAB Signal Processing 101: Tackling Echoes in Practical Applications (A shallow Dive)]]></title>
            <link>https://medium.com/@godswillchidubam/matlab-signal-processing-101-tackling-echoes-in-practical-applications-a-shallow-dive-30ab93d940ee?source=rss-1d7367176c9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/30ab93d940ee</guid>
            <category><![CDATA[matlab]]></category>
            <category><![CDATA[assignment-help]]></category>
            <category><![CDATA[signal-processing]]></category>
            <category><![CDATA[matlab-assignment-help]]></category>
            <category><![CDATA[assignment-writing-help]]></category>
            <dc:creator><![CDATA[Otto Godswill Phronexus]]></dc:creator>
            <pubDate>Sat, 10 Feb 2024 10:29:10 GMT</pubDate>
            <atom:updated>2024-02-10T10:29:10.524Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><p>Welcome back, fellow signal-processing enthusiasts! For our next adventure, we will be rescuing our favorite song that was attacked by an echo. Before we tune into the next adventure, let’s harmonize with a quick recap of our journey so far. In our previous tutorial, we orchestrated MATLAB’s symphony, mastering the basics and unlocking the power of plots. With MATLAB as our trusted conductor, we embarked on a melodic quest to conquer signal processing challenges—a quick recap.</p><p>Recap:</p><p>We’ve danced with codes, navigated through the interfaces, and harnessed the magic of MATLAB to visualize data like never before. But our signal journey is far from over — it’s just warming up. Now, armed with newfound skills, it’s time to hit the right notes and confidently tackle real-world problems.</p><p>Next Steps:</p><p>But before we strike the next chord, let’s take a brief interlude. In this tutorial, we’ll take a shallow dive into a practical application: solving the echo problem. Imagine fine-tuning your favorite tunes, removing unwanted echoes to create a seamless melody. Stay tuned as we continue to compose our journey through the signal processing landscape, one note at a time.</p><p>Here lies our rhythmic roadmap:</p><p>- A discuss on echo</p><p>- Understanding echoes</p><p>- The mathematics behind it</p><p>- Retrieving the song</p><p>- Matlab Magic</p><p>- Result</p><p>- Conclusion</p><p>So as my tutor would say put on your “thinking cap” and let’s have a joyous ride!</p><h3><strong>A discuss on echo</strong></h3><p>Echoes, those mesmerizing reverberations of sound, arise when waves bounce off surface(s) and return to the observer. The phenomena of echoes, those enchanting whispers of the past rebounding through space and time, have captivated humanity for centuries. Orchestrated by nature herself, these audible reflections bound off our world’s perimeters spin tales about resonance as well as reflection. This phenomenon, both enchanting and mysterious, forms the essence of our auditory experience, inviting us to unravel its secrets and embrace it.</p><p>Echoes have been used for various things, such as obstacle detection, navigation and sonar systems, medical imaging etc. But specifically, we would be talking about the noisy interference of an echo when listening to an orchestra in an empty musical theater, or watching a favorite movie in an empty movie cinema.</p><h4><strong>Understanding echoes</strong></h4><p>Echoes emerge from the intricate choreography of sound waves as they traverse through space. When a sound is emitted, it dances across surface(s), each encounter birthing a reflection that echoes back to us, carrying the essence of the original melody.</p><p>Note, that the use of the word “echo” would explicitly mean sound echo.</p><p>When sound is produced or released it bounces against surfaces creating an echo that comes back towards itself holding on to the character of the initial rhythm. This delayed duplicate response could create havoc on the initial music/rhythm.</p><p>For simplicity in this post, we would consider a single bounce of sound waves on a surface!</p><p>It is noted that for humans to hear an echo it has to delay for at least 0.1 second.</p><h3><strong>The mathematics behind it</strong></h3><p>To translate the mystical allure of echoes into the language of mathematics, we dive into the realm of discrete signals. Let x[n] symbolize the pristine sound signal. The echoed sound w[n] can be mathematically articulated as</p><p>w[n] = α x[n-D]</p><p>Here, D denotes the delay between the inception of sound and the arrival of its echoed counterpart, α represent the ratio of the signal that returned.</p><p>However, the overall melody that is being recorded by the observer is y[n]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*VDrCu6afiWWsL57Y0CJD2w.png" /><figcaption>Equation for echo</figcaption></figure><p>The figure below shows the discrete signal block diagram of this model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*7Fr2dY5gMfQO8WybPivB6w.png" /><figcaption>The discrete signal block diagram</figcaption></figure><h3><strong>Reverse it!</strong></h3><p>To clean echo from a music or recorded sound you will traditionally start with the echoed sound, y[n], and try to get the non-echo sound x[n].Let&#39;s get the transfer function to create the filter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*BaLXcw7Y0Q-W3TbjCBvv1g.png" /></figure><p>Get the transfer function to create the filter</p><figure><img alt="Transfer function of the echo" src="https://cdn-images-1.medium.com/max/379/1*R2qtJSZMPziioqXrkVQS3w.png" /></figure><p>So, let’s reverse the equation as what we have now is y[n]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*4IQ_v4AR3-wpVq-n79cURQ.png" /></figure><p>Design the filter with these parameters.</p><h3><strong>Retrieving the song!</strong></h3><p>I have provided an echo song “Pentatonix 12 days of Christmas”, a cut version with an echo you can download it from my github <a href="https://github.com/wayne-chi/Clean_echo/blob/main/Echo_Pentatonix_12_daysof_christmas_cut.wav">here </a>( <a href="https://github.com/wayne-chi/Clean_echo/blob/main/Echo_Pentatonix_12_daysof_christmas_cut.wav">https://github.com/wayne-chi/Clean_echo/blob/main/Echo_Pentatonix_12_daysof_christmas_cut.wav</a> ). Listen and notice that the echo returns at about 0.5 seconds after the source sound. This is a disaster 😂🤦🏽‍♂️.</p><h3><strong>MATLAB Magic</strong></h3><p>To effectively create a filter, we would have to figure out the values of alpha and D. We have already established that D is about 0.5 seconds. We can attempt to get alpha by comparing the echo sound with the original. alpha is usually a value with magnitude between 0 and 1. Play around with this value and see what you come up with we would be using a value of 0.85. You can experiment and see what different alpha values give you.</p><p>MATLAB has a filter function and it takes the numerator and denominator vector as input parameters.</p><p>The first step is to read the file, MATLAB has the “<em>audioread</em>” function, it reads an audio file with certain formats WAV, FLAC, and OGG. It doesn&#39;t read the mp3 format, it could if you combine it with certain 3rd party packages. But let&#39;s keep it single and move on. (the downloaded file is in .wav format)</p><p>Code:</p><pre>[echo_sound, Fs] = audioread(&#39;Echo_Pentatonix_12_daysof_christmas_cut.wav&#39;);</pre><p>It returns a list of two outputs, the signal and the sample frequency</p><p>Next, set the delay time and alpha value</p><pre>% Set the delay, By listening I have assumed/estimated the echo %delay to be around .5 seconds  you may vary in 0.1 seconds ... <br>delay_time = 0.5;<br>D = round(delay_time * Fs); <br>% estimate the  alpha value :: what fraction of the value of the %signal are you hearing <br>alpha = 0.85;</pre><p>Next, build the coefficients of the transfer function.</p><p>Note that these vectors start at coefficients of ｚ^０to ｚ^(-m) the most negative z power term. In this case, there are only two terms, Ｚ^０ ａｎｄ Ｚ^(-Ｄ) and every other coefficient is 0. So create that vector.</p><p>Code</p><pre>a= [1,zeros(1,D),alpha];</pre><p>Next, Use the MATLAB filter function</p><pre>filtered_sound = filter(1,a,echo_sound);</pre><h3><strong>Result</strong></h3><p>Use the MATLAB function to compare the sounds produced then appreciate your new skill.</p><p>Code</p><pre>sound(echo_sound,Fs);<br>pause(25);<br><br>sound(filtered_sound,Fs);<br>pause(25);</pre><p>The sound function plays the sound and the pause function tells MATLAB to hold on for some seconds (25) before it continues evaluation. This is to give the sound enough time to play the music completely.</p><p>Save the filtered sound using the code below:</p><pre>% save to output file<br>audiowrite(&#39;filter_Pentatonix_12_daysof_christmas_cut.wav&#39;,filtered_sound,Fs);</pre><h3><strong>Conclusion</strong></h3><p>In conclusion, we’ve embarked on a journey through the world of echoes, exploring how these reverberations are formed, and the methods to filter them out. Through our discussion, we’ve gained insights into the discrete equations that govern echo creation and the corresponding filter design to mitigate their effects and harnessed MATLAB’s filter design capabilities to counter their impact.</p><p><strong>What’s Next</strong>:</p><p>Moving forward, you may want to delve deeper into the world of digital signal processing, exploring advanced filtering techniques such as windowing methods (e.g., Hamming, Hanning, Blackman), adaptive filtering, and spectral analysis. These techniques offer sophisticated approaches to signal processing, allowing for finer control and enhanced performance in various applications.</p><p>As you continue your journey, remember that the field of signal processing is vast and dynamic, offering endless opportunities for learning and innovation. Stay curious, keep exploring, and let your passion for discovery drive you towards new horizons in the fascinating world of digital signal processing.</p><p>Here is the <a href="https://github.com/wayne-chi/Clean_echo/tree/main">link </a>to the GitHub repo for this code</p><p>If you have any questions or have any specific tutorial you want me to make, reach out to me on my Facebook page <a href="https://web.facebook.com/profile.php?id=61555879295314">Phronexus Freelance Services</a>, or leave a comment below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30ab93d940ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MATLAB Signal Processing 101: Your First Filter Design Adventure]]></title>
            <link>https://medium.com/@godswillchidubam/matlab-signal-processing-101-your-first-filter-design-adventure-b7e22d8eb95c?source=rss-1d7367176c9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7e22d8eb95c</guid>
            <category><![CDATA[signal-processing]]></category>
            <category><![CDATA[signal]]></category>
            <category><![CDATA[matlab-assignment-help]]></category>
            <category><![CDATA[matlab]]></category>
            <category><![CDATA[matlabsolution]]></category>
            <dc:creator><![CDATA[Otto Godswill Phronexus]]></dc:creator>
            <pubDate>Sun, 28 Jan 2024 13:50:41 GMT</pubDate>
            <atom:updated>2024-01-28T13:50:41.247Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>Hey, aspiring engineers and signal-processing enthusiasts! 🚀 Ever wondered what it feels like to bring those textbook theories to life? Well, get ready to embark on your very first filter design journey using MATLAB. No, we’re not diving into advanced masterclasses just yet; this is all about getting your hands dirty for the first time and turning those theoretical concepts into tangible, real-world designs.</p><p>Picture this: you’ve soaked in the classroom wisdom, nodded along to the theories, and now, it’s your chance to put that knowledge into action. I’m here to be your guide as we navigate the maze of filter design, making the seemingly complex world of MATLAB feel like a playground rather than a puzzle.</p><p>So, if you’re ready to break the seal on your MATLAB adventure, if you want to transform those theoretical lessons into practical skills, then grab your coding spirit, unleash your inner engineer, put on your thinking cap and let’s dive into the world of hands-on filter design. It’s not about mastering everything or solving a real world challenge just yet! Chill! it’s about giving you that first taste of practical magic. Let’s make MATLAB filter design your playground! 💻🔧✨</p><p>So lets get filtering. 😂 Here is the walkthrough.</p><p>· Welcome to MATLAB</p><p>· Meet the Playground — MATLAB Interface</p><p>· Saying Hello with Code</p><p>· Data Generation Magic</p><p>· Adding a Dash of Noise</p><p>· Let’s Mix Things Up — Combining Data</p><p>· Visualize the Symphony — Data Plotting</p><p>· Enter the Filter Designer’s Lair</p><p>· Filtering Out the Noise</p><p>· Admire the clean data and your new skillset</p><p>· The grand finale and beyond</p><p>· See you next time!</p><p>At this moment if you want to code along you should get your laptop ready and a comfortable seat! Otherwise, just have fun learning.</p><h3><strong>Welcome to MATLAB</strong></h3><p>MATLAB (Matrix Laboratory) is the trusted ally of engineers and scientists. It is a versatile tool that seamlessly integrates theory and practice, empowering you to bridge concepts with real-world applications. It has an intuitive interface and unmatched prowess in handling matrices and vectors. Its extensive array of functions for signal processing makes it a standout choice. From data generation to noise manipulation, MATLAB ensures this journey into signal processing is smooth, efficient, and, most importantly, user-friendly!</p><p>To install MATLAB, you can follow this <a href="https://www.youtube.com/watch?v=lH424VXzHxo">video link</a> or read this <a href="https://servicedesk.msstate.edu/TDClient/45/Portal/KB/ArticleDet?ID=1985">article</a>.</p><p>However, the good thing is if you are a student in an engineering school and you require the course in your study, your school probably already provided MATLAB access via your school’s portal. Also, check out for that!</p><h3><strong>Meet the Playground — MATLAB Interface</strong></h3><p>The Matlab environment looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*VlSiAjtt4LsH-w7SbZMcxw.png" /><figcaption>Matlab interface</figcaption></figure><p>Use this button below to navigate to the directory you want to work on. I suggest you create that directory beforehand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*kYcW5T3ZVcbAjxttHE1Uog.png" /><figcaption>Navigate to work folder</figcaption></figure><p>In case your layout is not exactly like mine or you prefer it some other way use the layout button to choose the one you prefer. You want to have your command window and your workspace showing as well. Optionally it is occasionally good to have your directory showing so it is easy to see files you want to work with. Click layout then select default or whichever you like then tick the current folder and workspace option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*kG6fNZn3RsyBnj3sjpUGLg.png" /><figcaption>Layout Button</figcaption></figure><p>You can quickly experiment with the command window but when writing codes, you want to use the script. So, click new script let&#39;s dive into saying hello world.</p><h3><strong>Saying Hello with Code</strong></h3><p>Now, let the fun Begin. Dive into the coding sea with our “Hello, World!” program. Fear not if you’re not a coding maestro yet — we’re keeping it simple</p><p>The code for hello world is below</p><pre>Disp(“Hello World, I just started by signal processing journey!”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*aDtIIlbi4Hdy_xqrz4Q5nw.png" /><figcaption>Hello world!</figcaption></figure><p>Hopefully it worked for you!</p><p>Another example</p><pre>t = [0: 1/100 : 1];<br>disp(t);</pre><p>Here, we have created an array from 0 to 1 with steps of 1/100 (0.01) and saved to a variable t.</p><p>In signals you can think of it as time variable sampled at 100Hz and each time stamp (n) is increased by 0.01. Note the “;” is there to stop the varaibles from spilling or litering the command window, use it frequently unless you want to see the output of that line of code.</p><p>Easy peasy ya?</p><h3><strong>Data Generation Magic</strong></h3><p>Now that you’ve exchanged greetings, let’s conjure some magic — data magic, to be exact. We’re about to unravel the basics of generating clean data. Picture conducting a symphony, but this time, your orchestra is comprised of functions and numbers. Our chosen instrument? The unassuming sine wave. Simple, elegant, and the backbone of signal processing.</p><p>Let’s generate a simple wave with a given frequency of say 900Hz. Now, for the sake of filtering we would join this wave with another of higher frequency say 1200 Hz. To avoid distortion, theoretically have your sample frequency &gt;= 2 times the max frequency so put sample frequency at 3000 Hz.</p><p>Let’s use the sine function with an amplitude of 4.</p><p>y = 4 sin2 π ft</p><p>Code:</p><pre>% sample frequency 3000 Hz, this means a max freq of 1500Hz can be used<br>% without aliasing distortion<br>Fs = 3000;<br><br>% Sample time<br>T = 1/Fs;<br>f1 = 900;<br>% generate data <br>t = [0:T:1-T];<br>sig = 4 * sin(2*pi *f1 *t);</pre><p>Explanation of the above code</p><p>A sample frequency is defined as 3000 Hz, then the sample time is calculated. The data on the sample time is calculated and the function is inputted.</p><h3><strong>Adding a Dash of Noise</strong></h3><p>Now, let’s add some noise to the data.</p><p>Let’s create another wave of a different frequency (1200 Hz) which we will filter out in this tutorial.</p><pre>f2 = 1200;<br>noise = 12 *sin(2*pi* f2 *t) +  0.2 * randn(size(t));</pre><h3><strong>Let’s Mix Things Up — Combining Data</strong></h3><p>Now, let’s play with our data. Mixing clean and noisy data gives us a taste of real-world complexity. Here’s how you can blend these signals</p><pre>dirtyData = cleanData + noise;</pre><h3><strong>Visualize the Symphony — Data Plotting</strong></h3><p>Now, we want to dive into the world of plotting in MATLAB. Easy peasy! First off, fire up the plot using the figure command and unleash the plot function for some visual magic. Take a look at this snippet:</p><pre>plot(t, cleanData);<br>xlabel(&#39;Time &#39;);<br>ylabel(&#39;Amplitude&#39;);<br>title(&#39;Time Domain plot of clean signal&#39;);<br><br>figure;<br>plot(t, dirtyData);<br>xlabel(&#39;Time &#39;);<br>ylabel(&#39;Amplitude&#39;);<br>title(&#39;Time Domain plot of dirty signal&#39;);</pre><p>Boom! You’ve got yourself some sleek time domain plots.</p><figure><img alt="Time Domain Plot for the clean data" src="https://cdn-images-1.medium.com/max/854/1*PU6cFSotUhn_0ix8Q2njxw.png" /><figcaption>TIme Domain Plot — Clean Data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*a0jSvyBDiPcTArIzbS1_GQ.png" /><figcaption>Time Domain Plot — Dirty Signals</figcaption></figure><p>But wait, there’s more!</p><p>Ever thought about seeing your signal in the frequency domain? MATLAB’s got your back with the Fourier transform.</p><p>To convert the signal to the frequency domain use the fourier transform. Matlab provides as alternative to DFT, a matlab function fft (fast fourrier transform). Then use the absolute function to get the absolute value of the magnitude</p><p>Check out this snippet:</p><pre>dirtyDataFd = abs(fft(dirtyData));<br>cleanDataFd = abs(fft(cleanData));<br><br>figure;<br>plot(dirtyDataFd);<br>xlabel(&#39;Frequency (Hz)&#39;);<br>ylabel(&#39;Magnitude&#39;);<br>title(&#39;Magnitude Spectrum of dirty Signal&#39;);<br><br>figure;<br>plot(cleanDataFd);<br>xlabel(&#39;Frequency (Hz)&#39;);<br>ylabel(&#39;Magnitude&#39;);<br>title(&#39;Magnitude Spectrum of clean Signal&#39;);</pre><p>Here is the output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*IoUwopH_WYK9xmqklMvKXg.png" /><figcaption>Magnitude Spectrum of Dirty signal</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*oA9dsM_ZRklry-iICwBRbA.png" /><figcaption>Magnitude Spectrum Of Clean data</figcaption></figure><p>Voilà! You just unleashed the power of magnitude frequency plots. See the difference? The clean signal has one frequency chillin’, while the dirty one’s got a duo. Now, our mission? Filter out that interfering frequency. Ready for the next adventure?</p><h3><strong>Enter the Filter Designer’s Lair</strong></h3><p>To conquer the higher frequency you need a .. (you probably already guessed it!) ya a Low Pass Filter. Writing that I feel like Sokka from the airbending cartoon. 😂</p><p>Matlab provides a tool called Filter Designer. Let&#39;s explore it! Summon the filter designer by typing “filterDesigner” in your command window. A new window unfolds, brimming with possibilities — close the tip of the day, we’ve got work to do.</p><p>Now, there are a lot of useful features but let’s keep it simple. We’re crafting a low pass filter with a band stop above 900 and below 1200. Here’s the playbook:</p><ul><li>Select “Lowpass” in Response type.</li><li>Set Fpass at 900 (or a smidge above) and the bandstop at 1000 (below 1200).</li><li>Set the sample frequency to 3000 Hz.</li><li>Leave everything else at default.</li><li>Click “Design filter” at the bottom</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*ZKy-MoBQPJwiYqUrCyM7Ag.png" /><figcaption>Filter designer interface</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*9EI-mXn9y91DTRVkf_OYbw.png" /><figcaption>Filter Designed</figcaption></figure><p>Once your masterpiece is ready, click “Files,” then “Export,” and choose “Export to Workspace” and “Export as Objects.” Name your filter (let’s call it “mylowpassfilter”) and hit export.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*9RPDAvc8rUZgeEgJtDML6Q.png" /><figcaption>Filter designer export window</figcaption></figure><p>After exporting let&#39;s get back to the Matlab interface!</p><h3><strong>Filtering Out the Noise</strong></h3><p>Now, let’s tackle the unruly “dirtyData” with our freshly crafted filter and Matlab filter function. Behold the magic in code:</p><p>Code snippet</p><pre>% filtering<br>filterData= filter(mylowpassfilter, dirtyData);<br>filterDataFd = abs(fft(filterData));</pre><h3><strong>Admire the clean data and your new skillset</strong></h3><p>Visualize and admire your result!</p><pre>figure;<br>% Plot the magnitude spectrum of the filtered signal<br>plot(filterDataFd);<br>xlabel(&#39;Frequency (Hz)&#39;);<br>ylabel(&#39;Magnitude&#39;);<br>title(&#39;Magnitude Spectrum of Filtered Signal&#39;);</pre><p>Output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*XQHDYgM762SxOl8inhqN5Q.png" /><figcaption>Magnitude Spectrum of Filtered signal</figcaption></figure><p>Hooray! Bask in the glory of your new skill. You’ve just unleashed the power of filtering in MATLAB. Cheers to cleaner signals and a brighter signal processing journey.</p><h3><strong>The grand finale and beyond</strong></h3><p>As our MATLAB journey comes to a close, let’s gather around for a quick recap. You’ve conquered the basics, danced with codes, and unleashed the power of plots. MATLAB, your trusty companion, has proven itself a signal processing wizard.</p><p>But hey, this isn’t the end; it’s just the beginning. Imagine applying these newfound skills to real-world scenarios. Picture this: filtering out the unwanted noise from your favorite song, creating a symphony of pure audio bliss. Filters aren’t just lines of code; they’re the maestros orchestrating harmony in our tech-driven world.</p><h3><strong>See You Next Time</strong></h3><p>As we bid adieu for now, remember, that this is just a pause, not the end. What’s next on our agenda? More textbook assignments to polish those skills, preparing you for the grand stage of real-life applications. Soon, you’ll be weaving MATLAB’s magic in music and silencing the noise in the data-driven world of robotics. Stay curious, stay excited — see you in the next tutorial!</p><p>If you have any questions or have any specific tutorial you want me to make, reach out to me on my Facebook page <a href="https://web.facebook.com/profile.php?id=61555879295314">Phronexus Freelance Services</a> or leave a comment here.</p><p>Here is a <a href="https://github.com/wayne-chi/LowpassFilter/tree/main">link </a>to my github repo with the code.</p><p>Adieu!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7e22d8eb95c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>