<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pei Seng Tan on Medium]]></title>
        <description><![CDATA[Stories by Pei Seng Tan on Medium]]></description>
        <link>https://medium.com/@peisengtan?source=rss-7835637487b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*XM6-qaJ3gD_Emfs9V8dtig.jpeg</url>
            <title>Stories by Pei Seng Tan on Medium</title>
            <link>https://medium.com/@peisengtan?source=rss-7835637487b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:48:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@peisengtan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Resume Information Extraction (Part 2)]]></title>
            <link>https://medium.com/vitrox-publication/resume-information-extraction-part-2-5732899e10db?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5732899e10db</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[resume]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Thu, 05 May 2022 06:41:57 GMT</pubDate>
            <atom:updated>2022-05-05T06:41:57.801Z</atom:updated>
            <content:encoded><![CDATA[<h4>Research Papers That You Should Not Let Slip</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yCbJlE5DRlrWa5zQ" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article is the sequel of <a href="https://medium.com/vitrox-publication/resume-information-extraction-part-1-290d561a15f4">Resume Information Extraction (Part 1) — Research Papers That You Should Not Let Slip</a>. For those who haven’t read it yet, please do so.</p><p><a href="https://medium.com/vitrox-publication/resume-information-extraction-part-1-290d561a15f4">Resume Information Extraction (Part 1)</a></p><h4>Paper 4: Resume Information Extraction with A Novel Text Block Segmentation Algorithm (Zu and Wang, 2019)</h4><p>The authors suggest an end-to-end resume parsing pipeline based on <strong>neural network-based classifiers</strong> and <strong>distributed embeddings</strong> to make use of position-wise line information and integrated word representations inside each text block. The authors’ motivation for proposing this is as follows:</p><ol><li>Neural network-based feature extraction outperforms hand-crafted features in capturing more semantics from texts.</li><li>Word embeddings served as “universal feature extractors” can better represent words than human-designed features.</li><li>Pre-trained word embeddings are convenient to use.</li></ol><p>Figure 1 depicts the whole pipeline for the proposed resume information extraction technique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*w-ClGV8V8NEIo0LHnvSjdw.png" /><figcaption>Figure 1: Pipeline for the proposed resume information extraction technique.</figcaption></figure><p>The proposed <strong>text block classification</strong> method trains two kinds of line classifiers, namely <strong>line type classifier</strong> and <strong>line label classifier</strong>. The line type classifier’s purpose is to break resumes into basic parts based on four different types of generic layouts: header, content, metadata, and footer. Personal, education, work, project, skill, and publishing are the six general information areas that the line label classifier aims to segment. Table 1 illustrates the categories and subcategories of six general information fields.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*HTaSGyz-xpZCx5K8q2bj2Q.png" /><figcaption>Table 1: Predefined information field.</figcaption></figure><p>Pre-trained <strong>word embeddings</strong> particularly for the resume-related domain are generated by using the Word2Vec model from the <a href="https://pypi.org/project/gensim/">Gensim</a> tool as the default model to train word embeddings by applying to phrase collections or corpora effectively to transform line lists in text resumes into word vectors. Other word representation models, such as GloVe and BERT, are also tested. During training, these embeddings would be fine-tuned. Text-CNN, RCNN, Adversarial LSTM, Attention BLSTM, and Transformer are the types of neural network architecture that are considered for text block classification.</p><p>The process of <strong>text sequence labeling</strong> is carried out after the completion of text block classification. The sub-categories, which are mentioned in Table 1, are used for the named entity recognition (NER) which tags phrases in the sentence. Bi-LSTM-CRF, Bi-GRU-CRF, IDCNN-CRF, and BLSTM-CNNs-CRF are four sequence labeling classifiers that are trained and assessed in terms of performance and decoding speed.</p><p>The authors used the k-means technique to cluster the detected named entities and compute the cosine similarity between them based on Term Frequency–Inverse Document Frequency (TFIDF) to translate the named entity candidates to the standard attribute names.</p><p>As demonstrated in Figure 2, Attention BLSTM beat other text classifiers in the <strong>line type classification</strong>, with F-1 values of 0.96, 0.93, 0.96, and 0.97 for header, content, metadata, and footer, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*jakhtPNIE4aiA20aj6JFeQ.png" /><figcaption>Figure 2: Precision (P), Recall (R), and F-1 Measure (F-1) of line type classification performed by five text classifiers for four generic layouts.</figcaption></figure><p>With regards to the <strong>line label classification</strong> for the six general information fields, as shown in Figure 3,</p><ol><li>Attention BLSTM and Adversarial LSTM outperform other classifiers in classifying long sentences with higher recalls and F-1 measures.</li><li>Text-CNN outperforms the other text classifiers for short phrases.</li><li>RCNN achieves better classification performance over Text-CNN in terms of long sentences.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*E_1fP5KUqQA0DUCY4gcdoA.png" /><figcaption>Figure 3: Precision (P), Recall (R), and F-1 Measure (F-1) of line label classification performed by five text classifiers for six general information fields.</figcaption></figure><p>Given Attention BLSTM’s favorable classification performance and strong robustness against both short and long sentences, the authors decided to use Attention BLSTM to segment text blocks in practical implementation.</p><p>Table 2 compares the outcomes of four suggested <strong>sequence labeling classifiers</strong> when it comes to identifying resume facts. The authors came to three findings based on their comparative investigation.</p><ol><li>BLSTM-CNNs-CRF outperforms the other three sequence labeling classifiers.</li><li>The greedy ID-CNN outperforms both Bi-LSTM and Bi-GRU when paired with Viterbi-decoding.</li><li>Bi-GRU slightly outperforms Bi-LSTM in the NER task when paired with the CRF.</li></ol><p>Regarding the decoding speed, as shown in Table 2 as well, IDCNN-CRF has the fastest decoding speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*wwaKx2UV_8lN1Cfh0QIlwA.png" /><figcaption>Table 2: The F-1 Measures for resume facts identification by four sequence labeling classifiers.</figcaption></figure><p>Other findings provided by the authors include:</p><ol><li>BLSTM-CNNS-CRF’s CNN Layer is an Effective Text Feature Extractor compared to LSTM Layer</li><li>The proposed approach outperformed Writing-Style and CHM.</li><li>BERT is the most advanced word representation algorithm compared to randomized initialization, Word2Vec, and GloVe.</li></ol><h4>Paper 5: Resume Parsing Framework for E-recruitment (Sajid et. al., 2022)</h4><p>The disadvantage of supervised learning, as used in Paper 4, is that it necessitates a substantial amount of resume annotation. In the actual world, gathering a large number of annotated datasets is difficult. To address this, this author’s study proposed a resume parsing framework that addresses the shortcomings of earlier methodologies, such as rule-based, supervised, and semantics-based methods. The proposed structure for resume processing in this paper is shown in Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*vCXxVz4zOltKPHiivgOShA.png" /><figcaption>Figure 4: Proposed structure for resume processing.</figcaption></figure><p>The raw text is taken from resumes first. PDFBox is a utility for converting PDF files to text files. Non-ASCII characters, missing or additional spaces, and punctuation marks are all handled correctly. For text block categorization, the blocks are divided using Boolean Naive Bayes (BNB). The entities are then extracted using BERT for NER.</p><p>Unlike prior works, the named entities are further extended with a <strong>custom-built ontology</strong> to address data sparsity employing skills from online job portals, notably the CSO and European Skills/Competence, Qualification, and Occupation ontologies (ESCO). Ontology is a representation of a knowledge base that allows for the construction of a semantic model of data connected with specific domain knowledge. Ontology is also used to establish relationships between various types of semantic knowledge in a domain.</p><p>The process of building an ontology includes:</p><ol><li>Ontology Building: Specify motivation, define informal competency questions, build schema and construct axioms.</li><li>Skill Enrichment and Normalization (an Example is shown in Figure 5).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*F20Whr8mAsoeQDjsIhswNw.png" /><figcaption>Figure 5: Skill Enrichment.</figcaption></figure><p>The conceptual models of skill ontology are shown in Figure 6. The circle and rectangles depict the classes and instances respectively. Edges represent the relationships between skills. (a) shows the overall schema of the skill ontology containing classes, subclasses, and instances. (b) depicts the details between multiple instances. The SameAs property shows the equivalence between two skills whose representations might be different in the real-world, for instance, Microsoft Office.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BnBKC2uZBxA2qdY0-TdEjA.png" /><figcaption>Figure 6: Conceptual models of skill ontology</figcaption></figure><h4>Other Related Papers:</h4><p>The papers listed below are ones I’ve read but didn’t include in this or earlier articles.</p><ol><li>An Ontology-Based Information Extraction Approach for Résumés (Celik and Elci, 2012)</li><li>Combination of Neural Networks and Conditional Random Fields for Efficient Resume Parsing (Ayishathahira et. al., 2018)</li><li>Catapa resume parser: End to end Indonesian resume extraction (Lumban et. al., 2019)</li><li>Resume extraction with conditional random field method (Yu, et. al., 2020)</li><li>A Contextual Model for Information Extraction in Resume Analytics Using NLP’s Spacy (Channabasamma et. al., 2021)</li></ol><h4>Conclusion:</h4><ol><li>[Paper 4] Attention BLSTM and BLSTM-CNNs-CRF are the best candidates for text-block classification and NER tasks respectively.</li><li>[Paper 4] Work Embeddings significantly improve the performance of neural network classifiers or recognizers.</li><li>[Paper 5] Ontology helps to address data sparsity employing skills from online job portals.</li></ol><h4>References:</h4><ol><li>Zu, S., &amp; Wang, X. (2019). Resume information extraction with a novel text block segmentation algorithm. <em>Int J Nat Lang Comput</em>, <em>8</em>, 29–48.</li><li>Sajid, H., Kanwal, J., Bhatti, S. U. R., Qureshi, S. A., Basharat, A., Hussain, S., &amp; Khan, K. U. (2022, January). Resume Parsing Framework for E-recruitment. In <em>2022 16th International Conference on Ubiquitous Information Management and Communication (IMCOM)</em> (pp. 1–8). IEEE.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5732899e10db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/vitrox-publication/resume-information-extraction-part-2-5732899e10db">Resume Information Extraction (Part 2)</a> was originally published in <a href="https://medium.com/vitrox-publication">ViTrox-Publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Resume Information Extraction (Part 1)]]></title>
            <link>https://medium.com/vitrox-publication/resume-information-extraction-part-1-290d561a15f4?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/290d561a15f4</guid>
            <category><![CDATA[resume]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Thu, 24 Mar 2022 01:30:28 GMT</pubDate>
            <atom:updated>2022-03-24T01:30:28.420Z</atom:updated>
            <content:encoded><![CDATA[<h4>Research Papers That You Should Not Let Slip</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W5CkCDqaNytORdAj" /><figcaption>Photo by <a href="https://unsplash.com/@joaoscferrao?utm_source=medium&amp;utm_medium=referral">João Ferrão</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The fast transition of recruitment from traditional job fairs to web-based e-recruiting platforms has occurred over the last 10 years. According to the statistics, well-known third-party e-recruitment sites, such as <a href="https://www.monster.com.my/">Monster</a> and <a href="https://www.linkedin.com/">LinkedIn</a>, publish more than 300 million resumes each year (Zu and Wang, 2019).</p><p>The <strong>resume</strong> is a formal document used by job searchers to demonstrate their skills and competencies to the human resources departments of targeted firms or headhunters to secure desired positions. The job of automatically extracting structured information from unstructured or semi-structured machine-readable documents and other electronically represented sources is known as <strong>information extraction</strong>.</p><p>Because of the large volumes of personal data in various formats, how to efficiently access each resume is a serious issue that has successfully piqued the interest of academicians (Chen et al., 2018). Therefore, the motivation that drives me to write this article is to dissect the resume information extraction pipeline from the perspectives of academicians.</p><h3>Paper 1: Resume Information Extraction With Cascaded Hybrid Model (Yu et al., 2005)</h3><p>According to the study on the ways human beings prepare their resumes, resume information can be typically defined as a two-layered hierarchical structure. The first layer consists of sequential generic information blocks such as personal information, education, and so on. Inside each general information block, detailed information pieces can be further analyzed. For an instance, in the block of personal information, detailed information, such as name, address, email, and so on, can be extracted. Table 1 shows the information types mentioned by the authors in the paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*ZzzXxSOpm3KOUHJ6lZ4JGA.png" /><figcaption>Table 1: Information types (Yu et al., 2005)</figcaption></figure><p>The authors proposed a <strong>cascaded information extraction framework</strong> to perform resume information extraction. Instead of searching the entire resume, as is done with the flat model, a resume will firstly be segmented into consecutive blocks attached with labels indicating the information types, followed by the identification of detailed information within a specific block.</p><p>Figure 1 shows the whole pipeline of resume information extraction using the proposed cascaded framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*kfYyeL_IlJ8GvJHNH9MSlQ.png" /><figcaption>Figure 1: Structure of cascaded hybrid models (Yu et al., 2005)</figcaption></figure><p>Experimenting with 1200 Chinese resumes, the result reveals that the hierarchical structure of resume information extraction with the suggested cascaded framework enhances the average F-score of detailed information extraction, as shown in Table 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*aoalq5pwx0BjmeFpbcsQ5A.png" /><figcaption>Table 2: Result with cascaded and flat models (Yu et al., 2005)</figcaption></figure><p>By correctly mixing multiple information and extraction models in different layers, it is also effective to attain excellent precision and recall. Based on the results shown in Table 3 and Table 4, the <strong>Hidden Markov Model (HMM)</strong> is good in handling the general information extraction and educational detailed information extraction whereas the <strong>Support Vector Machine (SVM)</strong> is recommended for the personal detailed information extraction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/1*TnCFeZQhnghK0nO37vAAVw.png" /><figcaption>Table 3: General information extraction with different models (Yu et al., 2005)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/1*bRwJnuUuCntVo05INIh6rw.png" /><figcaption>Table 4: Detailed information extraction with different models (Yu et al., 2005)</figcaption></figure><h3>Paper 2: Information Extraction from Resume Documents in PDF Format (Chen et al., 2016)</h3><p>Similar to Paper 1, the authors adopt a hierarchical information extraction approach to access data from resume documents.</p><p>Initially, a page is divided into blocks based on heuristic rules. Following that, a well-trained classification model, which is a <strong>Support Vector Machine (SVM)</strong> model, is used to categorize each block into pre-defined categories. The detailed information extraction work is then viewed as a sequence labeling issue by using a <strong>Conditional Random Fields (CRF)</strong> model to perform the task.</p><p>Figure 2 shows the flow of the entire resume information extraction pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*agUGkhKxmtx0RjAebQXDfw.png" /><figcaption>Figure 2: Flow of resume information extraction pipeline (Chen et al., 2016)</figcaption></figure><p>Different from Paper 1, the authors proposed to take the advantage of the structure and layout information of PDF documents to improve the accuracy of resume information extraction by feeding the classification model with two kinds of features, which are <strong>content-based</strong> and <strong>layout-based</strong>. The content-based features are normally texts whereas the layout-based features refer to the font size, font style, bounding box, and line position information.</p><p>The experimental result verified with 400 resume documents shows that the introduction of layout-based features into modeling significantly increases the precision, recall, and F1-score, as shown in Table 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*Y9EzPyTe0YKopNnjNsQB6g.png" /><figcaption>Table 5: Contribution of features (Chen et al., 2016)</figcaption></figure><p>Due to the page being divided into blocks based on heuristic rules, the performance of entire resume information extraction may be <strong>varied with the block sizes</strong>. To confirm this, the authors had a study on this.</p><p>The block size of the flat structure is the entire document. The hierarchical structure of tiny blocks treats each paragraph as a text block. The hierarchical structure of large blocks is the result of page segmentation based on the heuristic rules defined by the authors.</p><p>To go deep, the heuristic rules are designed according to a recursive bottom-up algorithm. The size of blank spaces between lines is ranked. Little blocks are merged vertically and horizontally based on certain thresholds and bounded by constraints to prevent miss-segmenting or over-segmenting.</p><p>As shown in Table 6, the hierarchical structure outperforms the flat structure in terms of precision and F1-score, with just a minor loss in the recall. It strengthens the Paper 1 authors’ view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*2gu53urpHQ6Vw_GlTDuPvg.png" /><figcaption>Table 6: The effect of the different block sizes (Chen et al., 2016)</figcaption></figure><p>In addition, the experimental result demonstrates that the hierarchical model with big blocks is more efficient in terms of F1-score. Choosing a little block increases the number of mistakes in the categorizing block stage.</p><p>By further performing error analysis on the built pipeline, the authors managed to improve the performance by addressing specific problems in detailed information extraction, including:</p><ol><li>The issue of the under-graduate university and graduate university of job searchers are the same.</li><li>The issue of the name attribute does not appear in the personal block but appears as the title of the documents.</li></ol><h3>Paper 3: A Two-Step Resume Information Extraction Algorithm (Chen et al., 2018)</h3><p>Similar to Paper 1 and Paper 2, the authors applied a two-step resume information extraction method to get data from resume documents.</p><p>In the first step, the raw text of the resume is recognized as separated blocks. Different from aforesaid papers, a unique feature, namely <strong>Writing Style</strong>, is generated from sentence syntax information modeling to further improve resume information extraction. The Writing Style encodes not only word and punctuation indexes but also word lexical attributes and prediction results of classifiers. In order words, the Writing Style is a kind of syntax feature about the structure of a line in a resume. In the second step, multiple classifiers are adopted to recognize distinct features within each separate resume block.</p><p>Figure 3 shows the proposed pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*c7FU1oXtEJPTITINqrYrJQ.png" /><figcaption>Figure 3: Proposed Pipeline (Chen et al., 2018)</figcaption></figure><p>To obtain the <strong>Writing Style</strong> features, the authors use <a href="https://tika.apache.org/">TIKA</a> to process raw resume documents. The limitation of TIKA includes the raw text is not in accordance with the original layout. There is a lot of noise within the lines in each text file, such as continuous blank, incorrect newlines, and the essential space missing. three types of operations are involved, namely merging, splitting, and trimming, to process raw resume documents. More details about the heuristic rules for data cleaning are stated in Table 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*QbMFN6kwABhU0IInIMNP4A.png" /><figcaption>Table 7: Heuristic rules for data cleaning (Chen et al., 2018)</figcaption></figure><p>After the raw resume documents are processed, a simple multi-class classifier is trained to perform a similar task like named entity recognition. The labels include university name, job position name, department name, ID number, address, and date. The probability distribution on a different class is combined with the position of the phrase and the symbol to form Writing Style features. In other words, the probability distribution on a different class is obtained to enrich the inputs before both text block classification and resume facts identification steps.</p><p>In the step of <strong>text block classification</strong>, the authors define three types of different lines to facilitate the follow-up work as follows:</p><ol><li>Simple — The line is a short text and may contain a few blanks.</li><li>KeyValue — The line follows the key and value structure, with comma punctuation.</li><li>Complex — The line is a long text, which contains more.</li></ol><p>The results displayed in Table 8–10 reveal that the Writing Styles help to improve the performance of text block classification while comparing to the performance of PROSPECT (Singh et al., 2010) or CHM (Yu et al., 2005).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*HBdoBuQelirkLLQXWRrV_A.png" /><figcaption>Table 8: Education Block Classification (Chen et al., 2018)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*VUmr4W5f6veJ3UOsxo1WSQ.png" /><figcaption>Table 9: Work Experience Block Classification (Chen et al., 2018)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*xPrL-DTE9bb2mn05NpCpvg.png" /><figcaption>Table 10: Basic Information Block Classification (Chen et al., 2018)</figcaption></figure><p>In the step of <strong>resume facts identification</strong>, those lines with key-value structures are considered to be the candidate attribute. Firstly, the Cosine Similarity is computed based on Term Frequency-inverse Document Frequency (TF-IDF) and the K-means is applied to perform attribute clustering for standard attribute name matching.</p><p>The proposed method shows significant improvement in terms of the resume annotation hours required. As presented in Figure 4, the proposed system does not need too much manually annotated training set, which can save lots of human effort and time, delivering a piece of good news to the project teams which are not big in size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*GpLq6_CQX5j3M5TuYOReDw.png" /><figcaption>Figure 4: Hours of annotations (Chen et al., 2018)</figcaption></figure><h3>Conclusion</h3><p>The main key idea that I got from these 3 papers for resume information extraction include:</p><ol><li>[Paper 1] The hierarchical cascaded model structure performs better than the flat model structure.</li><li>[Paper 2] The layout-based features help to improve the performance of resume information extraction.</li><li>[Paper 2] The size of the resume block is matters as it will affect the performance of block classification.</li><li>[Paper 3] The syntax features, including word lexical attribute and prediction results of text classifiers, help to improve the performance of resume information extraction.</li><li>[Paper 3]The use of clustering algorithm, like K-means, reduces the human work in annotations.</li></ol><h3>References:</h3><ol><li>Yu, K., Guan, G., &amp; Zhou, M. (2005). Resume information extraction with cascaded hybrid model. In <em>Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)</em> (pp. 499–506).</li><li>Chen, J., Gao, L., &amp; Tang, Z. (2016). Information extraction from resume documents in pdf format. <em>Electronic Imaging</em>, <em>2016</em>(17), 1–8.</li><li>Chen, J., Zhang, C., &amp; Niu, Z. (2018). A two-step resume information extraction algorithm. <em>Mathematical Problems in Engineering</em>, <em>2018</em>.</li><li>Zu, S., &amp; Wang, X. (2019). Resume information extraction with a novel text block segmentation algorithm. <em>Int J Nat Lang Comput</em>, <em>8</em>, 29–48.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=290d561a15f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/vitrox-publication/resume-information-extraction-part-1-290d561a15f4">Resume Information Extraction (Part 1)</a> was originally published in <a href="https://medium.com/vitrox-publication">ViTrox-Publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differences: DevOps, ITOps, MLOps, DataOps, ModelOps, AIOps, SecOps, DevSecOps [Part 1/3]]]></title>
            <link>https://medium.com/vitrox-publication/differences-devops-itops-mlops-dataops-modelops-aiops-secops-devsecops-part-1-3-8b238cf72942?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b238cf72942</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[itops]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Mon, 05 Jul 2021 07:08:35 GMT</pubDate>
            <atom:updated>2021-07-05T07:08:35.210Z</atom:updated>
            <content:encoded><![CDATA[<h4>All of them are derived from AGILE Methodology</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W13HDTpfBrfeQkq7Cr_2ag.png" /><figcaption>The babies of Agile Methodology</figcaption></figure><p>Sometimes, I found out that Junior Full Stack Developers, Software Engineers, Machine Learning Model Developers or Data Scientists were confused with various terms that are combined with the word “ops”. After a few hours of reading, I planned to explain them with a better flow. <strong>When you trace back the origins of these terminologies, you can easily figure out that actually, they are the babies of Agile Methodology</strong>.</p><h3>So, What is Agile?</h3><p>Agile is defined as an interactive approach specifically designed for project management and software development to deliver value to their customers faster and more reliable. Instead of evaluating the entire development process after completion, you can actually chop the entire development process into multiple small blocks and evaluate each of them separately and asynchronously.</p><p>Agile advocates the work within a team should be delivered in small units, but consumable and incremental, so the team can respond to the change quickly [1]. It also promotes adaptive planning, evolutionary development, early delivery, and continual improvement [2]. Currently, it contains a set of frameworks and practices based on the values and principles expressed in the <a href="https://www.agilealliance.org/agile101/the-agile-manifesto/">Manifesto for Agile Software Development </a>and the <a href="https://www.agilealliance.org/agile101/12-principles-behind-the-agile-manifesto/">12 Principles</a> [3].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/0*hjyPhfKGxB452SU3.png" /><figcaption>Most 5 valuable benefits of Agile Methodology [<a href="https://www.denysys.com/blog/5-benefits-of-agile-methodology/">Denysys</a>]</figcaption></figure><p>In short, Agile consists of <strong>a set of standard practices</strong> where <strong>the project management and software development teams should follow to establish a product</strong>, mainly software, that is <strong>user-friendly, high performance, easy to maintain, high extendability, high reliability, high quality and safe to be used</strong>.</p><h3>DevOps</h3><p>To make the story simple, I start with the term “DevOps”. <strong>Let’s imagine that you are asked to develop a simple food ordering web application</strong>. As the head of the software development team, <strong>what do you think the development flow of a simple food ordering web application should be?</strong> Take a minute to think further about it before you proceed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_kYXY---H10Xpwxu.png" /><figcaption>Example of a food ordering application [<a href="https://www.ordertiger.com/">Order Tiger</a>]</figcaption></figure><p>Probably the answer in your mind is similar to the development flow as follows [4]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*-npW9vly3IqXEfxMprYBCQ.png" /><figcaption>Traditional software development process [<a href="https://admhelp.microfocus.com/octane/en/15.1.60/Online/Content/Flows/flow_SW_Development.htm">ALM Octane</a>]</figcaption></figure><ol><li>Requirements: List the requirements of what the product needs to do. This may or may not include a list of features.</li><li>Design: Define how the requirements work inside the product.</li><li>Implementation: Develop the product.</li><li>Testing/Verification: The quality assurance team tests and verifies the planned features. Open defects for those problems encountered in testing.</li><li>Release/Maintenance: Release the finished and approved product.</li></ol><p>Basically, this flow is a part of the DevOps. The set of practices that works to automate and integrate the processes between software development team members is known as DevOps. The addition of the automation and integration elements into the traditional software development process allows the aforementioned team to build, test, and release software faster and more reliable [5].</p><p>Automation, iteration, self-service, continuous improvement, collaboration and continuous testing are the keys principles that underscore the Agile philosophy. DevOps extends the Agile mindset to development and operation. Each phase in the DevOps lifecycle is focusing on closing the loop between continuous integration (CI), continuous delivery (CD), continuous deployment, continuous monitoring and feedback [6].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*YlJrqlM8NAvLgv_D.jpg" /><figcaption>The DevOps lifecycle [<a href="https://liveframe.org/devops-process-description/">LifeFrame</a>]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XT74FBaVqooKiMnf.jpg" /><figcaption>The CI, CD, continuous Deployment and DevOps [<a href="https://medium.com/@ferhatsukrurende/continuous-integration-delivery-deployment-ci-cd-e1f3cbd14d5e">Medium</a>]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QfMyK4gIBDkuqIBi_gPygw.png" /><figcaption>The details of each stage in DevOps lifecycle (Excluded Monitor) [<a href="https://cloudtweaks.com/2019/05/devops-secure-and-scalable-ci-cd-pipeline-with-aws/">CloudTweaks</a>]</figcaption></figure><p>The technologies that have successfully gained a lot of popularities recently due to DevOps include Containerization and Microservices [7].</p><p>In short, <strong>DevOps extends the Agile mindset</strong> by <strong>including the concept of CI, CD, continuous deployment, continuous monitoring and feedback</strong>.</p><p>Congratulations! These are fundamental knowledge that you should know about DevOps. Let proceed to the next one, ITOps.</p><h3>ITOps</h3><p>As mentioned, DevOps is a standard guideline to convert software project idea into production readiness. For software developers to practise DevOps, other underlying services including platform and infrastructure (storage, compute, network) are required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*M-ORvKUYdf3kcbGF.jpg" /><figcaption><em>Scheme of IT Infrastructure management [</em><a href="https://www.smartsheet.com/it-infrastructure-management-services-guide"><em>SmartSheet</em></a><em>]</em></figcaption></figure><p>This is where ITOps comes into place. The Information Technology (IT) team of a company is normally responsible for acquiring, designing, deploying, configuring and maintaining the physical and virtual components that comprise the IT infrastructure [8]. In specific, an IT team uses ITOps to deliver and maintain various services, applications, and technologies needed to run a business [9].</p><p>The scopes of ITOps normally include [10]:</p><ul><li><strong>Infrastructure Management </strong>— The setup, provisioning, maintenance and updating of all the hardware and software in the organization to ensure that all existing infrastructure components run smoothly and new solutions are integrated seamlessly.</li><li><strong>Development Management</strong>—The preparation of the exact guidelines, goals, security standards and workflow that software development teams need to succeed.</li><li><strong>Security Management </strong>— The work of keeping the hardware and software secure, managing access control, implementing security within DevOps processes and ensuring security standards are met across the environment.</li><li><strong>Problem Management </strong>—The solution to deal with outages and cyberattacks, and perform disaster recovery and help desk services.</li></ul><p>To conclude, <strong>ITOps is a set of standard practices adopted by the IT department to perform infrastructure, development, security and problem management.</strong></p><p>Congratulations! These are fundamental knowledge that you should know about ITOps.</p><h3>Case Study: The Integration of DevOps and ITOps</h3><h4>The Implementation of DevOps</h4><p>Let’s continue the imagination regarding <strong>you are head of the software development team for the simple food ordering web application</strong>.</p><p>According to ITOps introduced by the company’s IT department, you are required to initiate the new project with the DevOps methodology.</p><blockquote>Step 1: Plan</blockquote><p>According to the DevOps lifecycle as mentioned above, firstly, you need a <strong>plan</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/0*FLDxpcUcDuvmpKND.png" /><figcaption>The process flow of requirements engineering [<a href="https://medium.com/omarelgabrys-blog/software-engineering-process-activities-part-3-ca1ef6818fd6">Medium</a>].</figcaption></figure><ul><li>You ask your subordinate to <strong>conduct a software requirement analysis</strong> based on the requirement engineering process flow to <strong>rank the essential features </strong>for the simple food ordering web application.</li><li>The first feature may be <strong>allowing the users to perform some filtering process based on food type, price range and rating to speed up the user selection process.</strong></li><li>The second feature may be <strong>recommending users what to eat based on their previous orders and personal preferences</strong>.</li></ul><blockquote>Step 2: Code</blockquote><p>Next, you assign the coding tasks based on essential features to your subordinates.</p><ul><li>They <strong>develop codes</strong> on their devices.</li><li>They push codes to the dedicated repository located on a cluster of internal servers using <a href="https://git-scm.com/">Git</a> for code integration and merging.</li><li>Before they push the codes, they perform unit tests on their written modules.</li></ul><p>After a few weeks of hard work, all the modules are completely developed and integrated. The primary version of the simple food ordering web application is considered done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hOVrg13AF6CqvCXs.png" /><figcaption>An example of web application backend architecture design for food ordering purpose [<a href="https://hackernoon.com/how-to-build-a-food-ordering-app-an-ultimate-guide-on-food-ordering-mobile-app-c19441826aca">Harkernoon</a>].</figcaption></figure><blockquote>Step 3 and 4: Built and Test</blockquote><p>Then, you and your subordinates try to<strong> build</strong> and <strong>test</strong> the entire application on the dedicated development server assigned by the IT department.</p><ul><li>For the first launch, there are multiple bugs in the web application leads to a major crash.</li><li>You and your subordinates fix the bugs and conduct the next round of testing repetitively.</li><li>The process which involves iterative coding, building and testing that is automated is named <strong>“continuous integration (CI)</strong>”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/0*zHCE08v_4sgUREn1.png" /><figcaption>The CI, CD and continuous deployment [<a href="https://www.redhat.com/en/topics/devops/what-is-ci-cd">Red Hat</a>]</figcaption></figure><blockquote>Step 5: Release</blockquote><p>After a few round of testing, the web application performs more stably. You decide to <strong>release </strong>it to production.</p><ul><li>Someones from the business development team and IT department prepare for the new software launch and infrastructure setup respectively.</li><li>Internal testers, who are playing the roles of <strong>quality assessment</strong> to the web application, also perform multiple tests to ensure the stability of the web application in the real production environment.</li><li>The automatic process of iterative code development, release and testing before the deployment of the web application to the public is known as <strong>“continuous delivery (CD)”</strong>.</li></ul><blockquote>Step 6: Deploy</blockquote><p>When everything internally is ready, the web application is allowed to be <strong>deployed </strong>to real production<strong>. </strong>The readiness here include:</p><ul><li>The front end of the web application is available to be downloaded from the App Store.</li><li>The back end of the web application running on top of IT infrastructure is ready to serve all the requests from the front end.</li></ul><blockquote>Step 7 and 8: Operation and Monitor</blockquote><p>The <strong>operation</strong> of the web application is constantly <strong>monitored </strong>when the users are interacting with it at all times. The user feedback is collected periodically for enhancement.</p><p>The automatic process of interactive code development, release and deployment to the public is known as <strong>“continuous deployment”</strong>. As an application user, you get to know this only when you get a phone message to remind you to update the application. CI and CD are always done behind the scene without letting you know.</p><p>Finally, the story of how the head of the software development team implements DevOps comes to the end.</p><h4>The Implementation of ITOps</h4><p>Do you <strong>spot any role played by the IT department</strong> in the story above?</p><p>Referring to the story above, the IT department uses ITOps in/to:</p><blockquote>Development Management:</blockquote><ul><li><strong>Enforce all the business development teams in a company to practice DevOps methodology.</strong> For example, the ITOps can specify that all the software development team should submit their hardware application forms 7 days in advance so the IT department has an adequate period to fulfil the requirements.</li></ul><blockquote>Infrastructure Management:</blockquote><ol><li><strong>Prepare the hardware and software to facilitate the software development process.</strong> For example, personal devices for coding, a dedicated repository located on a cluster of internal servers for code integration and merging, and a dedicated development server for code building and testing.</li><li><strong>Control the infrastructure for production.</strong> For example, deployment, monitoring, and maintenance, and updating of the back end of the web application running on a cluster of internal servers.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7sG0ukK8-tKG7GyS.png" /><figcaption>The dashboard provided by <a href="https://www.terraform.io/">Terraform</a> to monitor hardware utilities [<a href="https://medium.com/@chfrank_cgn/updated-building-a-kubernetes-cluster-on-google-cloud-with-rancher-and-terraform-4ac57d5eb303">Medium</a>].</figcaption></figure><blockquote>Security Management:</blockquote><ol><li><strong>Ensure the security of hardware and software.</strong> It includes personal coding devices, development servers and production servers for the web application.</li><li><strong>Manage access control.</strong> It includes access to the dedicated repositories for code integration and merging (normally hosted on <a href="https://github.com/">Github</a> or <a href="https://about.gitlab.com/">Gitlab</a>)</li><li><strong>Make sure that the web application is well covered in all the security aspect.</strong> For example, the user is required to key in the password to make the changes on personal registered details.</li></ol><blockquote>Problem Management:</blockquote><ol><li><strong>Make sure that the infrastructure set up for production is fault-tolerant.</strong> For example, the is at least a backup for the database that stores the customer information.</li></ol><h4>The Overlapping Region Between DevOps and ITOps</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*TVnX91iQvUb7LpeD.png" /><figcaption>DevOps as a joint effort and areas of responsibility for R&amp;D, QA, and ITOps [<a href="http://DevOps as a joint effort and areas of responsibility for R&amp;D, QA, and ITOps">STXNEXT</a>]</figcaption></figure><p>In the nutshell, DevOps covers the portions of the development (plan, code and build), quality assurance (test) and operation (release, deploy, operate and monitor) of the software. On the other hand, ITOps focus more on the operation and less on the development (ITOps urge to provide general guidelines) and quality assurance (ITOps urge to conduct tests for the backend of application deployed on IT infrastructure).</p><h3>To Be Continued</h3><p>Hopefully, you find the story above is interesting and help you to understand DevOps and ITOps. The next part will introduce MLOps, DataOps, ModelOps, and explain how they relate to DevOps and ITOps.</p><h3>References</h3><p>[1] <a href="https://www.atlassian.com/agile">https://www.atlassian.com/agile</a></p><p>[2] <a href="https://www.semanticscholar.org/paper/Manifesto-for-Agile-Software-Development-Beck-Beedle/3edabb96a07765704f9c6a1a5542e39ac2df640c">https://www.semanticscholar.org/paper/Manifesto-for-Agile-Software-Development-Beck-Beedle/3edabb96a07765704f9c6a1a5542e39ac2df640c</a></p><p>[3] <a href="https://www.agilealliance.org/agile101/">https://www.agilealliance.org/agile101/</a></p><p>[4] <a href="https://admhelp.microfocus.com/octane/en/15.1.60/Online/Content/Flows/flow_SW_Development.htm">https://admhelp.microfocus.com/octane/en/15.1.60/Online/Content/Flows/flow_SW_Development.htm</a></p><p>[5] <a href="https://www.atlassian.com/devops">https://www.atlassian.com/devops</a></p><p>[6] <a href="https://afrin.me/devops-process-flow/">https://afrin.me/devops-process-flow/</a></p><p>[7] <a href="https://wishdesk.com/blog/devops-trends-2019">https://wishdesk.com/blog/devops-trends-2019</a></p><p>[8] <a href="https://sciencelogic.com/blog/itops-vs-devops">https://sciencelogic.com/blog/itops-vs-devops</a></p><p>[9] <a href="https://www.graylog.org/post/itops-vs-devops-what-is-the-difference">https://www.graylog.org/post/itops-vs-devops-what-is-the-difference</a></p><p>[10] <a href="https://www.ibm.com/cloud/learn/it-operations">https://www.ibm.com/cloud/learn/it-operations</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b238cf72942" width="1" height="1" alt=""><hr><p><a href="https://medium.com/vitrox-publication/differences-devops-itops-mlops-dataops-modelops-aiops-secops-devsecops-part-1-3-8b238cf72942">Differences: DevOps, ITOps, MLOps, DataOps, ModelOps, AIOps, SecOps, DevSecOps [Part 1/3]</a> was originally published in <a href="https://medium.com/vitrox-publication">ViTrox-Publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating Expressiveness and Effectiveness of Informative Charts]]></title>
            <link>https://medium.com/vitrox-publication/evaluating-expressiveness-and-effectiveness-of-informative-charts-9f0455474bf1?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f0455474bf1</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Fri, 13 Nov 2020 03:21:45 GMT</pubDate>
            <atom:updated>2020-11-17T10:57:09.822Z</atom:updated>
            <content:encoded><![CDATA[<h4>Analysis of 8 recent well-spread Malaysia Covid-19 charts on Facebook</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*UfvAdo_9jbednvA6.png" /><figcaption>Image Source: <a href="https://pmtips.net/article/how-data-visualization-tools-can-enhance-your-project-management">PM Tips</a></figcaption></figure><p>To be a certified data analyst, I personally feel that it is very important to know how to perform data visualization expressively and effectively to deliver informative messages to the readers.</p><p>Before we go to the case studies, there are two questions here.</p><h3><strong>How do we measure expressiveness in the context of data visualization?</strong></h3><p>According to Tamara Munzer (2014) who wrote a book named “Visualization Analysis and Design”,</p><blockquote>The chart is considered <strong>expressive</strong> if it can express the information in the dataset attributes.</blockquote><p>The dataset attributes normally consist of 3 main types, which are nominal, ordinal and quantitative. To summarize, both nominal and ordinal attributes are qualitative attributes. However, nominal attributes have no implicit ordering whereas ordinal attributes are the vice versa. Both ordinal and quantitative attributes have implicit ordering but there is no meaning in applying arithmetic operations on ordinal attributes only. The examples of nominal, ordinal and quantitative attributes include:</p><ul><li>Nominal — Types of car, Gender</li><li>Ordinal — Date, Size of T-shirt</li><li>Quantitative — Revenue, Profit</li></ul><p>If the attribute is ordered, it should appear in the chart as ordered attributes. Conversely, unordered data should not be shown in a way that perceptually implies an ordering that does not exist. For an instance, if x-axis is a nominal attribute and y-axis is a quantitative attribute, a bar chart should be drawn instead of a line chart. If x-axis is an ordinal attribute and y-axis is a quantitative attribute, a line chart should be drawn instead of a bar chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/0*mJjuL28GmncORVf5.png" /><figcaption>Image Source: <a href="https://www.microsoft.com/en-us/microsoft-365/blog/2012/06/21/combining-chart-types-adding-a-second-axis/">Microsoft 365</a></figcaption></figure><p>The figure above is a common mistake. The analyst tried to combine the charts of units sold across months and total transaction across months. It should be the combination of two line charts instead of a line chart and a bar chart because the x-axis variable is an ordinal attribute. Thus, it is considered as a bad example.</p><p>Plotting two line charts in the same environment may lead to confusion, therefore, we need to know about the ranking of magnitude and identity channels (which will be covered in the following) to avoid confusion. In this case, different colour of lines can be used to achieve both expressive and effective data visualization.</p><h3>How do we measure effectiveness in the context of data visualization?</h3><blockquote>The chart is considered effective if the importance of all the attributes match the salience of the channels.</blockquote><p>The figure below shows the channel rankings for ordered (quantitative and ordinal) attributes and categorical (nominal) attributes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GyCqKkY3wl5AINk_DsHUaQ.png" /><figcaption>Image Source: Tamara Munzer (2014)</figcaption></figure><p>As shown, the channels for ordered attributes are known as magnitude channels whereas the channels for categorical attributes are known as identity channels.</p><p>Higher the positions of the channels, higher the ranking of effectiveness.</p><p>According to Tamara Munzer (2014), the most important attributes should be encoded with the most effective channels to be most noticeable, and then decreasingly important attributes can be matched with less effective channels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*zzS2AjKSMPuO2Rl3.png" /><figcaption>Image Source: <a href="https://blogs.sas.com/content/graphicallyspeaking/2013/09/20/stacked-bar-chart-with-segment-labels/">SAS</a></figcaption></figure><p>For example, as shown in the figure above, the product attribute is more important than the quarter attribute. Thus, the product attribute is put at the x-axis and the quarter attribute is displayed in different colour blocks. If the readers wish to know the actual sales for each quarter, this chart cannot directly show the figures as the readers have to manually add up the values for each quarter (in same colour blocks).</p><h3>Case Studies:</h3><h4>Example 1:</h4><p>Tables below show the numbers of new and total Covid-19 cases across different clusters. Do you think that the chart below is a good example of data visualization?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/0*OaTYvZXFwt6Xj56O" /><figcaption>Image Source: <a href="https://www.facebook.com/kementeriankesihatanmalaysia/photos/pcb.10157472949491237/10157472942676237/">Ministry of Health Malaysia</a></figcaption></figure><p>According to Tamara Munzer (2014), there are 4 common dataset types, which are tables, networks, fields and geometry. The table is considered as one of the dataset types instead of a visualization chart type. Therefore, it should not be used for expressive and effective data visualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*nv08QXqby9umUiAreiho7A.png" /><figcaption>Image Source: Tamara Munzer (2014)</figcaption></figure><p>Referring to the tables above, there are 3 attributes, namely clusters, number of Covid-19 cases and types of Covid-19 cases (either new or old). The total number of Covid-19 cases is the sum of the number of old and new Covid-19 cases. A stack bar chart should be plotted as shown in the figure below because the attributes’ types of the clusters and the number of Covid-19 cases are nominal and quantitative respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYwjIlL9-YxAewfOWcYKkA.png" /><figcaption>Figure: Self-Drawn Stack Bar Chart (Only cover the content from the left table)</figcaption></figure><p>The total number of Covid-19 cases is displayed in the x-axis and the number of old and new Covid-19 cases are displayed in different colours.</p><p>The clusters are sorted according to the number of new cases like what has been done in the tables above.</p><p>Rumah Merah has the highest number of new Covid-19 cases and Penjara Kepayan has the highest number of total Covid-19 cases.</p><p>Compared to a table, a stack bar chart has a better visual display of quantitative information as people will not be easily distracted with a bunch of numbers. Besides, a stack bar chart gives a better overview of the dataset.</p><h4>Example 2:</h4><p>The figure below shows the distributions of new Covid-19 cases in different districts of Selangor. Do you think that the chart below is a good example of data visualization?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZtfxqUunwVnn_-m3" /><figcaption>Image Source: <a href="https://www.facebook.com/kementeriankesihatanmalaysia/photos/a.390879946236/10157472895606237/">Ministry of Health Malaysia</a></figcaption></figure><p>In my opinion, yes, it is. The data analyst segmented the number of new Covid-19 cases into groups and represented these groups with different colours by using the advantages of the rainbow scale. Red, yellow and green colours rank the seriousness of Covid-19 cases from high to low accordingly.</p><p>If you convert the rainbow scale into grayscale, they are at a certain level of luminance and saturation in order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*UBHPz-XcKo-I8oVfkVa-Cw.png" /><figcaption>Image Source: <a href="https://clauswilke.com/dataviz/color-pitfalls.html">Fundamentals of Data Visualization</a></figcaption></figure><p>Therefore, hue colours can be implemented as one of the magnitude channels as well if the rainbow grayscale is used as the reference. The red colour cannot be replaced with other colours like blue and pink colours as it will reduce the expressiveness and effectiveness of the entire chart.</p><p>Besides, the light-coloured rings are not necessary to be displayed to avoid readers’ confusion.</p><h4>Example 3:</h4><p>If the table is the only choice to visualize the dataset, what can you do to further improve the quality of the table as shown in the figure below?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/0*2svdoyu7eb-8CUa2" /><figcaption>Image Source: <a href="https://www.facebook.com/kementeriankesihatanmalaysia/photos/a.10151657414821237/10157472627206237/">Ministry of Health Malaysia</a></figcaption></figure><p>In my opinion, there are two suggestions that we can further improve this table, which are:</p><ol><li>Introducing the ranking index and locating it to the left of the “country” column.</li><li>Deriving a new feature named “percentage of Covid-19 cases by country”.</li></ol><p>The ranking index is crucial especially when the readers wish to know the ranking of certain countries that located in the middle parts of the table worldwide. Is it possible to tell the readers that what are the rankings of Singapore, Denmark or Malaysia in term of the number of Covid-19 cases worldwide within 5 seconds? The answer is definitely not due to the absence of the ranking index.</p><p>Referring to the table, the readers know that the US ranked the first worldwide in term of the number of Covid-19 cases. However, what is the percentage of Covid-19 cases contributed by the US worldwide? Again, this question cannot be answered directly without further calculations.</p><h4>Example 4 and 5:</h4><p>The table below shows the number of new Covid-19 cases across different states in Malaysia. After the discussion of some examples previously, are you able to give any suggestion to further improve this?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/0*57-oztxhezJJ1XLW" /><figcaption>Image Source: <a href="https://www.facebook.com/kementeriankesihatanmalaysia/photos/a.390879946236/10157472894056237/">Ministry of Health Malaysia</a></figcaption></figure><p>Compared to the previous example, the element of sorting is missing in this example. The states are sorted randomly which makes the readers spending more time to figure out which state contains the highest or lowest number of new Covid-19 cases. The readers cannot directly answer the question like which state ranks fifth in term of the number of Covid-19 cases in Malaysia.</p><p>Another example which contains the same mistake is shown below. The reporter does not take the element of sorting into consideration while preparing this informative content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*PuXtMpMxhpiMzx91" /><figcaption>Image Source: <a href="https://www.facebook.com/pocketimes/posts/2795446717380531">PocketTimes</a></figcaption></figure><h4><strong>Example 6:</strong></h4><p>The figure below shows the distributions of new Covid-19 cases in different districts of Selangor, Kuala Lumpur and Putrajaya.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nnYLWBt9v5Ny8frrYugxCQ.jpeg" /><figcaption>Image Source: <a href="https://www.facebook.com/ChinaPressCP/photos/3618565281569224">China Press</a></figcaption></figure><p>Compared to the charts in Example 4 and 5, the reporter really does a good job in preparing this informative content as he arranges the name of Selangor districts to be displayed according to the number of new Covid-19 cases.</p><p>Compared to the chart in Example 2, he separates the number of new Covid-19 cases between Selangor, Kuala Lumpur and Putrajaya before he further reports the number of new Covid-19 cases across different districts in Selangor to show the differences of status between states and federal territories.</p><p>For the information to non-Malaysia readers, Selangor is considered as one of the Malaysia states and Kuala Lumpur and Putrajaya are two out of three Malaysia federal territories. They should not be mixed while reporting the daily number of new Covid-19 cases.</p><p>Besides, the reporter clarified clearly in the title by mentioning that the informative content will cover the data of Selangor, Kuala Lumpur and Putrajaya.</p><h4>Example 7:</h4><p>The figure below shows the chart of the total number of Covid-19 cases by China and South East Asia countries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HUQAQUzDxT4xT7M_HVameg.png" /><figcaption>Image Source: <a href="https://www.facebook.com/100366473389140/videos/484512085843436">China Press</a></figcaption></figure><p>The chart used in this example, which is a stacked bar chart, is appropriate as the name of the countries is a nominal attribute and the total number of Covid-19 cases is a quantitative attribute. The countries containing the highest and lowest number of Covid-19 cases are ranked top and bottom respectively. The ordering is taken into consideration by the reporter while preparing this chart.</p><h4>Example 8:</h4><p>The figure below shows the summary of Covid-19 cases in Malaysia on 31st Oct 2020.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U7PkoOq5J34fpe-X" /><figcaption>Image Source: <a href="https://www.facebook.com/kementeriankesihatanmalaysia/photos/a.390879946236/10157472762626237/">Ministry of Health Malaysia</a></figcaption></figure><p>By reading this informative content, you may get the summary of Covid-19 cases quickly but it is not expressive and effective enough in the context of data visualization. There are a few questions for you:</p><ol><li>Is the relationship between the number of the total confirmed cases with the number of new cases same with the number of the total confirmed cases with the number of total death cases, total active cases and total recovered cases?</li><li>What is the relationship between new cases and active cases? Will the number of new cases contributes to active cases?</li></ol><p>In the construction of a mindmap or spider diagram, all the child nodes will always have the same relationship with the parent node. In this case, it is not at all.</p><p>Instead of using a mindmap, a tree diagram should be used to visualize the dataset expressively and effectively in this example.</p><p>According to the informative content above, we can summarize the rules as follows:</p><ol><li>Total Confirmed Cases = Total Death Cases + Total Active Cases + Toal Recovered Cases</li><li>New Death Cases is a subset of Total Death Cases</li><li>New Cases is a subset of Total Active Cases</li><li>New Recovered Cases is a subset of Total Recovered Cases</li></ol><p>By using these rules, we can construct a tree diagram easily as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0I-h5GB-1xtRq2K1GueWwg.png" /><figcaption>Figure: Self-Drawn Stack Tree Diagram</figcaption></figure><p>In the tree diagram, all the relationships are clearly defined and readers can understand it in a glance. The blue-coloured boxes represent the new cases (newly recovered, added or death cases).</p><h3><strong>Final words</strong></h3><p>The main objective of data visualization is to pass the messages to readers expressively and effectively. The priority of the beautifulness of the chart is put lower than the expressiveness and effectiveness of the chart.</p><h3>References:</h3><ol><li>Munzner, T. (2014). <em>Visualization analysis and design</em>. CRC press.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f0455474bf1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/vitrox-publication/evaluating-expressiveness-and-effectiveness-of-informative-charts-9f0455474bf1">Evaluating Expressiveness and Effectiveness of Informative Charts</a> was originally published in <a href="https://medium.com/vitrox-publication">ViTrox-Publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Calculate Manually: Deep Dive Into Logistic Regression]]></title>
            <link>https://medium.com/vitrox-publication/lets-calculate-manually-deep-dive-into-logistic-regression-7b357a350261?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b357a350261</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 04:26:24 GMT</pubDate>
            <atom:updated>2020-07-14T05:17:00.163Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn Machine Learning Through Calculation Example</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/0*ajbjjqK5dMXQeAeT" /><figcaption>Image: <a href="https://www.datasciencecentral.com/profiles/blogs/why-logistic-regression-should-be-the-last-thing-you-learn-when-b">Logistic Regression</a></figcaption></figure><h3>Motivation</h3><p>The algorithm of Logistic Regression has been well-explained by most of the machine learning experts through various platforms such as blogs, YouTube videos, online courses, etc. However, most of the time, they didn’t provide the calculation examples while they were explaining the algorithm. Imagine one day, if <a href="https://scikit-learn.org/stable/">scikit-learn</a> (a free machine learning library for Python) is taken away from your computing environment, can you still perform the process of modelling through the manual calculation especially on a simple dataset to get the similar result like calling the API of scikit-learn? If you can’t, this story is definitely for you.</p><h3>Brief Introduction to Logistic Regression</h3><h4><strong>What is Logistic Regression?</strong></h4><p>According to Ousley and Hefner (2005) and DiGangi and Hefner(2013), Logistic Regression is one of the statistical approaches that is similar to Linear Regression. Logistic Regression looks for the best equation to produce an output for a binary variable (<em>Y</em>) from one or multiple inputs (<em>X</em>). Linear Regression is capable to handle continuous inputs only whereas Logistic Regression can handle both continuous and categorical inputs. The mathematical concepts of log odds ratio and interactive maximum likelihood are implemented to find the best fit for group membership predictions. The assumption made by Logistic Regression includes all the inputs are independent of each other. Nevertheless, this assumption is not true for most of the times in reality [1, 2].</p><h4><strong>What is Multi-Class Logistic Regression?</strong></h4><p>It is known as Soft-max Regression which can handle the modelling process on the training dataset that contains more than 2 class labels.</p><h3>Challenge</h3><p>Given a set of 9 training data with 2-dimensional inputs and their corresponding class labels as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vqqw49574m_ZHPNQ_T8BIQ.png" /><figcaption>Table: Training Data</figcaption></figure><p>Can you:</p><ol><li>Convert the class labels into One-hot Representation?</li><li>Fit a Multi-Class Logistic Regression model to the training data using the algorithm of Gradient Descent? Provided that the learning rate is set to be 0.05, the number of training epoch is set to be 1 and the initial model parameters are set as follows.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TgNKRYTjYWFrxA-vwVm7FA.png" /><figcaption>Table: Initial Model Parameters</figcaption></figure><h3>Solution</h3><h4>Mathematical Alphanumeric Symbols</h4><p>To ease the display of mathematical equations, mathematicians use alphabets to represent variables. The alphabets will be used in this story include:</p><ul><li><em>N</em> = total number of training data (In this story, <em>N</em>=9)</li><li><em>K</em> = total number of class labels (In this story, <em>K</em>=3)</li><li>D = dimensions of training data (In this story, <em>D</em>=3) <br><strong>Special Note: </strong>D0 for all the inputs (<em>X</em>) normally will be assigned as the default value, which is 1.</li><li>α = learning rate</li><li><em>W</em> = model weight</li><li>W(new) = model weight before update</li><li>W(old) = model weight after update</li><li><em>X</em> = inputs</li><li><em>Y</em> = actual labels</li><li><em>^y</em> = predicted labels</li><li><em>J</em> = Total loss</li><li><em>L</em> = Loss for a particular number of training data</li></ul><h4>One-hot Encoding</h4><p>First and foremost, the class labels, which are integers, are encoded into One-hot Representation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*sefqc3U2HRgT09PMLhSkrw.png" /><figcaption>Image: The result of One-hot Encoding</figcaption></figure><h4>Inputs, Actual Labels and Weights Conversion Into Matrix Representation</h4><p>Next, the inputs (<em>X</em>), actual labels (<em>Y</em>) and initial weight parameters (<em>W</em>) are converted into matrices. The size of inputs, actual labels and weights should be 9x3, 9x3 and 3x3 for this story respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lidXOU5bcec2Em4Cvvp3CA.png" /><figcaption>Image: The input, output and initial weight parameters matrices.</figcaption></figure><h4>Feed-Forward Propagation</h4><p>Refer to <strong><em>Equation 1</em></strong> in the following image, the prediction matrix of the entire training dateset will be in the size of <em>N</em>x1, where <em>N</em> refers to the total number of training data. Refer to <strong><em>Equation 2</em></strong>, the prediction output of each training data will be the class label that contains the highest probability. The probability can be found by using <strong><em>Equation 3 </em></strong>[3].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DYNzP5UB_7yKhxl36EbEZg.png" /><figcaption>Image: The formula of Multi-Class Logistic Regression.</figcaption></figure><p>To compute the function (<em>f</em>), the inner product between <em>X</em> and <em>W</em> for different <em>k</em> should be obtained first. Then, the exponential of the inner product between <em>X</em> and <em>W</em> for different <em>k</em> is calculated. The output size of the matrix for the inner product between <em>X</em> and <em>W </em>for this story should be 9x3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u3vLf7ii72PU3jveRx8QCw.png" /></figure><p>The denominator of function (<em>f</em>) can be obtained by performing the sum of the exponential of the inner product between <em>X</em> and <em>W</em> for all <em>k</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*OULMdCjvU4Uj31C6EWkbHw.png" /></figure><p>As we all know, the division between matrices cannot be directly performed. However, there is a simple trick that can be done in this story, which directly performs the arithmetic division between matrices by using coordinates as a reference. For instances, matrix A got [<em>a1, a2, ……, a9</em>] and matrix B got [<em>b1, b2, ……, b9</em>], the matrix C where its <em>c1</em> value can be known by using <em>a1</em> divided by <em>b1</em>, <em>c2</em> value can be known by using <em>a2</em> divided by <em>b2</em> respectively. The size of the matrix after performed this trick remains to 9x1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*QsFvfw2iKZqgeu8WujkcPw.png" /></figure><p>The result of matrices for different <em>k</em> are combined and the output of the function (<em>f</em>) which is a matrix is found. The sum of the probability for predict all the class label for each training data will be always equal to 1 as the Soft-max function is applied above. The prediction output of each training data will be the class label that contains the highest probability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*4GcgDd1UPUJFKZT7Xj5QMA.png" /></figure><h4>Back Propagation</h4><p>The feed-forward propagation is temporarily done as the predicted classes for the training data are found for the first training epoch. How do we compute the loss? The loss can be figured out by using Multi-Category Cross Entropy. If the actual labels are not the same with the predicted labels, the loss value will be very big for that particular training data point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*a69pPvFOkwNRiIu0kpigjQ.png" /></figure><p>After the loss is computed, the weight parameters will be updated to make sure that the model will fit more the training data by using Gradient Descent. It will only update the weights once after all the training data have propagated through the model in a training epoch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*iQXsjubRldTN22GRY2EMiw.png" /></figure><p>After the weights are updated, the 1st training epoch is ended. You can continue to experiment it by performing the manual calculation for the next few training epoch and observe how the values of total loss and weight parameters change across the number of training epochs. The value of the total loss value should be reduced over training epochs if the model learns.</p><h3>Acknowledgement:</h3><p>A special thank to Lim Jia Hui, the Bachelor Degree of Biomedical Engineering’s student from University Technology Malaysia, providing me with the details of the challenge statement.</p><h3>References:</h3><p>[1] Ousley, S. D., &amp; Hefner, J. T. (2005). The statistical determination of ancestry. <em>Proceedings of the 57th annual meeting of the American Academy of Forensic Sciences</em> (pp. 21–26).</p><p>[2] DiGangi, E. A., &amp; Hefner, J. T. (2013). Ancestry estimation. <em>Research methods in human skeletal biology</em> (pp. 117–149). Academic Press.</p><p>[3] Raschka, S. (2019). <a href="http://rasbt.github.io/mlxtend/user_guide/classifier/SoftmaxRegression/#methods">Softmax Regression</a>. <em>mlxtend</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b357a350261" width="1" height="1" alt=""><hr><p><a href="https://medium.com/vitrox-publication/lets-calculate-manually-deep-dive-into-logistic-regression-7b357a350261">Let’s Calculate Manually: Deep Dive Into Logistic Regression</a> was originally published in <a href="https://medium.com/vitrox-publication">ViTrox-Publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Twitter Airline Sentiment Analysis]]></title>
            <link>https://medium.datadriveninvestor.com/twitter-airline-sentiment-analysis-3b147b565027?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b147b565027</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[social-media]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Sun, 19 Apr 2020 04:57:24 GMT</pubDate>
            <atom:updated>2020-04-20T10:28:23.868Z</atom:updated>
            <content:encoded><![CDATA[<h4>Text Mining from Descriptive Analysis to Predictive Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/0*1PYobPcnDWvQfxNP.jpg" /><figcaption>Image Source: <a href="https://www.dailymail.co.uk/travel/travel_news/article-3046994/A-bird-s-eye-view-Pilots-reveal-world-s-beautiful-cities-sit-best-vantage-point.html">Dailymail</a></figcaption></figure><p>According to <a href="https://www.iata.org/contentassets/a686ff624550453e8bf0c9b3f7f0ab26/wats-2019-mediakit.pdf">World Air Transport Statistic 2019</a>, the passenger demand showed robust growth. Compared to the year 2017, the industry-wide revenue passenger kilometres (RPKs) of the year 2018 is increased by 7.4%. China added the most number of passenger journeys and the Domestic United States remained the world’s largest single O-D passenger market in the year 2018. To improve revenue and profit, most of the airlines came out with attractive packages and high-quality services for their customers. Most of the decisions made are based on the fundamental of big data analytics. The skill to perform sentiment analysis on the data collected from various social media platforms is essential to be equipped by data scientists especially who aim to work in these airlines. This story demonstrates the step-by-step guide to analyze how travellers expressed their feelings on Twitter. The descriptive analysis is covered at the primer and the predictive analysis is covered at the latter.</p><h3>Prerequisite</h3><p>The dataset named “Twitter US Airline Sentiment” used in this story can be downloaded from <a href="https://www.kaggle.com/crowdflower/twitter-airline-sentiment">Kaggle</a>.</p><h3>Descriptive Analysis</h3><p>With the data provided in the dataset, some charts can be plotted to assist in descriptive analysis.</p><h4>Graph 1: The Count versus Type of Sentiment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*8Fiq5LKsOPH9Vue8MdB94A.png" /><figcaption>Figure: The count versus type of sentiments.</figcaption></figure><p>The dataset is imbalanced as it has more negative sentiments than positive and neutral sentiments. The number of negative sentiments is around 2.96 and 3.88 times the number of neutral and positive sentiments respectively. Nevertheless, it is common as <a href="https://www.nytimes.com/2012/03/24/your-money/why-people-remember-negative-events-more-than-positive-ones.html">people remember negative events more than positive events</a>.</p><h4>Graph 2: The Count of Sentiments versus Airlines grouped by Type of Sentiments</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*qofHOAdsv7bosYDQgSQ9CA.png" /><figcaption>Figure: The count of sentiments versus airlines grouped by type of sentiments.</figcaption></figure><p>Looking at the number of negative sentiments, United Airline ranks the first, followed by US Airways Airline and American Airline. The numbers of negative, neutral and positive sentiments for Virgin America Airline are almost the same to each other. In general, the number of negative sentiments is the most compared among the number of negative, neutral and positive sentiments for most of the airlines.</p><h4><strong>Graph 3: The Distributions of Sentiments among All The Negative Reasons</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*v6TiHO-udVWq8GPmMi6faQ.png" /><figcaption>Figure: The distributions of sentiments among all the negative reasons.</figcaption></figure><p>Most of the passengers (31.7%) give negative reviews because they do not satisfy to the quality of customer services provided by various airlines. 18.1% of the passengers have bad experience in the flights’ delay and 13.0% of them dislike the services provided by the airlines without providing any solid reason. The issue of damaged luggage is the least, which only occupies 0.8% of the entire pie chart.</p><h4>Graph 4: The distribution of Sentiments versus Negative Reasons grouped by Airlines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*jtdIHK71pTckERT-_02_kg.png" /><figcaption>Figure: The distribution of sentiments versus negative reasons grouped by airlines.</figcaption></figure><p>The issue of customer service is always ranked the top for most of the airlines except Delta Airline. Delta Airline is recommended to deal with the issue of flights’ delay before solving other problems.</p><h4>Graph 5: The Count of Negative Sentiments versus Airlines grouped by Date</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SrHsodgBjHj19nobPAcT_g.png" /><figcaption>Figure: The count of negative sentiments versus airlines grouped by date</figcaption></figure><p>United Airline ranks the first in term of the number of the negative sentiments starting from the date of 17th February to 21st February of year 2015. The number of negative sentiments for the date of 22nd February 2015 onward is led by American Airline. The number of negative reviews for Delta Airline is slowly decreased in the last 3 days ended with the date of 24th February 2015.</p><h3>Predictive Analysis</h3><p>With the labels provided on the sentiments, predictive analysis can be done by building a model to predict the type of sentiments, which are either “positive”, “neutral” or “negative”. The overall flow includes:</p><h4>Pre-processing</h4><p>Some required columns are selected and some common text processing algorithms are performed to:</p><ul><li>Remove empty reviews.</li><li>Convert all the reviews to lower case.</li><li>Remove numbers, tweet account names, website urls, special characters and white spaces.</li></ul><h4>Which Text Normalizer to be Used?</h4><p>The dataset is processed using different lemmatizers and stemmers are modelled with Random Forest with the aid of TF-IDF Vectorizer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*-9cEi-pkoA1gCalTZnV1tw.png" /><figcaption>Figure: The accuracy scores of the Random Forest trained by dataset processed using different lemmatizers and stemmers.</figcaption></figure><p>The model that is trained by the dataset processed using Porter Stemmer has the highest accuracy compared to the dataset processed using WordNet Lemmatizer with POS tag, WordNet Lemmatizer without POS tag and Snowball Stemmer. In the later process, Porter Stemmer is chosen to be used as the text normalizer.</p><p><a href="https://www.datadriveninvestor.com/2019/01/23/which-is-more-promising-data-science-or-software-engineering/">Which is More Promising: Data Science or Software Engineering? | Data Driven Investor</a></p><h4>Should I Augment and Up-sample Dataset?</h4><p>The texts in the minor classes are tried to be augmented and up-sampled to solve the issue of imbalance dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*--IJ8rXXOB7VW8wLL3Pl_g.png" /><figcaption>Figure: The accuracy scores of the Random Forest trained by augmented and non-augmented datasets.</figcaption></figure><p>The result shows that the augmented dataset helps in increasing the model accuracy significantly. In the later process, augmented dataset is chosen to be used for the input of the modelling.</p><h4>Should I Use Deep Learning Algorithm?</h4><p>Instead of using machine learning algorithm like Random Forest, simple deep learning algorithms are tried out in this project as well including self-designed Simple Neural Network (one embedding layer + one dense layer), self-designed Recurrent Neural Network (RNN) and self-designed Convolutional Neural Network (CNN).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*fbcXat7boI63KDQQeX5zdQ.png" /><figcaption>Figure: The accuracy scores of the models trained by different algorithms using augmented dataset.</figcaption></figure><p>Convolutional Neural Network has the highest accuracy score, followed by Simple Neural Network, Random Forest, and lastly Recurrent Neural Network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*KLCZgZbQW7Zc7EKx7YWv0w.png" /><figcaption>Figure: The graph of training and test loss for CNN.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*Q5cJsNoQMTjFhJzyZYUxNg.png" /><figcaption>Figure: The graph of training and test accuracy for CNN.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*mHOqZ8z3LNZiSDFQVTqiCA.png" /><figcaption>Figure: The classification report of the model trained by using CNN.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*Uw_R1M5XQyUtKPWw8BlCyg.png" /><figcaption>Figure: The confusion matrix of the model trained by using CNN.</figcaption></figure><h3>Extra: Word Clouds</h3><p>The meaningful words that can be spotted in the negative sentiments’ word cloud directly are “delay” and “cancel flight”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*GJKRvEH8_ye_ZyjnXiafDg.png" /><figcaption>Figure: The negative sentiments’ word cloud.</figcaption></figure><p>On the other hand, the meaningful words that can be spotted in the positive sentiments’ word cloud directly include “thank”, “best”, “great” and “awesome”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*-0QHMnpUhUhtFpodb2OQ6g.png" /><figcaption>Figure: The positive sentiments’ word cloud.</figcaption></figure><p>In overall, it seems that both word clouds display a lot of non-meaningful words at the same time as well like “thi”, “help”, “time”, “today”, “will”, etc. <strong>The current stopword filter is not really well-designed</strong>.</p><p>In addition, some of the words are not in the correct spellings such as “thi” (original word should be “this”) and “amaz” (original word should be “amaze”) due to <strong>the output of Porter Stemming is fed into the word cloud</strong> for visualization in this project.</p><h3>Available Code</h3><p>Download and open it if you can’t read the html file directly.</p><p><a href="https://drive.google.com/file/d/1HXhC1e3PFb19J4hr0hFEODazxasHxw_l/view?usp=sharing">Sentiment_Analysis.html</a></p><h3>Acknowledgement</h3><p>This project was contributed by Tan Pei Seng and Koay Yee Shean for CDS 522 Text and Speech Analytics, one of the elective subjects under Master of Data Science and Analytics in Universiti Sains Malaysia.</p><h3>References</h3><ol><li><strong>Twitter US Airline Sentiment</strong>: <a href="https://www.kaggle.com/crowdflower/twitter-airline-sentiment/kernels">https://www.kaggle.com/crowdflower/twitter-airline-sentiment/kernels</a></li><li><strong>World Air Transport Statistic 2019</strong>: <a href="https://www.iata.org/contentassets/a686ff624550453e8bf0c9b3f7f0ab26/wats-2019-mediakit.pdf">https://www.iata.org/contentassets/a686ff624550453e8bf0c9b3f7f0ab26/wats-2019-mediakit.pdf</a></li><li><strong>Praise Is Fleeting, but Brickbats We Recall</strong>: <a href="https://www.nytimes.com/2012/03/24/your-money/why-people-remember-negative-events-more-than-positive-ones.html">https://www.nytimes.com/2012/03/24/your-money/why-people-remember-negative-events-more-than-positive-ones.html</a></li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b147b565027" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/twitter-airline-sentiment-analysis-3b147b565027">Twitter Airline Sentiment Analysis</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[COVID-19 Detection in X-ray Images with Pytorch]]></title>
            <link>https://medium.datadriveninvestor.com/covid-19-detection-in-x-ray-images-with-pytorch-5c5602b4658f?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c5602b4658f</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[medical]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[image]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Fri, 27 Mar 2020 11:48:44 GMT</pubDate>
            <atom:updated>2020-03-30T14:57:55.673Z</atom:updated>
            <content:encoded><![CDATA[<h4>Artificial Intelligence (AI) in Radio-diagnosis and Imaging</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*ffCnI1dmkREAQGxAEUGSRg.png" /><figcaption>Pic: Which lungs belong to COVID-19 patients?</figcaption></figure><p>Yesterday, I was asked by people surroundings about a question: “Is it possible for us to predict COVID-19 using X-ray images? I immediately surfed through the internet and found out that there are a few groups of people over the world are working hard on it.</p><ul><li>A group of China researchers published a paper on “arxiv.org” with the title named “<a href="https://arxiv.org/ftp/arxiv/papers/2002/2002.09334.pdf">Deep Learning System to Screen Coronavirus Disease 2019 Pneumonia</a>”. Their deep learning model is capable to achieve 86.7% of overall accuracy for classifying the CT images into COVID-19, Influenza-A Viral Pneumonia or healthy cases.</li><li>Joseph Paul Cohen, Paul Morrison and Lan Dao are trying to build an <a href="https://github.com/ieee8023/covid-chestxray-dataset?fbclid=IwAR3yCPo_e55khIvhhqDdPhCI6OMCLXjZKEziNnUzZWC9_h3NkyhqdjTqD5c">open database</a> of COVID-19, MERS, SARS, and ARDS cases with chest X-ray or CT images.</li><li>Adrian Rosebrock published a tutorial on “pyimagesearch” in demonstrating the step by step guide to <a href="https://www.pyimagesearch.com/2020/03/16/detecting-covid-19-in-x-ray-images-with-keras-tensorflow-and-deep-learning/">detect COVID-19 in X-ray images with Keras, TensorFlow, and Deep Learning</a>. He managed to achieve 90% of model overall accuracy for classifying between COVID-19 and Non-COVID-19 X-Ray Images.</li></ul><p>The contribution of this story includes the demonstration of the approaches that I have implemented to achieve 100% of the overall model accuracy trained by the image dataset provided by Adrian Rosebrock using Pytorch, a deep learning framework.</p><h3>Download the dataset</h3><p>Firstly, I downloaded the dataset from the Adrian’s tutorial (<a href="https://www.pyimagesearch.com/2020/03/16/detecting-covid-19-in-x-ray-images-with-keras-tensorflow-and-deep-learning/">https://www.pyimagesearch.com/2020/03/16/detecting-covid-19-in-x-ray-images-with-keras-tensorflow-and-deep-learning/</a>) on “pyimagesearch” by inputting my email address to download both dataset and source codes. Next, I downloaded them through the link that is sent to my email from “pyimagesearch”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*a3Oqi1SI6RcxUbYSZNNqvg.png" /><figcaption>Pic: Key in your email address to get the dataset.</figcaption></figure><p>In the folder named “dataset”, there are two sub-folder named “covid” and “normal”. Each of the sub-folder contains 25 images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vP1IDPNJl3G3UZ31SX9FOA.png" /><figcaption>Pic: The images that are stored in “normal” sub-folder.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z5-t3sgqjjzGNh667nDLrg.png" /><figcaption>Pic: The images that are stored in “covid” sub-folder.</figcaption></figure><h3>Import the Libraries</h3><p>The libraries that are used in this project include Torch, Torchvision, Sklearn, PIL, Matplotlib, Numpy, Os, Time and Random.</p><pre># Import essential libraries<br>from sklearn.metrics import accuracy_score, classification_report, confusion_matrix<br>from sklearn.model_selection import train_test_split<br>from PIL import Image<br>import torchvision<br>import matplotlib.pyplot as plt<br>import numpy as np<br>import torch<br>import os<br>import time<br>import random<br>%matplotlib inline</pre><h3>Prepare the Dataset</h3><p>I prepared my dataset by storing all the name and label of the images as the lists with the variable names “img_name” and “label” respectively.</p><pre>path=&quot;dataset/&quot;<br>img_name=[]<br>label=[]<br>for root, directories, files in os.walk(path):<br>    for file in files:<br>        img_name.append(root+&quot;/&quot;+file)<br>        label.append(root.split(&quot;/&quot;)[-1])</pre><h3>Study the dataset</h3><p>By observing the dataset, I got some information:</p><ol><li>All the images are not in same size, meaning they don’t share the same values in height and width.</li><li>All the images are not in 1: 1 aspect ratio. If we resize the images to fixed ratio (224px width and 224px height), it might lead to the drop of the overall modal accuracy.</li><li>All the images are not in same zooming ratio. Some lungs look big and some lungs look small in the images.</li></ol><p>I drew a boxplot on the distribution of heights and widths of the images over the entire image dataset. Label “1” represents the widths and Label “2” represents the heights. There are some outliers as observed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*1z9pvCT1lT0wjKqauKgy0Q.png" /><figcaption>Pic: The boxplot of widths and heights of all the images in the dataset.</figcaption></figure><h3><strong>Split the dataset</strong></h3><p>I split the dataset into training and testing with test_size = 0.2, random_state=50 and stratification using Sklearn library. The value of the random_state is free to be changed to observe whether the results obtained by using different values of random_state are consistent or not.</p><pre>train_img_name, val_img_name, train_label, val_label = train_test_split(img_name, label, test_size=0.2, random_state=50, stratify=label)</pre><h3>Augment the Dataset</h3><p>In the tutorial prepared by Adrian, he augmented the images by implementing the settings of random_rotation=15 degree only. Besides random_rotation, I included the settings of random_zooming_ratio to be fixed in the range from 0.7 to 1.3 to ensure that the classifier is more robust by reducing the effect of different zooming ratio among all the images.</p><pre>train_Aug = torchvision.transforms.Compose([torchvision.transforms.Resize((224, 224)),                                 torchvision.transforms.RandomRotation((-20, 20)),                                          torchvision.transforms.RandomAffine(0, translate=None, scale=[0.7, 1.3], shear=None, resample=False, fillcolor=0),                                            torchvision.transforms.ToTensor()])<br>test_Aug = torchvision.transforms.Compose([torchvision.transforms.Resize((224, 224)),                                           torchvision.transforms.ToTensor()])</pre><h3>Prepare the Data Loaders</h3><p>I prepared two data loaders, which are train loader that loads the training images and test loader that loads the testing images.</p><pre>class CustomDatasetFromImages(torch.utils.data.Dataset):<br>    def __init__(self, img_name, label, transforms=None): <br>        self.image_arr = np.asarray(img_name)<br>        self.label_arr = np.asarray(label)<br>        self.data_len = len(img_name)<br>        self.transforms = transforms<br>    def __getitem__(self, index):<br>        single_img_name = self.image_arr[index]<br>        img_array = Image.open(single_img_name).convert(&#39;RGB&#39;)<br>        if self.transforms is not None:<br>            img_array = self.transforms(img_array)<br>        image_label = self.label_arr[index]<br>        return (single_img_name, img_array, image_label)<br>    def __len__(self):<br>        return self.data_len</pre><pre>train_set=CustomDatasetFromImages(train_img_name, train_label, transforms=train_Aug)<br>test_set=CustomDatasetFromImages(val_img_name, val_label, transforms=test_Aug)<br>trainloader= torch.utils.data.DataLoader(train_set, batch_size=64, num_workers=2,shuffle=True) <br>testloader= torch.utils.data.DataLoader(test_set, batch_size=64, num_workers=2, shuffle=False)</pre><h3>Initialize the Model Architecture</h3><p>I started to model the images with ResNet18 because the images are only in black and white. If ResNet18 is proved to be under-fitted, only deeper networks will be considered to be introduced in the later experiment. I used default pre-trained weight provided by Pytorch.</p><pre>model = torchvision.models.resnet18(pretrained=True)</pre><p>I changed the number of the output node of the last layer of ResNet 18 to be 2.</p><pre>model.fc = torch.nn.Linear(512, len(unique_class))</pre><h3>Configure the Hyperparamters</h3><p>The lost function that I used is Cross Entropy Loss.</p><pre>criterion = torch.nn.CrossEntropyLoss()</pre><p>The optimizer that I used is Adam with initial learning rate=0.001 and weight decay=0.0005.</p><pre>optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.0005)</pre><p>The learning rate scheduler that I used is ReduceLROnPlateau with factor=0.1 and patience=10. It means that if the training loss is not decreased anymore continuously for 10 epoch during the process of training, the learning rate will be changed to the 0.1 times of its original value. i.e: Before change is made = 0.001 and After change is made=0.0001.</p><pre>scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor=0.1, patience=10, eps=1e-06)</pre><h3>Train and Evaluate the model</h3><p>After the model architecture and all the hyper-parameters are defined, The model now can be trained with training images and evaluated with testing images. The total number of epoch that I set is 50. The model that contains the model weights that achieves the lowest validation loss followed by the highest validation accuracy is saved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*odIImS4C9Fhd6-jJJ-uQxw.png" /><figcaption>Pic: The Training Loss, Training Accuracy, Testing Loss and Testing Accuracy in Epoch 1–5.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*4xVy_ZL0N7AF39VqadxabA.png" /><figcaption>Pic: The Training Loss, Training Accuracy, Testing Loss and Testing Accuracy in Epoch 26–30.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*ozOd5M05g6um6iUsYToD2A.png" /><figcaption>Pic: The Training Loss, Training Accuracy, Testing Loss and Testing Accuracy in Epoch 46–50.</figcaption></figure><h3>Discussion</h3><p>The model that contains the best result is from Epoch 50. The accuracy score, classification report and confusion matrix are shown in the figure below. The time taken for entire modeling process is 95.91s. The modelling process is running on high specification’s GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*anJCRGx9nFHCW-V-Hhaq5A.png" /><figcaption>Fig: The best result (from epoch-50).</figcaption></figure><p>The result of the 10 testing images are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*lGlUHcGIQbns90EkUzNj0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*ZTUkZMPJHFR5vQCoRodmKg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*XzWJlYmkh8sSJfqYu5naXA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*qKkyxjsy9MTOi3F5-UbnaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*Xgu0lDkTrrxkNMKqWo6ETA.png" /><figcaption>Pic: The result of the 10 testing images.</figcaption></figure><h3>Final Words:</h3><p>With the addition of the data augmentation settings of random_zooming_ratio, the model is managed to be improved from 90% (9 images are predicted correctly, have been done by Adrian) to 100% (10 images are predicted correctly).</p><p><a href="https://www.datadriveninvestor.com/2020/02/12/has-general-ai-exceeded-the-intellectual-capacity-of-humans/">Has General AI Exceeded the Intellectual Capacity of Humans? | Data Driven Investor</a></p><h3><strong>References:</strong></h3><ul><li>Adrian Rosebrock, <a href="https://www.pyimagesearch.com/2020/03/16/detecting-covid-19-in-x-ray-images-with-keras-tensorflow-and-deep-learning/">Detecting COVID-19 in X-ray images with Keras, TensorFlow, and Deep Learning</a>.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c5602b4658f" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/covid-19-detection-in-x-ray-images-with-pytorch-5c5602b4658f">COVID-19 Detection in X-ray Images with Pytorch</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[7 Ways To Derive New Features From Existing Tabular Attributes]]></title>
            <link>https://medium.datadriveninvestor.com/7-ways-to-derive-new-features-from-existing-tabular-attributes-3a063c5aa807?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a063c5aa807</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Sun, 08 Mar 2020 16:46:00 GMT</pubDate>
            <atom:updated>2020-03-09T06:07:21.268Z</atom:updated>
            <content:encoded><![CDATA[<h4>Convert and Transform insignificant columns into useful representations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KrG3eo98PliDkI82IltGQ.jpeg" /><figcaption>Image taken by Pei Seng at Watsons Bay, Sydney, Australia</figcaption></figure><p>In most of the time, the data collected from the customer sides are not significant and modular enough to be directly applied in modelling and visualization. As mentioned by most of the data scientists, data pre-processing is always the main key to determine the success in descriptive, diagnostic and predictive analysis. Instead of performing feature selection or extraction, the following will be discussing more on how to explore and create new possible useful features to increase the chance of success in modelling and visualization.</p><h3>Method 1: Perform Arithmetic Operations on Quantitative Attributes</h3><p>The most common approach to derive new features from existing attributes is to perform simple arithmetic operations between any quantitative attributes like unit price, profit, weight or height. The examples of the new attributes that you can create include:</p><ol><li>The age (derived attribute) is equivalent to the current year (original attribute) subtracted by the year of birth (original attribute).</li><li>The estimated amount of company A annual sales (derived attribute) is equivalent to the average sales of the first month (derived attribute) multiplied by the 12.</li><li>Body Mass Index (derived attribute) is equivalent to the weight in kg (original attribute) divided by the square of the height in meter (original attribute).</li></ol><p>Although the original dataset might not provide the information that you desire, you can still proceed by deriving them using other quantitative attributes that you have obtained. For an instance, with provided weight and height information, you still can get the value of Body Mass Index although it is not in the original dataset. The drawback of this approach includes it is useless when it is applied to categorical or ordinal attributes as the results of arithmetic operations applied on categorical and ordinal attributes are not meaningful.</p><p><a href="https://www.datadriveninvestor.com/2019/01/23/which-is-more-promising-data-science-or-software-engineering/">Which is More Promising: Data Science or Software Engineering? | Data Driven Investor</a></p><h3>Method 2: Perform Logarithmic Transformation on Skewed Normal Distribution</h3><p>The normal distribution is widely applied in the understanding distributions of factors in the population because the normal distribution approximates most of the natural phenomena well. The real-life examples of normal distributions include weight, height, IQ, income distributions in the economy, shoe sizes, etc. Due to the bias in collecting small samples from a large population, the distribution of data collected tends to be skewed to the left or right as shown in Figure 2. Is there any approach to transform the skewed distribution into a normal distribution?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*1MsZkztVNFGljJp7iRalyg.png" /><figcaption>Figure 1: Real-life Examples of Normal Distributions: (a) IQ Score (b) Female Shoe Sales versus Size</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/0*7didSPJCh9aJV07D.png" /><figcaption>Figure 2: The Distribution Graph Before and After of Logarithmic Transformation</figcaption></figure><p>The answer is definitely yes, by performing Logarithmic transformation. After the Logarithmic transformation is performed, the distribution will entirely change to normal but the original values of the dataset will be manipulated as well. This approach is meaningful for modelling but not for visualization if the original values are required to be preserved.</p><h3>Method 3: Derive Unique Identifiers to Obtain Hierarchy Information</h3><p>If you have a column of data where each of the value is unique to each other, are you going to drop them? The answer definitely depends on the situation. The examples of Unique Identifiers include customer ID, Product ID, Passenger ID, etc. In most of the time, instead of dropping them, they can be further derived into more useful information like product category if the customers do not provide such information which is shown in Figure 3. Referring to Figure 4, more hidden information is uncovered if the right derivation technique is implemented on the unique identifiers. The product ID indirectly provides the information of hierarchy, which means under product category S, there are S1, S2, S3, S4 and S5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*28lOl5wEYNmW0SzGweGkcw.png" /><figcaption>Figure 3: The new attribute named Product Category is derived from original attributes named Product ID and Count</figcaption></figure><p><strong><em>Note:</em></strong><em> Yellow highlighted cells are original attributes, blue highlighted cells are derived attributes.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0liCiJJhcI7VaXLvd8WeYw.png" /><figcaption>Figure 4: The hidden information about the count for each product category is uncovered</figcaption></figure><h3>Method 4: Expand Date Attributes</h3><p>With a series of date values, they can be further derived into years, quarters, months, weeks, days, hours, minutes or seconds, as shown in Figure 5. The derived attributes can be used as the grouping clue to cluster the data that shares the same characteristic like same month, week or day, which provides more features for modelling and visualization. It can also be indexed for performing time series analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*8jYIalMGrXqn3yaTMfYZfA.png" /><figcaption>Figure 5: Derive date attributes into years, quarters, months….</figcaption></figure><p><strong><em>Note:</em></strong><em> Yellow highlighted cells are original attributes, blue highlighted cells are derived attributes.</em></p><h3>Method 5: Convert Numerical Attributes to Categorical Attributes</h3><p>Instead of keeping the attributes types to default, you are free to convert the attributes into other types. The most common practice is to convert the nominal attributes into categorical attributes. One of the very good examples is human age, where people commonly will record this attribute as the type of numeric. A specific range can be defined to group the values up. For instances, as shown in Figure 6, 0 to 5 years old kids are grouped as Group 1, 6 to 10 years old kids are grouped as Group 2, and so on. Data Analysts applied this technique a lot to convert daily defected count to weekly or monthly defected count, which is shown in Figure 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*PWcBhLu_YvsM_CIB_tC2Qg.png" /><figcaption>Figure 6: Derive new attributes named Age Group and Average IQ Scores from original attributes named Age and IQ Scores</figcaption></figure><p><strong><em>Note:</em></strong><em> Yellow highlighted cells are original attributes, blue highlighted cells are derived attributes.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ry_7BR8M2vu7U302RSbqTA.png" /><figcaption>Figure 7: The graph of Average IQ Scores versus Age Group can be plotted by using Age and IQ Scores.</figcaption></figure><h3>Method 6: Convert Discrete Attributes to Continuous Attributes</h3><p>By providing a few discrete sampling data points, is it possible to join them smoothly into a continuous curve? The answer is definitely yes again, and this type of conversion is known as interpolation, where the additional data points will be created in between original data points to form a continuous line curve. The common type of interpolation for 1 dimensional array describing 1 attribute include Nearest-Neighbor Interpolation (as shown in Figure 8), Linear Interpolation (as shown in Figure 9) and Polynomial Interpolation (as shown in Figure 10).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1nWNKska7TXwBKtU.png" /><figcaption>Figure 8: Nearest-Neighbor Interpolation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AF0yXDa9obZPZgiP.png" /><figcaption>Figure 9: Linear Interpolation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o4DYkoLw6uAUKyNC.png" /><figcaption>Figure 10: Polynomial Interpolation</figcaption></figure><h3><strong>Method 7: Add on Additional Related Data</strong></h3><p>If you think that other attributes from additional dataset are useful, you are welcome to import and join it with current dataset. Most of the data analysts visualized the location of cities by taking the coordinates (latitude and longitude) of cities from additional dataset. Besides geographical location, the customer demography dataset is always imported and joined with transaction data to study the customer buying behaviors. Like the example shown in Figure 11, besides the cases by date of report is shown, the coordinates of the cities are imported to visualize the total number of Coronavirus for each country.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*oJq4Tagy30DmCfzl" /><figcaption>Figure 11: The graph of Coronavirus cases by dates and cities.</figcaption></figure><h3>References:</h3><ol><li>Book: Visualization Analysis and Design <a href="https://www.cs.ubc.ca/~tmm/private/book/book.120803.pdf">https://www.cs.ubc.ca/~tmm/private/book/book.120803.pdf</a></li><li>9 Real Life Examples Of Normal Distribution <a href="https://studiousguy.com/real-life-examples-normal-distribution/">https://studiousguy.com/real-life-examples-normal-distribution/</a></li><li>Interpolation <a href="https://en.wikipedia.org/wiki/Interpolation">https://en.wikipedia.org/wiki/Interpolation</a></li><li>Types of Data in Statistics — Nominal, Ordinal, Interval, and Ratio Data Types Explained with Examples <a href="https://www.freecodecamp.org/news/types-of-data-in-statistics-nominal-ordinal-interval-and-ratio-data-types-explained-with-examples/">https://www.freecodecamp.org/news/types-of-data-in-statistics-nominal-ordinal-interval-and-ratio-data-types-explained-with-examples/</a></li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a063c5aa807" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/7-ways-to-derive-new-features-from-existing-tabular-attributes-3a063c5aa807">7 Ways To Derive New Features From Existing Tabular Attributes</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Named Pipes Communication between Python Server and Python Client on Window]]></title>
            <link>https://medium.datadriveninvestor.com/named-pipes-communication-between-python-server-and-python-client-on-window-8cdf64504801?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cdf64504801</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[windows-10]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Wed, 29 Jan 2020 03:11:49 GMT</pubDate>
            <atom:updated>2020-02-02T19:43:20.911Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to used a named pipe for inter-processes communication?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/0*dbp1rh4KAx-VMFzO" /><figcaption>Photo by Pei Seng Tan at Queen Victoria Building</figcaption></figure><p>According to Wikipedia, a named pipe is defined as one of the approaches commonly used to perform inter-process communication (IPC), where the processes can share the data easily between them. A pipe is treated as a file and the processes are allowed to write to or read from a named pipe. Up to today, it is able to be implemented on both Linux and Window operating systems. In this example, a Python Server and a Python Client will be created to demonstrate how to perform a named pipe communication between them on Window using Python-based Win32 API.</p><h3>Python Named Pipe Server</h3><p>The APIs that will be used to create Python Named Pipe Server include:</p><ul><li><strong>win32pipe.CreateNamedPipe:</strong> In order to use the named pipe for IPC, firstly, we need to initialize a named pipe by assigning a specific name to it. It will create an instance of a named pipe and returns a handle for subsequent pipe operations. The string must follow the format of “\.\pipe&lt;i&gt;pipename” where the pipename is set to “ABC” in this example. Other parameters likes openMode, pipeMode, nMaxInstances, nOutBufferSize, nInBufferSize, nDefaultTimeOut and lpSecurityAttributes can be set to default or according to your needs. The openMode is set to be “ePIPE_ACCESS_DUPLEX” indicates that both server and client processes are allowed to read from and write to the named pipe. The pipeMode is set to be “PIPE_TYPE_MESSAGE” meaning that the data will be written to the named pipe as a stream of messages or “PIPE_READMODE_MESSAGE” allowing the data to be read from the named pipe as a stream of messages or “PIPE_WAIT” turning on the blocking mode.</li></ul><pre>pipe = win32pipe.CreateNamedPipe(r&#39;\\.\pipe\ABC&#39;, win32pipe.PIPE_ACCESS_DUPLEX, win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_READMODE_MESSAGE | win32pipe.PIPE_WAIT, 1, 65536, 65536, 0, None)</pre><ul><li><strong>win32pipe.ConnectNamedPipe: </strong>After the named pipe is initialized, we need to connect the server to the client through the named pipe. This function enables the named pipe process to wait for the client to connect to the server.</li></ul><pre>win32pipe.ConnectNamedPipe(pipe, None)</pre><ul><li><strong>win32file.WriteFile: </strong>After the client is connected to the server through the named pipe, the server now can send the data to the client. This function will write multiple encoded strings to the client through the named pipe. The function will return two values, where the primer represents the error code and the latter represents the number of bytes sent. The return value of the error code will be zero if the function fails to perform or the process is completely asynchronously. In this example, the processes of client and server are asynchronous. Therefore, the value stored in “err” variable will always zero in value.</li></ul><pre>err, bytes_written=win32file.WriteFile(pipe, some_data)</pre><ul><li><strong>win32file.FlushFileBuffers:</strong> The buffers will be flushed so all the buffered data will be written to the named pipe. By calling this function, we can make sure that all the messages are received by the client.</li></ul><pre>win32file.FlushFileBuffers(pipe)</pre><ul><li><strong>win32pipe.DisconnectNamedPipe:</strong> The connection between the server and the named pipe is disconnected. It may lead to the client handle invalid if the connection between the client and the named pipe is still under connection. Additional work is required to handle this error at the client-side.</li></ul><pre>win32pipe.DisconnectNamedPipe(pipe)</pre><ul><li><strong>win32pipe.CloseHandle:</strong> Lastly, the handle to the pipe instance is closed.</li></ul><pre>win32file.CloseHandle(pipe)</pre><h3>Python Named Pipe Client</h3><p>The APIs that will be used to create Python Named Pipe Client include:</p><ul><li><strong>win32file.CreateFile:</strong> After the server initialized the named pipe, the client will be required to connect to the server through the named pipe using this function or another naming as “CallNamedPipe”. If a pipe instance is available, a handle will be returned back to the client. The dwDesiredAccess is set to be either “GENERIC_READ” or “GENERIC_WRITE”, which allows the client to read from or write to the named pipe. The parameter of dwCreationDisposition is set to be “OPEN_EXISTING” indicates that the named pipe will be opened if exits. The parameter of dwFlagsAndAttributes is set to be “FILE_ATTRIBUTE_NORMAL” because the named pipe does not have other attributes set.</li></ul><pre>handle = win32file.CreateFile(r&#39;\\.\pipe\ABC&#39;,win32file.GENERIC_READ | win32file.GENERIC_WRITE, 0, None, win32file.OPEN_EXISTING, win32file.FILE_ATTRIBUTE_NORMAL, None)</pre><ul><li><strong>win32pipe.SetNamedPipeHandleState: </strong>After the client is connected to the server through the named pipe, we need to set to read mode in order to receive messages from the server. The “lpMode” is set to be “PIPE_READMODE_MESSAGE” so the data will be read from the named pipe as a stream of messages. You will encounter failure if the client is connected to the byte-type pipe instead of the message-type pipe. In this example, the message-type pipe is used.</li></ul><pre>res = win32pipe.SetNamedPipeHandleState(handle, win32pipe.PIPE_READMODE_MESSAGE, None, None)</pre><ul><li><strong>win32file.ReadFile:</strong> After the read mode is set, the client is ready now to read the data from the named pipe. A sequence of messages will be read until the connection between the server and the named pipe or the client and the named pipe is disconnected.</li></ul><pre>resp = win32file.ReadFile(handle, 65536)</pre><h3>Sample Application: Send Data from Python Server to Python Client</h3><p>To know more details on how to turn the above mentioned APIs into a simple application, you may refer to the github link below. In overall, the Python Server will send 10 lines of string encoded with the date time information to the Python Client at the period of 0.2 seconds continuously for 3 cycle. Figure 1 shows the console from the Python Server and Figure 2 shows the console from the Python Client.</p><p><a href="https://github.com/PeiSeng/Named_Pipe_Communication">PeiSeng/Named_Pipe_Communication</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*xUl6foPk_1cexNvgeaYuWA.png" /><figcaption>Figure 1: Console of Python Server</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*TdvOTfCTbo1Yng9N8hMrOw.png" /><figcaption>Figure 2: Console of Python Client</figcaption></figure><h3>References</h3><ol><li>Inter-process communication: <a href="https://en.wikipedia.org/wiki/Inter-process_communication">https://en.wikipedia.org/wiki/Inter-process_communication</a></li><li>Python and Windows Named Pipes: <a href="https://stackoverflow.com/questions/48542644/python-and-windows-named-pipes">https://stackoverflow.com/questions/48542644/python-and-windows-named-pipes</a></li><li>Named Pipe Operations: <a href="https://docs.microsoft.com/en-us/windows/win32/ipc/named-pipe-operations">https://docs.microsoft.com/en-us/windows/win32/ipc/named-pipe-operations</a></li><li>Python for Win32 Extensions Help <a href="http://timgolden.me.uk/pywin32-docs/contents.html">http://timgolden.me.uk/pywin32-docs/contents.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cdf64504801" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/named-pipes-communication-between-python-server-and-python-client-on-window-8cdf64504801">Named Pipes Communication between Python Server and Python Client on Window</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speech Analysis of Vowels for Japanese Language]]></title>
            <link>https://medium.com/data-science/speech-analysis-of-vowels-for-japanese-language-fdee6ec1a4ba?source=rss-7835637487b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdee6ec1a4ba</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[language]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[japanese]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Pei Seng Tan]]></dc:creator>
            <pubDate>Sat, 28 Dec 2019 03:51:38 GMT</pubDate>
            <atom:updated>2022-01-18T02:59:04.457Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Analytics</h4><h3>Acoustic &amp; Articulatory Analysis on Japanese Vowels</h3><h4>Did you know how the Japanese Vowel Chart is plotted?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eh6bCilKSt7qs8GX" /><figcaption>Photo by <a href="https://unsplash.com/@miguel_photo?utm_source=medium&amp;utm_medium=referral">Miguel Henriques</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In order to build a <strong>speech recognizer</strong> or <strong>speech to text converter</strong> for <strong>English</strong> or <strong>Japanese</strong>, typically you need three components, which are an <strong>acoustic model</strong>, a <strong>lexicon model</strong> and a <strong>language model</strong> to strike a success.</p><ol><li>An <strong>acoustic model</strong> converts the sound signal into phonemes.</li><li>A <strong>lexicon model</strong> maps the phonemes into words.</li><li>A <strong>language model</strong> restricts the word search. For example, predict the next words by using previous words.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*FKBajbl83Tz0Uz7cINq6UQ.png" /><figcaption>Figure: Overview of a speech recognizer.</figcaption></figure><p>Most of the researchers will normally collect the speech signals from the native speakers in order to maintain the quality of an acoustic model.</p><p>What if you really want to collect your own voice signals as the dataset for acoustic modelling? Or you wish to know how much does your pronunciation as a non-native speaker deviate from the native speakers?</p><p>In the following parts, I am going to <strong>show you how to perform acoustic and articulatory analysis on Japanese vowels by using your own recorded voice as inputs</strong>.</p><h3>Step 1: Dataset Preparation</h3><p>Firstly, you need to prepare the dataset by pronouncing some of the words using your own voice and recording them using your headset. Make sure that the surrounding environment is highly silent to maintain the quality of the voice recordings.</p><p>Each voice recording should contain only one word. For each vowel, find five words with the particular vowel and record with your own voice.</p><p>The words that are chosen for the demonstration are shown in the table below. The underlined characters are the vowels that are going to be further analyzed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*EVkYr7qV3PYiS36-ZmWMNQ.png" /><figcaption>Figure: The words that I chose for the demonstration.</figcaption></figure><p>Currently, the number of vowels and consonants for Japanese are 5 and 14 respectively. The Japanese vowels include あ/ア(a), い/イ(i), う/ウ(u), え/エ(e) and お/オ(o).</p><h3>Step 2: Voice Annotation</h3><p>After all the voice recordings are ready, the next procedure will be annotating the voice with phonemes. The tool that I would like to introduce is <strong>Praat</strong>, which is a free computer software package for speech analytics in phonetics. You can download it <a href="http://www.fon.hum.uva.nl/praat/download_win.html">here</a>.</p><p>The steps that you need to perform for voice annotation include:</p><ol><li>Download and install <strong>Praat</strong>.</li><li>Import all your voice recordings into <strong>Praat</strong>.</li><li>Create a blank <strong>Text Grid</strong> file for each word annotation.</li><li>Save all the <strong>Text Grid</strong> files to permanently record the annotation.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*hDBPjIdJyVTbhkn_wuWfsg.png" /><figcaption>Figure: The main user interface of Praat.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*AlnhkwOfy96Lys3lNL7GoQ.png" /><figcaption>Figure: The annotation of the Japanese word “こく”.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*QyeBavBdGPdNufnygrpNAA.png" /><figcaption>Figure: The annotation of the Japanese word “まち”.</figcaption></figure><h3>Step 3: First and Second Formants Search for Vowels</h3><p>Human speech production consists of two stages, which are <strong>source</strong> and <strong>filter</strong>.</p><p>The <strong>source</strong>, which is the vocal folds, will vibrate to generate a complex periodic voice signal. The frequency of the complex periodic voice signal is known as the fundamental frequency.</p><p>On the other hand, the <strong>filter</strong>, which is the vocal tract, will change its shape with the aid of articulation to modify the harmonics of the complex periodic voice signal. The frequency of the voice signal after modification is known as formant.</p><p>The average of the first and second formants can be easily figured out by going to the menu bar, clicking <strong>Formant</strong> and choosing <strong>First Formant</strong> or <strong>Second Formant</strong> according to your necessity. Then, the average first and second formant for each vowel is obtained and the result is recorded as shown in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*-afZ1cDOKllMjFtxNcQ2ww.png" /><figcaption>Figure: The procedure to get the first and second formant of vowels for each word.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*Lrq73Dw9cx3FzpDdb8TMkQ.png" /><figcaption>Table: The average of the first formant for each vowel.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*v2XK_WNVEmWb4xYYrGlv8Q.png" /><figcaption>Table: The average of the second formant for each vowel.</figcaption></figure><h3>Step 4: Plot Japanese Vowel Chart</h3><p>After the average readings of the first and second formants for each vowel are collected, the <strong>graph of the values of the first formant versus the second formant</strong> is plotted. The former shows the result derived from my voice recordings whereas the latter shows the result derived from the voice recordings provided by the Japanese native speakers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*vonSlPXRzuyfZ-jBn_0o7Q.png" /><figcaption>Figure: The result derived from my voice recordings.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*SLZK0KXvYt4J7qaPrfw8sw.png" /><figcaption>Figure: The result derived from Japanese native speakers [<a href="https://en.wikipedia.org/wiki/Japanese_phonology">Wikipedia</a>].</figcaption></figure><h3>Step 5: Acoustic &amp; Articulatory Analysis</h3><p>By comparing two charts drawn above:</p><ul><li>It shows that <strong>the locations of the Japanese vowels derived from my own voice recording are similar to Japanese native speakers</strong>. The vowels “い/イ(i)” and “う/ウ(u)” are located at the most top. The vowel “あ/ア(a)” is located at the most bottom. The most left is “い/イ(i)” and the most right is “お/オ(o)” as well.</li><li>Nevertheless, the <strong>horizontal distance between vowels “お/オ(o)” and “う/ウ(u)” for own voice recording is less than the standard</strong>. The <strong>vertical distance between vowels “い/イ(i)” and “う/ウ(u)” are not as large as what it is shown in the standard</strong>.</li></ul><h3>Conclusion</h3><p>As a non-native speaker of Japanese, I figured out that <strong>my pronunciation for vowel “う/ウ(u)” is slightly different compared to a Japanese native speaker</strong>. My pronunciation for other vowels looks good and are similar to Japanese native speakers. Instead of using the voice recording from Japanese native speakers for building the acoustic model, I can build my own acoustic model with self voice recordings.</p><h3>Reference</h3><ol><li>N. Shmyrev, “<a href="https://cmusphinx.github.io/wiki/tutorialconcepts/">Basic concepts of speech recognition</a>,” CMUSphinx Open Source Speech Recognition.</li><li>C. Smith, “<a href="https://www.researchgate.net/publication/248717410_Handbook_of_the_International_Phonetic_Association_A_guide_to_the_use_of_the_International_Phonetic_Alphabet_1999">Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet</a>, ” Phonology. 17. 291–295, 1999.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdee6ec1a4ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/speech-analysis-of-vowels-for-japanese-language-fdee6ec1a4ba">Speech Analysis of Vowels for Japanese Language</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>