<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Martin vega on Medium]]></title>
        <description><![CDATA[Stories by Martin vega on Medium]]></description>
        <link>https://medium.com/@mjvnor?source=rss-f8554dc82461------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Mep-o9SsfBfuEKhO</url>
            <title>Stories by Martin vega on Medium</title>
            <link>https://medium.com/@mjvnor?source=rss-f8554dc82461------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:04:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mjvnor/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Natural language analysis of Mexican president’s morning lectures with SQLite and spaCy]]></title>
            <link>https://medium.com/mcd-unison/most-used-words-in-the-mexican-presidents-morning-conferences-cf2005986cc1?source=rss-f8554dc82461------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf2005986cc1</guid>
            <category><![CDATA[amlo]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[word-cloud]]></category>
            <dc:creator><![CDATA[Martin vega]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 06:51:22 GMT</pubDate>
            <atom:updated>2022-12-14T01:50:30.730Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*_V6c70AcIE7UtipoQ-yHQA.png" /><figcaption>Most mentioned words in the Mexican morning conferencess</figcaption></figure><p>In the following text we will explain how to make a wordcloud of the most used words in the morning conferences of President Lopez Obrador.</p><p>Wordclouds are a useful tool for visualizing and analyzing text data. They allow you to quickly and easily identify the most frequently used words in a body of text, providing a visual representation of the words and their relative importance. This can be useful for identifying patterns and trends in the text, as well as for generating ideas for further analysis.</p><p>Additionally, wordclouds can be a useful tool for communicating the results of text analysis to others. They provide a visually appealing and easy-to-understand representation of the data, which can be useful for presentations or reports.</p><p>Overall, wordclouds are a useful tool for text analysis and visualization, and can provide valuable insights into a body of text.</p><p>First we create the function to save the data in SQLite.</p><pre>#Create Data Base<br>def createDB():<br>    conn = sql.connect(&quot;conferences.db&quot;)<br>    conn.commit()<br>    conn.close()<br>if not os.path.exists(&#39;./conferences.db&#39;):<br>    createDB()</pre><pre># Delete table<br>def dropTable(tabla):<br>    <br>    #Connecting to sqlite<br>    conn = sql.connect(&#39;conferences.db&#39;)<br><br>    #Creating a cursor object using the cursor() method<br>    cursor = conn.cursor()<br><br>    #Doping table if already exists<br>    cursor.execute(&quot;DROP TABLE IF EXISTS {}&quot;.format(tabla))<br>    print(&quot;Table dropped... &quot;)<br><br>    #Commit your changes in the database<br>    conn.commit()<br><br>    #Closing the connection<br>    conn.close()</pre><pre># Create Tables<br>def createTableParticipants():<br>    conn = sql.connect(&quot;conferences.db&quot;)<br>    cursor = conn.cursor()<br>    cursor.execute(&quot;&quot;&quot;CREATE TABLE participants(<br>        IDPerson INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE,<br>        name TEXT NOT NULL,<br>        unique (name)<br>    )&quot;&quot;&quot;)<br>    conn.commit()<br>    conn.close()<br><br>def createTableDialogs():<br>    conn = sql.connect(&quot;conferences.db&quot;)<br>    cursor = conn.cursor()<br>    cursor.execute(&quot;&quot;&quot;CREATE TABLE dialogs (<br>        IDDialog INTEGER PRIMARY KEY AUTOINCREMENT,<br>        Dialog TEXT NOT NULL,<br>        IDPerson INTEGER NOT NULL,<br>        Date datetime,<br>        FOREIGN KEY (IDPerson)<br>            REFERENCES participants(IDPerson)<br>    )&quot;&quot;&quot;)<br>    conn.commit()<br>    conn.close()<br># Using the functions to create the two tables.<br>createTableDialogs()<br>createTableParticipants()</pre><pre># Insert Data<br>def insertRowPersonas(name):<br>    conn = sql.connect(&quot;conferences.db&quot;)<br>    cursor = conn.cursor()<br>    instruction = f&quot;INSERT INTO participants(name) VALUES(&#39;{name}&#39;)&quot;<br>    cursor.execute(instruction)<br>    conn.commit()<br>    conn.close()<br><br>def insertRowDialog(dialog, IDParticipant, date):<br>    conn = sql.connect(&quot;conferences.db&quot;)<br>    cursor = conn.cursor()<br>    instruction = f&quot;INSERT INTO dialogs(Dialog,IDPerson,Date) VALUES(&#39;{dialog}&#39;,&#39;{IDParticipant}&#39;,&#39;{date}&#39;)&quot;<br>    cursor.execute(instruction)<br>    conn.commit()<br>    conn.close()</pre><pre># This function creates a view called conference_view that contains <br># the data from the two tables participants and dialogs. <br># You can call the function as follows:<br>def create_conference_view():<br>    conn = sql.connect(&quot;conferences.db&quot;)<br>    cursor = conn.cursor()<br><br>    cursor.execute(&quot;&quot;&quot;CREATE VIEW conference_view AS<br>        SELECT participants.IDPerson, participants.name, dialogs.Dialog, dialogs.Date<br>        FROM participants<br>        INNER JOIN dialogs<br>        ON participants.IDPerson = dialogs.IDPerson&quot;&quot;&quot;)<br><br>    conn.commit()<br>    conn.close()</pre><pre># Create stored procedure two create tables<br>def createConferencesTables():<br>    conn = sql.connect(&quot;conferences.db&quot;)<br>    cursor = conn.cursor()<br>    cursor.execute(&quot;&quot;&quot;CREATE PROCEDURE createConferencesTables()<br>    BEGIN<br>        CREATE TABLE participants(<br>            IDPerson INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE,<br>            name TEXT NOT NULL,<br>            unique (name)<br>        );<br>        CREATE TABLE dialogs (<br>            IDDialog INTEGER PRIMARY KEY AUTOINCREMENT,<br>            Dialog TEXT NOT NULL,<br>            IDPerson INTEGER NOT NULL,<br>            Date datetime,<br>            FOREIGN KEY (IDPerson)<br>                REFERENCES participants(IDPerson)<br>        );<br>    END;&quot;&quot;&quot;)<br>    conn.commit()<br>    conn.close()</pre><blockquote><em>(Note that SQLite stored procedures are limited to containing a single SELECT statement. If you need to execute a series of statements, you must use a user-defined function instead.)</em></blockquote><p>Second we download the data, these are stored in this <a href="https://github.com/NOSTRODATA/conferencias_matutinas_amlo/">link</a>:</p><p>That repository contains the transcripts of the stenographic versions of President Andres Manuel López Obrador’s morning conferences since the beginning of the conferences in December 2018 organized in CSVs.</p><pre># For downloading Data<br>import wget<br>import zipfile<br>import os<br>if not os.path.exists(&#39;./conferencias_matutinas_amlo-master.zip&#39;):<br>    # Url of President Andres Manuel Lopez Obrador&#39;s Morning Conferences ( Mañaneras AMLO )<br>    # Link to zip:https://github.com/NOSTRODATA/conferencias_matutinas_amlo<br>    url=&#39;https://github.com/NOSTRODATA/conferencias_matutinas_amlo/archive/refs/heads/master.zip&#39;<br>    wget.download(url)<br>    # Unzip the zip file<br>    with zipfile.ZipFile(&#39;./conferencias_matutinas_amlo-master.zip&#39;, &#39;r&#39;) as zip_ref:<br>        zip_ref.extractall(&quot;./&quot;)<br>    # Delete the zip file to keep the folders<br>    os.remove(&quot;conferencias_matutinas_amlo-master.zip&quot;)</pre><p>We use python to obtain a dataframe of all the words said by each person speaking at the conferences.</p><pre>from pathlib import Path<br># Get all the files<br>path_of_the_directory = &quot;./conferencias_matutinas_amlo-master/&quot;<br>paths = Path(path_of_the_directory).glob(&#39;**/mananera*.csv&#39;)<br><br># Read the files into dataframes<br>dfs = [pd.read_csv(f,index_col=False) for f in paths]<br><br># we generate a dataframe with the downloaded data<br>df = pd.concat(dfs, ignore_index=True)<br><br># create a date column taking the data from the columns &quot;Anio&quot;, &quot;Mes&quot;, &quot;Dia&quot;<br>cols=[&quot;Anio&quot;,&quot;Mes&quot;,&quot;Dia&quot;]<br>df[&#39;date&#39;] = df[cols].apply(lambda x: &#39;-&#39;.join(x.values.astype(str)), axis=&quot;columns&quot;)<br>df[&#39;date&#39;]=pd.to_datetime(df[&#39;date&#39;])</pre><p>This is how our dataframe is displayed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6F69F4xbkgCe-weVNqUA4Q.png" /><figcaption>Dataframe of all the words said by each person</figcaption></figure><p><strong>Data preprocessing and insertion for the table “participants”</strong></p><p>We need to enter data to the table “participants”, for that we take the column “Participant” from our dataframe, and we make a new dataframe with each participant of the “Mañaneras”.</p><pre>participants = df[&quot;Participante&quot;].value_counts().to_frame()<br>participants = participants.reset_index()<br>participants = participants.drop(columns=[&#39;Participante&#39;])<br>participants = participants.rename(columns={&quot;index&quot;: &quot;Persons&quot;})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*kSOHgB3Qlfm9ych2_EdsGA.png" /></figure><p><strong>Insert data to table “participants”</strong></p><p>We enter all the people who have spoken at the conferences into the database.</p><pre>for index, row in participants.iterrows():<br>    insertRowPersonas(row[&#39;Persons&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*ZOaF14rrAfQMyInyYMVA4Q.png" /></figure><p><strong>We crate a table of the participants to a new DataFrame</strong></p><pre>cnx = sql.connect(&#39;conferences.db&#39;)<br>persons = pd.read_sql_query(&quot;SELECT * FROM participants&quot;, cnx)<br>cnx.close()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*dE3FwUVkkhKafSBJNgAOBw.png" /></figure><p>Then we merge this dataframe whit our original to have the ID.</p><pre>res = pd.merge(persons.assign(grouper=persons[&#39;name&#39;]),<br>               df.assign(grouper=df[&#39;Participante&#39;]),<br>               how=&#39;left&#39;, on=&#39;grouper&#39;)<br>res = res.drop(columns=[&quot;name&quot;,&quot;grouper&quot;])<br>res.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKgE8U5NmHfUe5EeE-_i3g.png" /></figure><p>We upload this data to the Database.</p><pre>for index, row in res.iterrows():<br>    insertRowDialog(row[&#39;Texto&#39;],row[&#39;IDPerson&#39;],row[&#39;date&#39;])</pre><p>At the end we create the conference_view to see the data of both tables in one:</p><pre>create_conference_view()<br>create_conference_view()<br>cnx = sql.connect(&#39;conferences.db&#39;)<br>dfDialogs = pd.read_sql_query(&quot;SELECT * FROM conference_view&quot;, cnx)<br>cnx.close()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Z3gME9_bLZoKmATtH36OA.png" /></figure><p>The next step is to normalize our text, we use the following function to normalice the text:</p><pre>def df_normalize(df_text_rows,column):<br>  # Remove accents<br>  text_without_accents = []<br>  for text in df_text_rows[column]:<br>    accents_re = re.sub(r&quot;([^n\u0300-\u036f]|n(?!\u0303(?![\u0300-\u036f])))[\u0300-\u036f]+&quot;, r&quot;\1&quot;, normalize( &quot;NFD&quot;, text), 0, re.I)<br>    text_without_accent = normalize( &#39;NFC&#39;, accents_re)<br>    # add the text without accents to the list<br>    text_without_accents.append(text_without_accent) <br><br>  # Regular expressions to replace symbols with spaces<br>  replace_by_spaces_re = re.compile(&#39;[\n/(){}\[\]\|@,;\.]&#39;)<br><br>  # Stopwords are defined for the Spanish language<br>  words_empty = set(nltk.corpus.stopwords.words(&#39;spanish&#39;))<br><br>  clean_text = []<br>  for text in text_without_accents:<br>    # Change text to lowercase<br>    text = text.lower()<br><br>    # Removes symbols<br>    text = re.sub(replace_by_spaces_re, &#39; &#39;, text)<br><br>    #Elimina risas<br>    text = re.sub(r&#39;[ja]+aj\w&#39;, r&#39;JA&#39;, text)<br>    text = re.sub(r&#39;[je]+ej\w&#39;, r&#39;JE&#39;, text)<br>    text = re.sub(r&#39;[ji]+ij\w&#39;, r&#39;JI&#39;, text)<br>    text = re.sub(r&#39;[ha]+ah\w&#39;, r&#39;HA&#39;, text)<br>    text = re.sub(r&#39;[he]+eh\w&#39;, r&#39;HE&#39;, text)<br>    <br>    # Eliminates stop words <br>    text = &#39; &#39;.join([word for word in text.split() <br>                    if word not in words_empty])<br>    <br>    # The processed text is saved<br>    clean_text.append(text)<br><br>  # the clean texts are added in a new column<br>  df_text_rows[&#39;FULL_TEXT_NORM&#39;] = clean_text<br><br>  display(df_text_rows)<br><br># We use the function<br>df_normalize(dfDialogs,&#39;Dialog&#39;)</pre><p>The new column with the standardized text is shown here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bRxTQFTUO1r2EAr6lUyj-A.png" /></figure><p>We take the column “FULL_TEXT_NORM” and generate an array with all the words, from there we use the library “WordCloud” and give it as input our array.</p><pre>from wordcloud import WordCloud, ImageColorGenerator<br><br># We take an image to generate the wordcloud<br>imgMask = np.array(Image.open(&#39;amlo.jpg&#39;))<br><br># Generate an array with all our words<br>allWords = &#39; &#39;.join( [dialog for dialog in dfDialogs[&#39;FULL_TEXT_NORM&#39;]] )<br><br># Change the background from white to black<br>WordCloud = WordCloud(width = 1920, height = 1080, background_color=&quot;black&quot;,random_state = 42, max_font_size = 200, mask = imgMask, max_words=473).generate(allWords)<br>image_colors = ImageColorGenerator(imgMask)<br><br># We show the WordCloud<br>plt.figure(figsize=(16,9), dpi=120)<br>plt.imshow(WordCloud.recolor(color_func=image_colors), interpolation = &#39;bilinear&#39;)<br>plt.title(&#39;WordCloud of the most mentioned words in the Mexican morning conferencess&#39; , fontsize=14)<br>plt.axis(&#39;off&#39;)<br>plt.show()<br>plt.savefig(&#39;wordcloudAmloManianera.png&#39;)</pre><p>Final image with the most mentioned words in the morning lectures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*VB3PjoqcKTpTr63z1c1-1Q.png" /><figcaption>Most mentioned words in the Mexican morning conferencess</figcaption></figure><p>Now the wordcloud of the most spoken words by Andres Manuel Lopez Obrador:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOaUs1Md0YUyovCV63xFnA.png" /><figcaption>Most mentioned words in the Mexican morning conferences</figcaption></figure><p>Using spacy to generate a wordcloud of the adjectives, verbs, nouns of the 2018 dialog of AMLO.</p><p>Creating a word cloud using only nouns can provide a more focused view of the most important objects, people, places, or concepts mentioned in a piece of text. This can be useful for quickly identifying key themes or ideas in the text. Additionally, using only nouns can make the word cloud more visually appealing, as the resulting cloud will typically have a more balanced and natural-looking distribution of words. This can make it easier to understand and interpret the word cloud, as the words will be more evenly distributed and will not be crowded together in a cluttered-looking cloud.</p><p>Making use of adjectives, a word cloud can help to highlight the descriptive words used in a text, which can give insight into the tone or sentiment of the writing. Additionally, using only adjectives can make the word cloud more aesthetically pleasing and visually striking, as adjectives are often more colorful and descriptive than other types of words.</p><p>By using only verbs, the word cloud can also provide a more dynamic and active view of the text, as it will highlight the actions and events that are central to the story or message being conveyed.</p><pre>#Crating a ned DF that only get what the president say<br>amlo_df = dfDialogs.loc[df[&#39;name&#39;]==&quot;PRESIDENTE ANDRES MANUEL LOPEZ OBRADOR&quot;]<br>amlo_df = amlo_df.loc[amlo_df[&#39;Anio&#39;]==2018]<br><br>nlp = spacy.load(&#39;es_core_news_md&#39;)<br>nlp.max_length = 17840260<br># Generate an array with all our words<br>allWords = &#39; &#39;.join(amlo_df.FULL_TEXT_NORM.values)<br>doc = nlp(allWords)</pre><p>Crating arrays of the adjectives, verbs and nouns.</p><pre># Adjectives<br>adjectives_words = &#39; &#39;.join(<br>    [ <br>     token.norm_ for token in doc<br>     if token.is_alpha and not token.like_num and not token.is_stop and<br>        not token.is_currency and token.pos_ in [&#39;ADJ&#39;]<br>    ]<br>)<br><br># Verbs<br>verbs_words = &#39; &#39;.join(<br>  [token.norm_ for token in doc if token.pos_ in [&#39;VERB&#39;]]<br>)<br><br># Nouns<br>nouns_words = &#39; &#39;.join(<br>    [ <br>     token.norm_ for token in doc<br>     if token.is_alpha and not token.like_num and not token.is_stop and<br>        not token.is_currency and token.pos_ in [&#39;NOUN&#39;]<br>    ]<br>)</pre><pre>from wordcloud import WordCloud, ImageColorGenerator<br><br># We take an image to generate the wordcloud<br>imgMask = np.array(Image.open(&#39;amlo.jpg&#39;))<br><br># Change the background from white to black<br>WordCloud = WordCloud(width = 1920, height = 1080, background_color=&quot;black&quot;,random_state = 42, max_font_size = 200, mask = imgMask, max_words=473).generate(adjectives_words)<br>image_colors = ImageColorGenerator(imgMask)<br><br># We show the WordCloud<br>plt.figure(figsize=(16,9), dpi=120)<br>plt.imshow(WordCloud.recolor(color_func=image_colors), interpolation = &#39;bilinear&#39;)<br>plt.title(&#39;WordCloud of the most mentioned adjectives in the Mexican morning conferences by the president of Mexico in 2018&#39; , fontsize=14)<br>plt.axis(&#39;off&#39;)<br>plt.savefig(&#39;wordcloudAmloManianera.png&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MOpxMaNoRqC3VI0VgDRL8Q.png" /></figure><pre>from wordcloud import WordCloud, ImageColorGenerator<br><br># We take an image to generate the wordcloud<br>imgMask = np.array(Image.open(&#39;amlo.jpg&#39;))<br><br># Change the background from white to black<br>WordCloud = WordCloud(width = 1920, height = 1080, background_color=&quot;black&quot;,random_state = 42, max_font_size = 200, mask = imgMask, max_words=473).generate(nouns_words)<br>image_colors = ImageColorGenerator(imgMask)<br><br># We show the WordCloud<br>plt.figure(figsize=(16,9), dpi=120)<br>plt.imshow(WordCloud.recolor(color_func=image_colors), interpolation = &#39;bilinear&#39;)<br>plt.title(&#39;WordCloud of the most mentioned nouns in the Mexican morning conferences by the president of Mexico in 2018&#39; , fontsize=14)<br>plt.axis(&#39;off&#39;)<br>plt.savefig(&#39;wordcloudAmloManianera.png&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xtnQxgceCtXYT78pgs0Pwg.png" /></figure><pre>from wordcloud import WordCloud, ImageColorGenerator<br><br># We take an image to generate the wordcloud<br>imgMask = np.array(Image.open(&#39;amlo.jpg&#39;))<br><br># Change the background from white to black<br>WordCloud = WordCloud(width = 1920, height = 1080, background_color=&quot;black&quot;,random_state = 42, max_font_size = 200, mask = imgMask, max_words=473).generate(verbs_words)<br>image_colors = ImageColorGenerator(imgMask)<br><br># We show the WordCloud<br>plt.figure(figsize=(16,9), dpi=120)<br>plt.imshow(WordCloud.recolor(color_func=image_colors), interpolation = &#39;bilinear&#39;)<br>plt.title(&#39;WordCloud of the most mentioned verbs in the Mexican morning conferences by the president of Mexico in 2018&#39; , fontsize=14)<br>plt.axis(&#39;off&#39;)<br>plt.savefig(&#39;wordcloudAmloManianera.png&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCR24rBkFlou__YkxGvuyg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf2005986cc1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mcd-unison/most-used-words-in-the-mexican-presidents-morning-conferences-cf2005986cc1">Natural language analysis of Mexican president’s morning lectures with SQLite and spaCy</a> was originally published in <a href="https://medium.com/mcd-unison">MCD-UNISON</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Al hacer aprendizaje automático en la industria, ¿en qué pienso?]]></title>
            <link>https://medium.com/lcc-unison/al-hacer-aprendizaje-autom%C3%A1tico-en-la-industria-en-qu%C3%A9-pienso-6825bd68bef0?source=rss-f8554dc82461------2</link>
            <guid isPermaLink="false">https://medium.com/p/6825bd68bef0</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[pensamientos]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[industria]]></category>
            <category><![CDATA[thoughts]]></category>
            <dc:creator><![CDATA[Martin vega]]></dc:creator>
            <pubDate>Mon, 06 Dec 2021 22:28:21 GMT</pubDate>
            <atom:updated>2021-12-06T22:28:21.015Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*laAios_9WJttJijNEQDKrA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@younis67?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">L N</a> on <a href="https://unsplash.com/s/photos/industry?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Introducción</h3><p>Con la visión por computadora y el aprendizaje automático, la inteligencia artificial puede ayudar a identificar objetos en una imagen de manera muy precisa y rápida, incluso si estos objetos no tienen características visuales distinguibles o no están ubicados en una ubicación específica dentro de la imagen. El sistema también se puede utilizar para detectar defectos o anomalías en las imágenes recopiladas, como arañazos.</p><h3>1. La AI en la industria</h3><p>En la industria, muchas actividades que implican trabajos de alta precisión se llevan a cabo con la ayuda de sistemas de inteligencia artificial. Estos sistemas proporcionan una mayor precisión y ahorran tiempo y recursos. Además, también tienen un buen nivel de seguridad, lo que los hace aptos para su uso en industrias donde existen riesgos involucrados.</p><p>Algunas industrias están utilizando IA para ayudarles a automatizar ciertos procesos. Por ejemplo, en la industria manufacturera, la IA se utiliza para el procesamiento de imágenes y la visión por computadora. Esto ayuda con la inspección de los productos y ayuda a garantizar que cumplan con ciertas especificaciones.</p><p>La IA también se ha utilizado para inspeccionar edificios por motivos de seguridad. Puede usarse para identificar riesgos potenciales como deformación del material, daños estructurales, problemas de aislamiento y más. Estas son cosas que los inspectores humanos generalmente pasan por alto porque no tienen el tiempo ni la mano de obra para examinar cada paso del proceso de inspección de un edificio.</p><h3>2. Recolectar imágenes</h3><p>Uno de los aspectos más importantes del desarrollo de la IA es la recopilación de imágenes. Las personas recopilan una gran cantidad de estas imágenes, pero algunas se recopilan mediante software.</p><p>La recopilación de imágenes para un modelo de IA se puede realizar a través de dos métodos: recopilación manual o recopilación automática. La recolección manual a menudo es costosa y requiere mucho tiempo. Requiere que una persona salga y recopile manualmente las imágenes para su procesamiento, lo que puede ser difícil dependiendo de lo raras que sean las fotos. El procesamiento de imágenes automatizado es más eficiente porque utiliza la visión por computadora para recopilar fotos sin intervención humana.</p><p>No es suficiente tener un conjunto de datos de imágenes, es necesario anotarlo para que los modelos de IA puedan aprender de él.</p><p>Es necesario mejorar la inspección de las imágenes recopiladas. Esto se debe a que es necesario tener un alto nivel de precisión para aquellas imágenes que provienen de inspecciones de cerca.</p><h3>3. Segmentarlas</h3><p>La segmentación de imágenes es un aspecto importante del modelo de aprendizaje automático. Cuando tenemos que entrenar un modelo sobre una gran cantidad de imágenes, la segmentación es tan importante para identificar los objetos en una imagen.</p><p>La segmentación de imágenes es un proceso que divide la imagen en partes, lo que permite que el algoritmo identifique el tipo de objeto en esa región.</p><p>Un modelo de aprendizaje automático puede usar estas segmentaciones como entrada para aprender nuevos tipos de objetos y aplicar esos modelos aprendidos para detectar estos objetos en nuevas imágenes.</p><h3>3.1 Témplate Matching</h3><p>La coincidencia de plantillas es una técnica en visión por computadora que encuentra un objeto o patrón en una imagen dada al buscar otros objetos o patrones similares dentro de la misma imagen. El algoritmo encuentra la plantilla coincidente calculando la correlación cruzada normalizada de dos funciones.</p><p>La coincidencia de plantillas se utiliza ampliamente en varios campos del reconocimiento de patrones. Tiene aplicaciones en imágenes médicas, seguimiento de objetivos, sistemas de reconocimiento facial y comparación de huellas dactilares.</p><p>También se puede utilizar para encontrar las coordenadas de un objeto en un plano a partir de imágenes de la cámara con coordenadas desconocidas.</p><h3>4. Un modelo neuronal</h3><p>La clasificación de imágenes es un proceso de identificación de la categoría de una imagen. Por ejemplo, si la entrada es una imagen de un perro, la salida sería “perro” o “gato”.</p><p>El aprendizaje profundo es una técnica que se ha utilizado ampliamente para la clasificación de imágenes. Los modelos de aprendizaje profundo utilizan redes neuronales artificiales para analizar imágenes y clasificarlas según determinadas categorías.</p><p>Un modelo de aprendizaje profundo para la clasificación de imágenes binarias debe tener dos salidas, una para cada categoría. Las entradas pueden ser imágenes o valores de píxeles. El modelo consta de varias capas y la primera capa evaluaría todos los píxeles de una imagen y le asignaría un valor de intensidad. Estas capas también tendrían la capacidad de aprender de los errores y mejorar con el tiempo con cada iteración o “epoch”.</p><h3>5. Probarlo</h3><p>Podemos probar un modelo de aprendizaje profundo dividiéndolo en dos partes.</p><p>La primera parte sería el conjunto de entrenamiento, donde recopilamos nuestros datos y entrenamos nuestro modelo con esos datos. La segunda parte es el conjunto de prueba, que usamos para ver qué tan bien se desempeña el modelo entrenado frente a datos invisibles. Luego comparamos los resultados de estos dos conjuntos para determinar qué tan preciso es nuestro modelo.</p><h3>6. Probarlo en planta</h3><p>El aprendizaje profundo es un tema de tendencia en la industria. Es realmente difícil probar un modelo de aprendizaje profundo debido a todos los parámetros involucrados.</p><p>El primer paso para probar un modelo de aprendizaje profundo sería compararlo con los modelos existentes. Si no hay modelos disponibles, es mejor hacer comparaciones con otras redes neuronales profundas.</p><p>Luego, después de comparar el nuevo modelo de aprendizaje profundo con uno existente, se debe comenzar ajustando los pesos a mano. Este paso puede ayudar a identificar algunos casos en los que los dos modelos son diferentes y también puede ayudar a identificar algunos casos en los que los dos modelos son similares.</p><p>Por último, se debe evaluar el rendimiento en conjuntos de datos que no se utilizaron para el entrenamiento o la validación del modelo.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6825bd68bef0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lcc-unison/al-hacer-aprendizaje-autom%C3%A1tico-en-la-industria-en-qu%C3%A9-pienso-6825bd68bef0">Al hacer aprendizaje automático en la industria, ¿en qué pienso?</a> was originally published in <a href="https://medium.com/lcc-unison">LCC-Unison</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to fail to generate “corridos tumbados” using AI]]></title>
            <link>https://medium.com/data-science/how-to-fail-to-generate-corridos-tumbados-using-ai-b10395d09e26?source=rss-f8554dc82461------2</link>
            <guid isPermaLink="false">https://medium.com/p/b10395d09e26</guid>
            <category><![CDATA[lstm]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[wavenet]]></category>
            <category><![CDATA[corridos-tumbados]]></category>
            <category><![CDATA[nsynth]]></category>
            <dc:creator><![CDATA[Martin vega]]></dc:creator>
            <pubDate>Mon, 24 May 2021 13:00:35 GMT</pubDate>
            <atom:updated>2021-05-24T18:34:47.943Z</atom:updated>
            <content:encoded><![CDATA[<h4>“Corridos tumbados” and Machine Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O44B_PlF1Etwqu_I" /><figcaption>Photo by <a href="https://unsplash.com/@russphotographer?utm_source=medium&amp;utm_medium=referral">Russ Photographer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Eliud Gilberto Rodríguez Martínez, Hugo Francisco Cano Beyliss, Martín José Vega Noriega.</blockquote><blockquote>Universidad de Sonora, Licenciatura, Hermosillo, Sonora,<br>México</blockquote><blockquote><a href="/service/mailto:mjvnor@outlook.com">mjvnor@outlook.com</a>, slabxcobra@gmail.com, eliud.giroma@gmail.com.</blockquote><p><strong>Abstract.</strong> This project was made with the objective of generating “corridos tumbados” using neural networks for the neural networks class of Professor Julio Waissman Vilanova. We took on the task of using different methods to generate songs, using different neural networks such as RNNs, WaveNets and synthesizers. Here we explain why we were able to carry out this task but with the inconvenient that the results are not as expected due to various problems that arose when carrying out this task, problems such as having a dataset with a lot of noise (they have more than one instrument) and the problem of not having the equipment required to handle large networks.</p><p><strong>Keywords: </strong>corridos tumbados,<strong> </strong>neural networks, lstm, wavenet, n-synth, keras</p><h3>Why we decided to do this project</h3><p>We decided to do this project due to a task assigned to us in the neural networks class of Professor Julio Waissman Vilanova, this task was about generating music with the use of recurrent neural networks (RNN) and we decided to take this opportunity to create “corridos tumbados”, which are very popular mainly in northern Mexico.</p><h4>What is a corrido tumbado?</h4><p>Corridos tumbados are an aberration of the Mexican regional, which are a variant of the traditional corrido but with inclusions of hip-hop and trap, their singers usually wear clothes characteristic of the trap. Definitely something that you should listen to.</p><h3>What we did…</h3><h4>The first thing we tried</h4><p>In our case, there is no MIDIs dataset of corridos tumbados, so we set about creating one. First we wanted to generate corridos tumbados with all the instruments these types of songs have, but we realized that it was too time consuming and difficult to implement, so we used the “SpleeterGui” software to get the requintos of the corridos tumbados that we had at our disposal. Why the requinto? The requinto is the main instrument of the corridos tumbados, without this instrument these are not corridos tumbados. We use an AI provided by Google called “Onsets and Frames” which is an automatic music transcription framework with piano and drums models, which will help us transform our requintos to MIDIs.</p><ul><li>Colab:</li></ul><p><a href="https://colab.research.google.com/github/MJVNOR/DatasetMidiCorridosTumbadosAcustico/blob/main/Onsets_and_Frames.ipynb">Google Colaboratory</a></p><p>We generated around 241 corridos tumbados in the MIDI format and from these we extract the notes and chords with the help of a library called “music21” and we use them as input to our network. We then take a long short-term memory recurrent neural network (LSTM) to generate lying runs and this is the result:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374694%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-ai-lstm&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4f2578a2bac62e202c7178f86538156/href">https://medium.com/media/e4f2578a2bac62e202c7178f86538156/href</a></iframe><ul><li>Colab:</li></ul><p><a href="https://colab.research.google.com/github/MJVNOR/DatasetMidiCorridosTumbadosAcustico/blob/main/GeneratorMusicLstm.ipynb">Google Colaboratory</a></p><h4>Why we failed</h4><p>We failed because our dataset was not good enough, it contained too much noise and other instruments besides the requintos. We also needed to do more epochs for the songs to be generated better, this was an issue because if we gave it more epochs it took more than 15 hours to finish and google colab no longer allows you to use their gpu when you exceed that time.</p><h4>The second thing we tried</h4><p>The second thing we tried was to use a magenta project called N-SYNTH, which is a neural synthesizer. N-SYNTH synthesizers manipulate data instead of manipulating sound; analyze audio files, learn how the data is related, and then produce new audio files from scratch. To listen to music with quality close to real life, 48,000 samples are required. This synthesizer generates one sample at a time based on a few thousand samples above, as well as the audio in which it has been trained and it sounds a little worse because instead of the 48,000 samples, 16,000 are used, which is still quite a bit.</p><ul><li>Result:</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1053874996%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Finterpolation-whit-amor-tumbado-and-ya-lo-supere&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-bUD7jeXYvFnI2tQv-DbWLfg-t500x500.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe434aed32c4b96f79d82612c525eab0/href">https://medium.com/media/fe434aed32c4b96f79d82612c525eab0/href</a></iframe><ul><li>Colab:</li></ul><p><a href="https://colab.research.google.com/github/MJVNOR/DatasetMidiCorridosTumbadosAcustico/blob/main/nsynth.ipynb">Google Colaboratory</a></p><p>The algorithm was trained with high pitched sounds, it trained with 300,000 notes from a thousand different instruments. When you combine two sounds that are completely different, it is more complicated for the synthesizer since it has to handle more sounds than just single notes, apart from the fact that the synthesizer is trained in high-pitched sounds, so it will try to interpret any other frequency as high-pitched sounds, which means that, apart from being created in low quality by using the 16,000 samples, weird melodies are created due to the possible lack of instruments, and wanting to take the frequency as high, generating aberrations of audio that look like horror movies.</p><h4>The last thing we tried</h4><p>The last thing we tried to use was a wavenet. A simplified version of the WaveNet architecture is used without adding residual connections or skip connections since the role of these layers is to improve fast convergence and WaveNet takes raw audio waves as input, but, in our case, the input would be a set of nodes and chords, since we are generating music.</p><p>Results:</p><p>Junior H (best 5) | 50 epoch</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374685%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-junior-h-best-5-50-epoch-wavenet&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/42dddc0717effe93f866722c524f1165/href">https://medium.com/media/42dddc0717effe93f866722c524f1165/href</a></iframe><p>Junior H (best 5) | 500 epoch</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374688%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-junior-h-best-5-500-epoch-wavenet&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbba9bd4f2d255de4518e8c6318fb333/href">https://medium.com/media/dbba9bd4f2d255de4518e8c6318fb333/href</a></iframe><p>Full dataset | 50 epoch</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374661%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-full-dataset-50-epoch-wavenet&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/19ecf88e79d91da18fe5c7fd832f27dd/href">https://medium.com/media/19ecf88e79d91da18fe5c7fd832f27dd/href</a></iframe><p>Full dataset | 200 epoch</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374673%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-full-dataset-200-epoch-wavenet&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/be68ac1145951de4e4305b0572618db2/href">https://medium.com/media/be68ac1145951de4e4305b0572618db2/href</a></iframe><ul><li>Colab:</li></ul><p><a href="https://colab.research.google.com/github/MJVNOR/DatasetMidiCorridosTumbadosAcustico/blob/main/AutoMusicGen.ipynb">Google Colaboratory</a></p><h4>Why we failed</h4><p>There are several possibilities but what we believe was that the model did not manage to extract the important characteristics of the songs, this is because the model worked with only one type of instrument, and since we have a dataset not as clean as we would like, this introduced noise affecting the learning phase, in addition there could be the possibility that the model needed more epochs to train or more songs.</p><h3>What is an LSTM?</h3><p>Long Short-Term Memory (LSTM) networks are a type of recurrent neural network, which extends the memory of recurrent neural networks, they have the short-term memory problem so they only use previous information so they they don’t have the list of all the above information available for the neural node. Due to their characteristic behavior, they are used to solve speech recognition problems, handwriting recognition, music composition, etc. complex problems that recurrent neural networks could not solve.</p><h3>What is a Wavenet?</h3><p>WaveNet is a type of neural network known as a deep convolutional neural network for generating sample-by-sample audio. The technique is capable of generating realistic sounds such as human voices by modeling waveforms directly using trained neural networks directly from samples of real human speech. Wavenet’s ability to generate waveforms allows any kind of audio to be modeled, including music. In WaveNet, the network samples a signal as an input and synthesizes sample by sample the output.</p><h3>What is an Interpolation?</h3><p>Interpolation consists of making an educated guess with the information from a certain data set. This is a “best guess” using the information you have at hand. Interpolation is something we use AI for often. Feed data to a computer and allow it to make educated guesses for us. In theory, interpolation is also useful for extracting data about situations and using known experiences to expand knowledge into unfamiliar areas. However, this is often called extrapolation.</p><h3>Conclusion</h3><p>Our conclusion is that we must try to improve our dataset much more, if it had been cleaner, the output of our second attempt would have improved a lot. We should also have used another system apart from google colab to run the models for a longer periods of time.</p><h3>Github repository</h3><p><a href="https://github.com/MJVNOR/DatasetMidiCorridosTumbadosAcustico">MJVNOR/DatasetMidiCorridosTumbadosAcustico</a></p><h3>References</h3><p>Hawthorne, C., Elsen, E., Song, J., Roberts, A., Simon, I., Raffel, C., … &amp; Eck, D. (2017). Onsets and frames: Dual-objective piano transcription. arXiv preprint arXiv:1710.11153.</p><ul><li><a href="https://deepai.org/machine-learning-glossary-and-terms/interpolation">Interpolation</a></li><li><a href="https://es.wikipedia.org/wiki/WaveNet">WaveNet - Wikipedia, la enciclopedia libre</a></li><li><a href="https://hub.packtpub.com/what-is-lstm/">What is LSTM? | Packt Hub</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b10395d09e26" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-fail-to-generate-corridos-tumbados-using-ai-b10395d09e26">How to fail to generate “corridos tumbados” using AI</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Como fallar en generar “corridos tumbados” mediante IA]]></title>
            <link>https://medium.com/lcc-unison/como-fallar-en-generar-corridos-tumbados-mediante-ia-28c74a0b23db?source=rss-f8554dc82461------2</link>
            <guid isPermaLink="false">https://medium.com/p/28c74a0b23db</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[nsynth]]></category>
            <category><![CDATA[corridos-tumbados]]></category>
            <category><![CDATA[wavenet]]></category>
            <category><![CDATA[lstm]]></category>
            <dc:creator><![CDATA[Martin vega]]></dc:creator>
            <pubDate>Mon, 24 May 2021 08:27:01 GMT</pubDate>
            <atom:updated>2021-05-24T20:45:20.497Z</atom:updated>
            <content:encoded><![CDATA[<h4>“Corridos tumbados” y Machine Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cp9c0LeQKaIUKcp-" /><figcaption>Photo by <a href="https://unsplash.com/@russphotographer?utm_source=medium&amp;utm_medium=referral">Russ Photographer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Eliud Gilberto Rodríguez Martínez, Hugo Francisco Cano Beyliss, Martín José Vega Noriega.</blockquote><blockquote>Universidad de Sonora, Licenciatura, Hermosillo, Sonora,<br>México</blockquote><blockquote>mjvnor@outlook.com, slabxcobra@gmail.com, eliud.giroma@gmail.com.</blockquote><p><strong>Resumen.</strong> Este proyecto fue hecho con el objetivo de generar corridos tumbados mediante redes neuronales para la clase de redes neuronales del profesor Julio Waissman. Nos dimos a la tarea de usar distintos métodos para generar canciones, utilizando diferentes redes neuronales como RNNs, WaveNets, y sintetizadores. Aquí explicamos porque pudimos realizar dicha tarea pero con el inconveniente de que los resultados no son los esperados debido a diversos problemas que surgieron a la hora de realizar dicha tarea como problemas con un dataset con mucho ruido (tienen mas de un instrumento) y los problemas de no tener un equipo requerido para manejar grandes redes.</p><p><strong>Palabras clave: </strong>corridos tumbados, redes neuronales, lstm, wavenet, n-synth, keras</p><h3>El por que del proyecto</h3><p>Decidimos hacer este proyecto debido a una tarea que nos asignaron en la clase de redes neuronales del profesor Julio Waissman Vilanova, esta tarea se trataba de generar música a partir del uso de redes neuronales recurrentes y nosotros decidimos aprovechar esta oportunidad para crear corridos tumbados, los cuales son muy populares principalmente en el norte de México.</p><p><strong>¿Qué son los corridos tumbados?</strong></p><p>Los corridos tumbados son una aberración del regional mexicano los cuales son una variante del corrido tradicional pero con inclusiones del hip-hop y trap, sus cantantes suelen portar vestimentas características del trap. Sin duda una chulada que deberías escuchar.</p><h3>Lo que hicimos…</h3><p><strong>Lo primero que intentamos</strong></p><p>Para nuestro caso no hay ningún dataset de MIDIs de corridos tumbados, así que nos dimos a la tarea de crear uno. Primero quisimos generar corridos tumbados con todos los instrumentos que tienen ese tipo de canciones, pero nos dimos cuenta que era demasiado tardado y difícil de implementar, así que utilizamos el programa “SpleeterGui” para sacar los requintos de los corridos tumbados que teníamos a nuestra disposición. ¿Por qué el requinto?, el requinto es el instrumento principal de los corridos tumbados, sin ese instrumento estos no serían corridos tumbados. Utilizamos una IA proporcionada por Google que se llama “Onsets and Frames” la cual es un framework de transcripción automática de música con modelos de piano y batería, que nos va a ayudar a transformar nuestros requintos a MIDIs.</p><ul><li>Colab:</li></ul><p><a href="https://colab.research.google.com/github/MJVNOR/DatasetMidiCorridosTumbadosAcustico/blob/main/Onsets_and_Frames.ipynb">Google Colaboratory</a></p><p>Generamos alrededor de 241 corridos tumbados en el formato MIDI y de estos sacamos las notas y los acordes con ayuda de una biblioteca llamada “music21” y las metemos como entrada a nuestra red. Tomamos entonces una red neuronal recurrente tipo lstm para generar corridos tumbados (hicimos lo mismo en nuestro ultimo intento) y este es el resultado:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374694%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-ai-lstm&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4f2578a2bac62e202c7178f86538156/href">https://medium.com/media/e4f2578a2bac62e202c7178f86538156/href</a></iframe><ul><li>Colab:</li></ul><p><a href="https://colab.research.google.com/github/MJVNOR/DatasetMidiCorridosTumbadosAcustico/blob/main/GeneratorMusicLstm.ipynb">Google Colaboratory</a></p><h4>Por que fallamos</h4><p>Fallamos debido a que nuestro dataset no fue lo suficientemente bueno, este contenía demasiado ruido y contenían otros instrumentos aparte de los requintos. También nos hizo falta hacer más epochs para que se generaran mejor las canciones ya que si dejábamos que hiciera más epochs tardaba más de 15 horas y el colab ya no te permite usar la gpu cuando sobrepasas ese tiempo.</p><p><strong>Lo segundo que intentamos</strong></p><p>Lo segundo que intentamos fue usar un proyecto de magenta llamado de N-SYNTH, que es un sintetizador neural. Los sintetizadores N-SYNTH manipulan datos en vez de manipular el sonido; analiza los archivos de audio, aprende cómo se relacionan los datos y luego produce nuevos archivos de audio desde cero. Para escuchar música con calidad cercana a la vida real, se necesitan 48000 samples. Este sintetizador genera un sample a la vez basado en unas de las cuantas miles de samples anteriores, así como el audio en el que ha sido entrenado y suena un poco peor debido a que en vez de los 48000, se usan 16,000 samples, que sigue siendo bastante.</p><ul><li>Resultado:</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1053874996%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Finterpolation-whit-amor-tumbado-and-ya-lo-supere&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-bUD7jeXYvFnI2tQv-DbWLfg-t500x500.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe434aed32c4b96f79d82612c525eab0/href">https://medium.com/media/fe434aed32c4b96f79d82612c525eab0/href</a></iframe><ul><li>Colab:</li></ul><p><a href="https://colab.research.google.com/github/MJVNOR/DatasetMidiCorridosTumbadosAcustico/blob/main/nsynth.ipynb">Google Colaboratory</a></p><p><strong>Por que fallamos</strong></p><p>El algoritmo fue entrenado con sonidos agudos, entrenó con 300,000 notas de mil instrumentos diferentes. Cuando se combinan dos sonidos que son completamente diferentes, es más complicado para el sintetizador ya que tiene que manejar más sonidos que solo notas simples, aparte del hecho que el sintetizador está entrenado en sonidos agudos, por lo que tratará de interpretar cualquier otra frecuencia como aguda, lo que hace que, aparte de crearse en baja calidad por usar los 16,000 samples, se creen melodías raras debido a la posible falta de instrumentos, y el querer tomar la frecuencia como aguda, generando aberraciones de audios que parecen de películas de terror.</p><p><strong>Lo último que intentamos</strong></p><p>Lo último que intentamos usar fue una wavenet. Se usa una versión simplificada de la arquitectura de una WaveNet sin añadir conexiones residuales ni conexiones saltables ya que el rol que cumplen estas capas es el de mejorar la convergencia rápida y WaveNet toma ondas de audio en bruto como entrada, pero, en nuestro caso, la entrada sería un set de nodos y acordes, ya que estamos generando música.</p><ul><li>Resultados:</li></ul><p>Junior H (best 5) | 50 epoch</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374685%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-junior-h-best-5-50-epoch-wavenet&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/42dddc0717effe93f866722c524f1165/href">https://medium.com/media/42dddc0717effe93f866722c524f1165/href</a></iframe><p>Junior H (best 5) | 500 epoch</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374688%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-junior-h-best-5-500-epoch-wavenet&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbba9bd4f2d255de4518e8c6318fb333/href">https://medium.com/media/dbba9bd4f2d255de4518e8c6318fb333/href</a></iframe><p>Full dataset | 50 epoch</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374661%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-full-dataset-50-epoch-wavenet&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/19ecf88e79d91da18fe5c7fd832f27dd/href">https://medium.com/media/19ecf88e79d91da18fe5c7fd832f27dd/href</a></iframe><p>Full dataset | 200 epoch</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1054374673%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmartin-vega-954546492%2Frequito-tumbado-full-dataset-200-epoch-wavenet&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-aJCsKIflmaq5D2cj-zxdzbA-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/be68ac1145951de4e4305b0572618db2/href">https://medium.com/media/be68ac1145951de4e4305b0572618db2/href</a></iframe><ul><li>Colab:</li></ul><p><a href="https://colab.research.google.com/github/MJVNOR/DatasetMidiCorridosTumbadosAcustico/blob/main/AutoMusicGen.ipynb">Google Colaboratory</a></p><h4>Por que fallamos</h4><p>Son varias las posibilidades pero lo que creemos fue que el modelo no consiguió sacar las características importantes de las canciones, esto debido a que el modelo funcionaba con un solo tipo de instrumento, y nosotros al tener un dataset no tan limpio como nos gustaría, este introducía ruido afectando a la fase de aprendizaje, además podría darse la posibilidad de que el modelo necesitara de más epoch para entrenar o más canciones.</p><h3><strong>¿Qué es una LSTM?</strong></h3><p>Las redes Long Short-Term Memory (LSTM), son un tipo de red neuronal recurrente, la cual extiende la memoria de las redes neuronales recurrentes, estas tienen el problema de memoria a corto plazo por lo que usan únicamente la información previa por lo que no tienen la lista de toda la información anterior disponible para el nodo neural. Debido a su característico comportamiento se utilizan para resolver problemas de reconocimiento de voz, reconocimiento de escritura a mano, composición de música, etc. problemas complejos que las redes neuronales recurrentes no podían resolver.</p><h3>¿Qué es una Wavenet?</h3><p>WaveNet es un tipo de red neuronal conocido como red neuronal convolucional profunda para generar audio muestra a muestra. La técnica es capaz de generar sonidos realistas como voces humanas modelando formas de onda directamente mediante redes neuronales entrenadas directamente a partir de muestras del habla humana real.​ La capacidad de Wavenet para generar formas de onda permite modelar cualquier clase de audio, incluyendo música.​ En WaveNet, la red toma las muestras de una señal como una entrada y sintetiza muestra a muestra la salida.</p><h3>¿Qué es una Interpolación?</h3><p>La interpolación consiste en realizar una suposición fundamentada con la información de un determinado conjunto de datos. Es una “mejor suposición” utilizando la información que se tiene a mano. La interpolación es algo para lo que usamos la IA a menudo. Alimentar datos a una computadora y permitir que haga conjeturas fundamentadas por nosotros. En teoría, la interpolación también es útil para extraer datos sobre situaciones y utilizar experiencias conocidas para expandir el conocimiento a áreas desconocidas. Sin embargo, esto se suele denominar extrapolación.</p><h3>Conclusión</h3><p>Nuestra conclusión es que hay que intentar mejorar nuestro dataset mucho más, si este hubiera estado más limpio, el output de nuestro segundo intento habría mejorado bastante.</p><p>También deberíamos de haber utilizado otro sistema además de google colab para correr más tiempo los modelos durante períodos de tiempo más largos.</p><h3>Repositorio de GitHub</h3><p><a href="https://github.com/MJVNOR/DatasetMidiCorridosTumbadosAcustico">MJVNOR/DatasetMidiCorridosTumbadosAcustico</a></p><h3>Bibliografía</h3><p>Hawthorne, C., Elsen, E., Song, J., Roberts, A., Simon, I., Raffel, C., … &amp; Eck, D. (2017). Onsets and frames: Dual-objective piano transcription. arXiv preprint arXiv:1710.11153.</p><ul><li><a href="https://deepai.org/machine-learning-glossary-and-terms/interpolation">Interpolation</a></li><li><a href="https://es.wikipedia.org/wiki/WaveNet">WaveNet - Wikipedia, la enciclopedia libre</a></li><li><a href="https://hub.packtpub.com/what-is-lstm/">What is LSTM? | Packt Hub</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28c74a0b23db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lcc-unison/como-fallar-en-generar-corridos-tumbados-mediante-ia-28c74a0b23db">Como fallar en generar “corridos tumbados” mediante IA</a> was originally published in <a href="https://medium.com/lcc-unison">LCC-Unison</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-time facial recognition application to identify Computer Science students at the University…]]></title>
            <link>https://medium.com/data-science/real-time-facial-recognition-application-to-identify-computer-science-students-at-the-university-of-253bc43d7c32?source=rss-f8554dc82461------2</link>
            <guid isPermaLink="false">https://medium.com/p/253bc43d7c32</guid>
            <category><![CDATA[face-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[mtcnn]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Martin vega]]></dc:creator>
            <pubDate>Wed, 19 May 2021 09:06:38 GMT</pubDate>
            <atom:updated>2021-05-24T21:37:18.022Z</atom:updated>
            <content:encoded><![CDATA[<h3>Real-time facial recognition application to identify Computer Science students at the University of Sonora using Pytorch and Multi-task Cascaded Convolutional Neural Networks (MTCNN)</h3><h4>Explanation and use, on how we completed our work</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zbPMmg3jYnNrkqr5q7jVng.png" /><figcaption>attributed from: <a href="https://www.freepik.es/vector-gratis/hombre-cara-escanear-tecnologia-digital-biometrica_5597121.htm#page=1&amp;query=face%20recognition&amp;position=6">Link de la imagen</a></figcaption></figure><p>Diana Laura Ballesteros Valenzuela, Eliud Gilberto Rodríguez Martínez, Hugo Francisco Cano Beyliss, Martín José Vega Noriega</p><p>Universidad de Sonora, Licenciatura, Hermosillo, Sonora,<br>México</p><p><a href="/service/mailto:mjvnor@outlook.com">mjvnor@outlook.com</a>, <a href="/service/mailto:dballesterosvalenzuela@gmail.com">dballesterosvalenzuela@gmail.com</a>, slabxcobra@gmail.com, eliud.giroma@gmail.com.</p><p>Github: <a href="https://github.com/MJVNOR/Real-time-facial-recognition-MTCNN/tree/main">https://github.com/MJVNOR/Real-time-facial-recognition-MTCNN/tree/main</a></p><p>Abstract. This project was made with the objective of detecting each computer science student at the University of Sonora in Hermosillo Sonora and with that to be able to carry out other projects based on being able to recognize their face. For this project we use image processing with neural networks for facial recognition. We implemented MTCNN which is a neural network that detects faces and facial landmarks in images and is one of the most accurate tools today.</p><p>Keywords: facial recognition, neural networks, pytorch</p><h3><strong>1. Introduction</strong></h3><p>The number of institutions that uses facial recognition and its implementation increases everyday, this is because it has many benefits, such as identifying lost people, identifying possible thieves, identifying which workers went to work on a certain day, etc. facial recognition is: given an image of an “unknown” face, finding an image of the same face in a set of “known” images.</p><p>Facial recognition has had several stages in which it made progress. We can see how before it was only a thing of fiction and today we can find it in many applications and places.</p><p>Woodrow Wilson Bledsoe is considered to be the pioneer of this technology since, in 1960 he worked on a system to classify the features of the human face, although the procedure was very manual, he used a stylus and coordinates to precisely locate the eyes, nose or mouth in people.</p><p>Today we can see how facial recognition is part of the artificial intelligence that allows us to detect and identify human faces.</p><p>In this work, the use of tools from the artificial intelligence area is implemented to be able to identify human faces in real time and also if this face is part of our data set of students and thus be able to recognize the person</p><h3><strong>2. Proposed Artifacts</strong></h3><p>Real-time facial recognition application.</p><p>2.1 Content modeling:</p><p>The first thing we did was use an MTCNN network to generate a dataset of the students’ faces and also have it return a probability that both the face is indeed a face (only one face per image) and this face is in the form of tensor.</p><pre>mtcnn0 = MTCNN(image_size=240, margin=0, keep_all=False, min_face_size=40)</pre><p>We use a default model (vggface2) to input our tensors dataset and to return the most important data of each face (in matrix form), this is called embedding.</p><pre>resnet = InceptionResnetV1(pretrained=&#39;vggface2&#39;.eval()</pre><pre>for img, idx in loader:<br>    face, prob <strong>=</strong> mtcnn0(img, return_prob<strong>=</strong>True)<br>    if face <strong>is</strong> <strong>not</strong> None <strong>and</strong> prob<strong>&gt;</strong>0.92:<br>        emb <strong>=</strong> resnet(face.unsqueeze(0))<br>        embedding_list.append(emb.detach())<br>        name_list.append(idx_to_class[idx])</pre><pre>data <strong>=</strong> [embedding_list, name_list]<br>torch.save(data, &#39;data.pt&#39;)</pre><p>In a cycle we access the device’s camera to begin with our recognition. We use a second network of type MTCNN, this one to detect if there is a face (if it detects a face it prints/draws a box around each face) and also gives us the probability that it is indeed a face, but this time the input of this network will be the frames of our camera (clarify that this network can detect more than one face at a time for each frame). We use the vggface2 model again to embed our detected faces.</p><p>With the embeddings of the saved faces and the embedding that was generated with the frames of our camera, we calculate the distances between the groups of the students’ faces against the faces that we have detected in our frame and with that we take the smallest distance and we say that the face is of some student.</p><pre>load_data <strong>=</strong> torch.load(&#39;data.pt&#39;)<br>embedding_list <strong>=</strong> load_data[0]<br>name_list <strong>=</strong> load_data[1]</pre><pre>cam <strong>=</strong> cv2.VideoCapture(0)</pre><pre>while True:<br>    img <strong>=</strong> Image.fromarray(frame)<br>    if img_cropped_list <strong>is</strong> <strong>not</strong> None:<br>        boxes, _ <strong>=</strong> mtcnn.detect(img)<br>        for i, prob in enumerate(prob_list):<br>            if prob<strong>&gt;</strong>0.90:<br>                emb <strong>= </strong>resnet(img_cropped_list[i].unsqueeze(0)).detach()<br>                dist_list <strong>=</strong> [] <br>                for idx, emb_db in enumerate(embedding_list):<br>                    dist <strong>=</strong> torch.dist(emb, emb_db).item()<br>                    dist_list.append(dist)</pre><pre>min_dist <strong>=</strong> min(dist_list)<br>                min_dist_idx <strong>=</strong> dist_list.index(min_dist) <br>                name <strong>=</strong> name_list[min_dist_idx]</pre><pre>box <strong>=</strong> boxes[i]<br>                original_frame <strong>=</strong> frame.copy()</pre><pre>if min_dist<strong>&lt;</strong>0.90:</pre><pre>frame <strong>=</strong> cv2.putText(frame, name<strong>+</strong>&#39; &#39;<strong>+</strong>str(min_dist), (box[0],box[1]), cv2.FONT_HERSHEY_SIMPLEX, 1, (63, 0, 252),1, cv2.LINE_AA)</pre><pre>frame <strong>=</strong> cv2.rectangle(frame, (box[0],box[1]) , (box[2],box[3]), (13,214,53), 2)</pre><pre>cv2.imshow(&quot;IMG&quot;, frame)</pre><h3>3. How the MTCNN works</h3><p>The MTCNN model was proposed in the paper “Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks” by Zhang, Zhang and Zhifeng. This model was made with the main goal of detecting its faces and features, it consists of three stages of convolutional networks.</p><p>The model is based on making use of multi-task learning in order to divide the detection tasks of faces and features, since doing this in the multi-task way improves generalization by taking advantage of the domain-specific information contained in the signals training of the related tasks, and this is done by training the tasks in parallel while using a shared representation of the information.</p><p>The first step before starting the stages is to take the image and resize it at different scales to build a pyramid of images, which is the entrance to the first part of the network.</p><p>The first stage consists of having a fully convolutional network (FCN), we use this instead of a CNN because it does not have a dense layer in its architecture. This proposed network is used to obtain the faces and their respective bounding box. The output of this network gives us all the possible candidates of the faces.</p><p>In the second stage of the network, the output of the first stage is used as input of a CNN network and this is called “The Refine Network (R-Net)”, this reduces the number of possible candidates, calibrates the bounding boxes and a non-maximum suppression (NMS) is done to unite our possible candidates. As an output it tells us if the input is a face or not.</p><p>Finally, the third is very similar to the R-Net, it gives us the position of the eyes, nose and mouth of our candidates.</p><h3>4. How we calculated the distance</h3><p>We used the Euclidean distance. The Euclidean distance represents the minimum distance between two points in a plane, it’s a straight line between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*tcA1gDE_Ckmq-JbOw9_M5w.png" /><figcaption><em>mage by author</em></figcaption></figure><p>This distance is calculated as the square root of the sum of the squared difference between the elements of two vectors, as indicated in the previous formula.</p><p>dist = torch.dist(emb,emb_db).item()</p><p>We give this function the face that we are detecting and we take the distance with each face from our database, from there we save all the distances and take the shortest one.</p><h3>4.1. Detection results</h3><p>This person does not exist, we took him from thispersondoesnotexist.com, we put him in our database and called him Andrew to show an example of the program working.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*SUeDQxwovHvhq_hd.png" /><figcaption>“Andrew” (non-existent person)</figcaption></figure><p>As we can see, the program detects “Andrew” with a distance of approximately 0.6237. The lower the value of the distance, the more likely that person is the one.</p><h3>5. Mistakes we made and errors we found</h3><p>We found that when we created the embeddings of the faces of our colleagues that we have in the database, one colleague’s computer ran out of memory, while another’s did not, in other words, you would have to have more than 1gb of memory (ram) available.</p><p>We also tried to implement multithreading and failed miserably.</p><h3>6. Conclusions and Future Work</h3><p>We can say that using an MTCNN and making use of embeddings to get the distances gave us very precise results (even with a small number of faces for each person), but the negative side is that when it detects a face the FPS decreases and when it detects you that you are “certain” person diminish even more.</p><p>We plan to change certain things in our application to achieve the lowest possible decrease in FPS (such as using another method in addition to calculating distances) and to be able to add more functionalities to the project in order to improve and make use of technology with our own colleagues.</p><h3>References</h3><ol><li>Zhang, K., Zhang, Z., Li, Z., &amp; Qiao, Y. (2016). Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10), 1499–1503. Lipschutz, S., Lars Lipson, M.: Set Theory: Discrete Mathematics. McGraw-Hill, pp. 1–22 (2007)</li><li>Caruana, R. Multitask Learning. Machine Learning 28, 41–75 (1997). <a href="https://doi.org/10.1023/A:1007379606734">https://doi.org/10.1023/A:1007379606734</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=253bc43d7c32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/real-time-facial-recognition-application-to-identify-computer-science-students-at-the-university-of-253bc43d7c32">Real-time facial recognition application to identify Computer Science students at the University…</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Aplicación de reconocimiento facial en tiempo real para identificar a alumnos de la licenciatura en…]]></title>
            <link>https://medium.com/lcc-unison/aplicaci%C3%B3n-de-reconocimiento-facial-en-tiempo-real-para-identificar-a-alumnos-de-la-licenciatura-en-2e2b53870995?source=rss-f8554dc82461------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e2b53870995</guid>
            <category><![CDATA[face-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Martin vega]]></dc:creator>
            <pubDate>Wed, 12 May 2021 05:41:02 GMT</pubDate>
            <atom:updated>2021-05-24T21:30:24.729Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Aplicación de reconocimiento facial en tiempo real para identificar a alumnos de la licenciatura en ciencias de la computación de la Universidad de Sonora usando Pytorch y redes neuronales convolucionales en cascada y en multitarea (MTCNN)</strong></h3><figure><img alt="Vector de Tecnología creado por gstudioimagen — www.freepik.es" src="https://cdn-images-1.medium.com/max/1024/1*O4c5fHW5HQaBuyon6BTy-A.jpeg" /><figcaption>attributed from: <a href="https://www.freepik.es/vector-gratis/hombre-cara-escanear-tecnologia-digital-biometrica_5597121.htm#page=1&amp;query=face%20recognition&amp;position=6">Link de la imagen</a></figcaption></figure><p>Diana Laura Ballesteros Valenzuela, Eliud Gilberto Rodríguez Martínez, Hugo Francisco Cano Beyliss, Martín José Vega Noriega.</p><p>Universidad de Sonora, Licenciatura, Hermosillo, Sonora,<br> México</p><p><a href="/service/mailto:mjvnor@outlook.com">mjvnor@outlook.com</a>, <a href="/service/mailto:dballesterosvalenzuela@gmail.com">dballesterosvalenzuela@gmail.com</a>, slabxcobra@gmail.com, eliud.giroma@gmail.com.</p><p>Github: <a href="https://github.com/MJVNOR/Real-time-facial-recognition-MTCNN/tree/main">https://github.com/MJVNOR/Real-time-facial-recognition-MTCNN/tree/main</a></p><p><strong>Resumen.</strong> Este proyecto fue hecho con el objetivo de detectar a cada estudiante de la licenciatura en ciencias de la computación de la universidad de Sonora en Hermosillo Sonora y con eso poder realizar otros proyectos a base de poder reconocer su rostro. Para este proyecto utilizamos el procesamiento de imágenes junto con las redes neuronales para el reconocimiento facial. Implementamos la utilización de MTCNN que es una red neuronal que detecta rostros y puntos de referencia faciales en imágenes y es una de las herramientas más precisas en la actualidad.</p><p><strong>Palabras clave: </strong>reconocimiento facial, redes neuronales, pytorch</p><p>Real-time facial recognition application to identify students of the Bachelor of Computer Science at the university of Sonora using pytorch and multi-task cascade convolutional neural network (MTCNN)</p><p><strong>Abstract. This project was made with the objective of detecting each student of the bachelor’s in computer science of the university of Sonora in Hermosillo Sonora and with that to be able to carry out other projects based on being able to recognize their face. For this project we use image processing together with neural network for facial recognition. We implemented the use of MTCNN which is a neural network that detects faces and facial landmarks in images and is one of the most accurate tools today.</strong></p><p><strong>Keywords: </strong>facial recognition, neural networks, pytorch</p><h3>1. Introducción</h3><p>Cada día aumenta el número de instituciones que se suman al uso del reconocimiento facial y su implementación ya que tiene muchos beneficios, como identificar personas perdidas, identificar posibles ladrones, identificar que trabajadores fueron a trabajar cierto día, etc. El objetivo de un sistema de reconocimiento facial es: dada una imagen de una cara “desconocida” encontrar una imagen de la misma cara en un conjunto de imágenes “conocidas”.</p><p>El reconocimiento facial ha tenido varias etapas en las que obtuvo avances. Podemos ver como antes solo era una cosa de ficción y hoy en día la podemos encontrar en muchas aplicaciones y lugares.</p><p>Se considera que Woodrow Wilson Bledsoe es el pionero de esta tecnología, ya que, en 1960 trabajo en un sistema para clasificar los rasgos del rostro humano, aunque le procedimiento era muy manual, utilizaba un lápiz óptico y coordenadas para situar los ojos, la nariz o la boca de forma precisa en las personas.</p><p>Hoy en día podemos ver como el reconocimiento facial forma parte de la inteligencia artificial que permite detectar e identificar rostros humanos.</p><p>En este trabajo, se implementa el uso de herramientas del área de inteligencia artificial para poder identificar en tiempo real rostros humanos y también si este rostro forma parte de nuestro data set de estudiantes y entonces así poder reconocer a la persona.</p><h3>2. Artefactos Propuestos</h3><p>Aplicación de reconocimiento facial en tiempo real.</p><p><strong>2.1</strong> <strong>Modelado de contenido:</strong></p><p>Lo primero que hicimos fue utilizar una red tipo MTCNN para generar un dataset de caras de los estudiantes y también que este nos devuelva una probabilidad de que tanto el rostro es efectivamente un rostro (solo un rostro por imagen) y este rostro esta en forma de tensor.</p><pre>mtcnn0 = MTCNN(image_size=240, margin=0, keep_all=False, min_face_size=40)</pre><p>Utilizamos un modelo predeterminado (vggface2) para darle de input nuestro dataset de tensores y que nos devuelva los datos más importantes de cada rostro (en forma de matriz), a esto se le llama embedding.</p><pre>resnet = InceptionResnetV1(pretrained=&#39;vggface2&#39;.eval()</pre><pre>for img, idx in loader:<br>    face, prob <strong>=</strong> mtcnn0(img, return_prob<strong>=</strong>True)<br>    if face <strong>is</strong> <strong>not</strong> None <strong>and</strong> prob<strong>&gt;</strong>0.92:<br>        emb <strong>=</strong> resnet(face.unsqueeze(0))<br>        embedding_list.append(emb.detach())<br>        name_list.append(idx_to_class[idx])</pre><pre>data <strong>=</strong> [embedding_list, name_list]<br>torch.save(data, &#39;data.pt&#39;)</pre><p>En un ciclo accedemos a la cámara del dispositivo para comenzar con nuestro reconocimiento. Utilizamos una segunda red de tipo MTCNN, esta para detectar si hay un rostro (si detecta un rostro imprime/dibuja un cuadro alrededor de cada rostro) y también que nos de la probabilidad de que efectivamente sea un rostro, pero esta vez el input de esta red será los frames de nuestra cámara (aclarar que esta red puede detectar más de un rostro a la vez por cada frame). De nuevo utilizamos el modelo vggface2 para hacer embeddings de nuestros rostros detectados.</p><p>Con los embeddings de los rostros guardados y el embedding que se generó con los frames de nuestra cámara, calculamos las distancias entre los grupos de los rostros de los alumnos contra los rostros que tenemos detectados en nuestro frame y con eso tomamos la distancia más chica y decimos que el rostro es de algún estudiante.</p><pre>load_data <strong>=</strong> torch.load(&#39;data.pt&#39;)<br>embedding_list <strong>=</strong> load_data[0]<br>name_list <strong>=</strong> load_data[1]</pre><pre>cam <strong>=</strong> cv2.VideoCapture(0)</pre><pre>while True:<br>    img <strong>=</strong> Image.fromarray(frame)<br>    if img_cropped_list <strong>is</strong> <strong>not</strong> None:<br>        boxes, _ <strong>=</strong> mtcnn.detect(img)<br>        for i, prob in enumerate(prob_list):<br>            if prob<strong>&gt;</strong>0.90:<br>                emb <strong>= </strong>resnet(img_cropped_list[i].unsqueeze(0)).detach()<br>                dist_list <strong>=</strong> [] <br>                for idx, emb_db in enumerate(embedding_list):<br>                    dist <strong>=</strong> torch.dist(emb, emb_db).item()<br>                    dist_list.append(dist)</pre><pre>                min_dist <strong>=</strong> min(dist_list)<br>                min_dist_idx <strong>=</strong> dist_list.index(min_dist) <br>                name <strong>=</strong> name_list[min_dist_idx] </pre><pre>                box <strong>=</strong> boxes[i]<br>                original_frame <strong>=</strong> frame.copy()</pre><pre>                if min_dist<strong>&lt;</strong>0.90:</pre><pre>                    frame <strong>=</strong> cv2.putText(frame, name<strong>+</strong>&#39; &#39;<strong>+</strong>str(min_dist), (box[0],box[1]), cv2.FONT_HERSHEY_SIMPLEX, 1, (63, 0, 252),1, cv2.LINE_AA)</pre><pre>                frame <strong>=</strong> cv2.rectangle(frame, (box[0],box[1]) , (box[2],box[3]), (13,214,53), 2)</pre><pre>    cv2.imshow(&quot;IMG&quot;, frame)</pre><h3>3. Como funciona la MTCNN</h3><p>El modelo de MTCNN fue propuesto en el paper “Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks” de Zhang, Zhang and Zhifeng. Este modelo fue hecho con la principal meta de detectar rostros y facciones de este, este consta de tres etapas de redes convolucionales.</p><p>El modelo esta basado en hacer uso del aprendizaje multi-task para así dividir las tareas de detección de rostros y facciones, ya que al hacer esto en la forma multi-task se mejora la generalización al aprovechar la información específica del dominio contenido en las señales de entrenamiento de las tareas relacionadas, y esto lo hace entrenando las tareas en paralelo mientras usa una representación compartida de la información.</p><p>El primer paso antes de empezar las etapas es tomar la imagen y cambiar su tamaño a diferentes escalas para construir una pirámide de imágenes, que sea la entrada de la primer parte de la red.</p><p>La primer etapa consta de tener una red tipo fully convolutional network (FCN) usamos esta en vez de una CNN porque no tiene una capa densa en su arquitectura. Esta red propuesta se utiliza para obtener los rostros y su respectiva bounding box. La salida de esta red nos da todos los posibles candidatos de los rostros.</p><p>En la segunda etapa de la red, se utiliza la salida de la primer etapa como entrada de una red CNN y a esta se le llamada “The Refine Network (R-Net)”, esta reduce el numero de posibles candidatos, calibra las bounding box y se hace una supresión no máxima (NMS) para unir nuestros posibles candidatos. Como salida nos dice si la entrada es una cara o no.</p><p>Por ultima la tercer es muy similar a la R-Net, esta nos da como salida la posición de los ojos nariz y boca de nuestros candidatos.</p><h3>4. ¿Cómo calculamos la distancia?</h3><p>Usamos la distancia euclidiana. La distancia euclidiana representa la menor distancia entre dos puntos en un plano, es una linea recta entre los dos puntos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*tcA1gDE_Ckmq-JbOw9_M5w.png" /></figure><p>Esta distancia es calculada como la raíz cuadrada de la suma de la diferencia cuadrada entre los elementos de dos vectore, como indicamos en la formula anterior.</p><p>dist = torch.dist(emb,emb_db).item()</p><p>Le damos a la función la cara que estamos detectando y tomamos la distancia entra cara y cara de nuestra base de datos, luego guardamos todas las distancias y tomamos la mas corta.</p><h3>4.1. Resultados de la detección</h3><p>Esta persona no existe, la tomamos de <a href="https://thispersondoesnotexist.com/">thispersondoesnotexist.com</a>, la metimos a nuestra base de datos y le llamamos Andrew para mostrar un ejemplo del programa funcionando.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*OSuaVVDXxv_dY99H.png" /><figcaption>“Andrew” (persona inexistente)</figcaption></figure><p>Como podemos observar el programa detecta a “Andrew” con una distancia de aproximadamente 0.6237. Mientras menor sea el valor de la distancia, mas seguro es que sea esa persona.</p><h3>5. Errores que cometimos y que encontramos</h3><p>Encontramos que cuando creamos los embeddings de las caras de nuestro compañeros que tenemos en nuestra base de datos, una computadora de un compañero de equipo se quedó sin memoria, mientras que a otro no, en otras palabras, debes de tener mas de 1gb de memoria (RAM) disponible.</p><p>También intentamos implementar un sistema multi-hilos pero fallamos miserablemente. Podemos atribuir esto a la falta de experiencia en el área.</p><h3>6. Conclusiones y Trabajo a Futuro</h3><p>Podemos decir que utilizar una MTCNN y hacer uso de embeddings para sacar las distancias nos dio resultados muy precisos (incluso con una cantidad pequeña de rostros por cada persona), pero el lado negativo es que cuando detecta un rostro los FPS disminuyen y cuando te detecta que eres “cierta” persona disminuyen aún más.</p><p>Planeamos cambiar ciertas cosas de nuestra aplicación para lograr la menor disminución posible de los FPS (como utilizar otro método además de calcular las distancias) y poder agregarle más funcionalidades al proyecto con el objetivo de mejorar y de hacer uso de la tecnología con nuestros propios compañeros.</p><h3>Referencias</h3><p>1. Zhang, K., Zhang, Z., Li, Z., &amp; Qiao, Y. (2016). Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10), 1499–1503. Lipschutz, S., Lars Lipson, M.: Set Theory: Discrete Mathematics. McGraw-Hill, pp. 1–22 (2007)</p><p>2. Caruana, R. Multitask Learning. Machine Learning 28, 41–75 (1997). <a href="https://doi.org/10.1023/A:1007379606734">https://doi.org/10.1023/A:1007379606734</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e2b53870995" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lcc-unison/aplicaci%C3%B3n-de-reconocimiento-facial-en-tiempo-real-para-identificar-a-alumnos-de-la-licenciatura-en-2e2b53870995">Aplicación de reconocimiento facial en tiempo real para identificar a alumnos de la licenciatura en…</a> was originally published in <a href="https://medium.com/lcc-unison">LCC-Unison</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[La manera mas facil y rapida de subir a la Web tus modelos de inteligencia artificial utilizando…]]></title>
            <link>https://medium.com/lcc-unison/la-manera-mas-facil-y-rapida-de-subir-a-la-web-tus-modelos-de-inteligencia-artificial-utilizando-c0e8717c30dd?source=rss-f8554dc82461------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0e8717c30dd</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[spanish]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[streamlit-app-deployment]]></category>
            <category><![CDATA[web-applications]]></category>
            <dc:creator><![CDATA[Martin vega]]></dc:creator>
            <pubDate>Tue, 12 Jan 2021 01:15:30 GMT</pubDate>
            <atom:updated>2021-01-15T01:00:57.455Z</atom:updated>
            <content:encoded><![CDATA[<h3>La manera mas facil y rapida de subir a la Web tus modelos de inteligencia artificial utilizando Python y Streamlit</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aPTgfOfZSVSYOSakvNDJoQ.jpeg" /></figure><p>En este blog te explicare como subir tus modelos de inteligencia artificial utilizando la aplicación Streamlit Sharing y todo esto usando Python y GitHub.</p><p>Pasos a seguir:</p><ul><li>Primero que nada debes tener un modelo el cual subir y que busques que la gente pruebe, en este ejemplo utilizaré un modelo que cree para clasificar tweets si son pro o en contra del gobierno (todo el código estará gratis en GitHub)</li></ul><pre>#Este es como se crea el modelo con sklearn (ya sabes que despues lo tienes que entrenar y probar)</pre><pre><strong>from</strong> <strong>sklearn.svm</strong> <strong>import</strong> SVC <em># Maquina de vector de soportes</em></pre><pre>ClasificadorTweets = SVC(gamma=0.01, C=100,probability=<strong>True</strong>, kernel= &#39;rbf&#39;)</pre><ul><li>Después de eso lo vas a guardar en un formato .pkl con la librería pickle ya que vamos a crear un nuevo archivo para crear la aplicación web de tu modelo.</li></ul><pre>#importamos librerías<br><strong>import</strong> <strong>pickle</strong><br><strong>from</strong> <strong>pickle</strong> <strong>import</strong> dump</pre><pre>#Le damos el nombre que queremos<br>nombreArchivo = &#39;ClasificadorTweets.pkl&#39;</pre><pre>#Lo guardamos ya como .pkl<br>pickle.dump(ClasificadorTweets, open(nombreArchivo, &#39;wb&#39;))</pre><ul><li>Ahora vamos a crear nuestra página web, donde primero crearemos un archivo nuevo .py en mi caso lo llamaré TutorialStreamlit.py:</li></ul><pre>#Empezaremos bajando las librerías necesarias</pre><pre>#Librería para usar los componentes de Streamlit<br>import streamlit as st</pre><pre>#librería para poder desempaquetar nuestro modelo<br>import pickle<br>from pickle import dump</pre><pre>#Con st.title creamos un título para nuestra aplicación <br>st.title(&#39;Como subir GRATIS a la Web tus modelos de inteligencia artificial&#39;)</pre><pre>#Dentro de la librería tenemos la opción de poder escribir con<br>#markdown<br>st.markdown(&#39;&#39;&#39;<br>Este es un ejemplo para poder usar Streamlit con un modelo que clasifica texto según que sí está en contra o a favor del gobierno Mexicano.<br>&#39;&#39;&#39;)</pre><pre>#<em>Aquí </em>desempacamos nuestro modelo<br>nombreArchivo = &#39;ClasificadorTweets.pkl&#39;<br>modeloCargado = pickle.load(open(nombreArchivo, &#39;rb&#39;))</pre><pre>#Lo guardo en una dupla porque yo guarde mi modelo y para poder convertir el texto utilizo Tfidf<br>ClasificadorTweets, Tfidf_vect = modeloCargado</pre><pre><em>#Con st.text_input hacemos un widget para tener un input del usuario<br></em>input = st.text_input(label=&#39;Inserte su texto&#39;, value=&#39;Amlo presidente&#39;)</pre><pre>#Convertimos el input a Tdfidf<br>prediccionEjemplo = Tfidf_vect.transform([input])</pre><pre>resultadoPorcentaje =                                                            ClasificadorTweets.predict_proba(prediccionEjemplo)</pre><pre>aux = []<br>for i in resultadoPorcentaje:<br>    for j in i:<br>        conservador = j<br>        aux = [j]</pre><pre>if conservador &lt;= 0.5:<br>    <em>#Con st.write mostramos texto en pantalla<br>    </em>st.write(&#39;Resultado: A favor del gobierno&#39;)<br>else:<br>    st.write(&#39;Resultado: En contra del gobierno&#39;)</pre><pre>st.write(&#39;Result: {}&#39;.format(resultadoPorcentaje))</pre><p>Para poder correr nuestra aplicación utilizamos la consola de comandos y escribimos:</p><pre>streamlit run TutorialStreamlit.py</pre><p>Se nos debe de abrir una pestaña en nuestro navegador con nuestra aplicación terminada:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WNnu_ikzr550SKBNI5FKmg.png" /></figure><p>Para cerrar el proceso lo único que tenemos que hacer es ir a la consola de comandos y presionamos ctrl + c</p><ul><li>De ahí tenemos que encontrar que librerías necesitamos para que nuestro modelo funcione en la web y esto es muy fácil si utilizaste un ambiente virtual, pero si no lo único negativo es que vas a tardar más en encontrarlas y para esto nos vamos a ir a nuestra terminal donde nos vamos a dirigir a escribir dos comandos:</li></ul><pre>pip freeze &gt; requirements.txt<br>cat requirements.txt</pre><p>Se nos creará un archivo requirements.txt donde nos aparecerán todas las bibliotecas de python que tenemos instaladas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*W_u7BZqZclvOb7pmalf-xw.png" /></figure><ul><li>Consiguiente borraremos las bibliotecas que no necesitamos para quedarnos con las que funcionan con nuestro modelo, donde en mi caso me quedaría con estas:</li></ul><pre>pickle-mixin==1.0.2<br>pickleshare==0.7.5<br>streamlit==0.71.0<br>scikit-learn==0.23.2</pre><ul><li>Ahora subiremos nuestra carpeta de trabajo a github, donde debemos tener el archivo de requirements.txt, el modelo a usar y nuestro programa en python.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zwfcjsIXqPHykI6iKE_L2A.png" /></figure><ul><li>Vamos a irnos a <a href="https://www.streamlit.io/">Streamlit</a>, nos registramos con Github</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zdOVxOcmTW7OUeHwoKiQwg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bI-lf8hKGX7kia6EZVLOVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_xQOSUVfMHfxo6_gakxwkA.png" /></figure><ul><li>Algo importante es que si te acabas de registrar muy probablemente no tengas acceso a poder subir tu aplicación web, pero no te preocupes, lo único que tienes que hacer es solicitar acceso y en cuestión de unos pocos días te lo darán.</li></ul><p>A continuación te pongo una serie de imágenes que serán los pasos para solicitar el acceso:</p><ol><li>Debes picarle al primer link de la imagen.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1hrkWFKk4dUAiIVnX2eecA.png" /></figure><p>2. Luego tendrás que rellenar un formulario para que te otorguen acceso.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GgBGlkdMpnA0hcQqcr-hmg.png" /></figure><p>3. Terminando el formulario esperarás a que te contacten por correo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B_dzLKIxI-QYPmeQdy6RLA.png" /></figure><ul><li>Ya que te hayas registrado y te hayan dado acceso para poder subir aplicaciones, te irás a este link <a href="https://share.streamlit.io/">https://share.streamlit.io/</a> y esta páginase te debe de mostrar:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PofObB-NXAVp0l4gakDanA.png" /></figure><ul><li>Tendrás que picarle al botón que dice ‘New app’, donde te saldra un menu desplegable donde haremos click a la casilla que dice ‘From existing repo’</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7x1_7_F1tMSi7tqjzcQzvA.png" /></figure><ul><li>Luego seleccionaremos el repositorio en donde tenemos los archivos que subimos a github, seleccionamos la Brach en la que tenemos nuestros archivos, casi por último sería en donde dice ‘Main file path’, darle el nombre de nuestro archivo donde contiene nuestra aplicación web y ya entonces haremos click en el botón de ‘Deploy!’ para que inicie nuestra aplicación web.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*neHi9OAiREL1PwOIOLhinw.png" /></figure><ul><li>Aquí dejamos que Streamlit descargue los documentos necesarios.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yaTJne1p81smSzQV8KScwQ.png" /></figure><ul><li>Para finalizar, tendremos corriendo nuestra aplicación web con nuestro modelo de inteligencia artificial en la red.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QZDHmKUz0fR81lKJBxb6JA.png" /></figure><ul><li><a href="https://share.streamlit.io/mjvnor/streamlitwebtutorial/TutorialStreamlit.py">[Link del modelo visto]</a></li><li><a href="https://github.com/MJVNOR/StreamLitWebTutorial">[Link del repositorio en github]</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0e8717c30dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lcc-unison/la-manera-mas-facil-y-rapida-de-subir-a-la-web-tus-modelos-de-inteligencia-artificial-utilizando-c0e8717c30dd">La manera mas facil y rapida de subir a la Web tus modelos de inteligencia artificial utilizando…</a> was originally published in <a href="https://medium.com/lcc-unison">LCC-Unison</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What happen if you mix Twitter, the president of México and natural language processing?]]></title>
            <link>https://medium.com/lcc-unison/what-happen-if-you-mix-twitter-the-president-of-m%C3%A9xico-and-natural-language-processing-bdb015959503?source=rss-f8554dc82461------2</link>
            <guid isPermaLink="false">https://medium.com/p/bdb015959503</guid>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[politica-mexico]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[pattern-recognition]]></category>
            <category><![CDATA[amlo]]></category>
            <dc:creator><![CDATA[Martin vega]]></dc:creator>
            <pubDate>Tue, 08 Dec 2020 05:49:30 GMT</pubDate>
            <atom:updated>2020-12-12T16:53:04.798Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*MCUUont22AOtTu8IYA5Xzw.jpeg" /></figure><p>Nowdays, there is a lot of information in twitter and a bunch of people using this social media dialy (which creates more information) so… what happen if you grab a bunch of tweets that have words related to the politics of México and mainly, of course, Andrés Manuel López Obrador (actual president of México) and do an analysis? in this post we will show you the analysis that we did in the courses of natural language processing and pattern recognition in the university of Sonora.</p><h4>Natural Language Processing Teacher:</h4><ul><li>Olivia Carolina Gutú Ocampo</li></ul><h4>Pattern Recognition Teacher:</h4><ul><li>Julio Waissman Vilanova</li></ul><h4>This analysis was done by the students:</h4><ul><li>Jesús Armando Báez Camacho</li><li>Diana Laura Ballesteros Valenzuela</li><li>Martín José Vega Noriega</li></ul><h4>Github repository:</h4><ul><li><a href="https://github.com/MJVNOR/MexicanPoliticsTweetClassification">https://github.com/MJVNOR/MexicanPoliticsTweetClassification</a></li></ul><h3>Spoiler alert!</h3><p>Our main goal was to train a model to tell us if a text was for or against the goverment of AMLO.</p><p>The evaluation criterio of our project is that our model yields a <strong>percentage</strong> of how much for or against a text is about the actual goverment given by a user.</p><p><a href="https://share.streamlit.io/mjvnor/mexicanpoliticstweetclassification/main/MexicanPoliticsTweetsRecognition.py">Test our final model</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*rpkBoqWkYJmVF7CMtEhHLw.png" /></figure><h3>Here’s how we did it!</h3><p>The firts thing we did was the information gathering, using the twitter API we collect as many tweets as we could (collecting 7985 tweets in total).</p><pre>df = pd.read_csv(‘../data/tweets_base_madre.csv’, encoding =’utf-8&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*s17FZKLFg0SzLreQyAo8Eg.png" /></figure><p>Then, we classify them manually, creating a new column in our data frame named “clasification”. In that column we putted a number from 0 to 3, depending the content of the tweet.</p><ul><li>0 — If the content of the tweet was in favor of the goverment</li><li>1 — If the content of the tweet was against the goverment</li><li>2 — If the content of the tweet was neutral</li><li>3 — If the content of the tweet wasn’t related to the subject.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*9w4GaCCbR1CzqolO9xFIbQ.png" /></figure><p>Before going deeper with tokenization, classification, etc. we did an analysis of which hashtags and arrobas people use when they mention AMLO and also the number of likes that the tweets have depending on their classification using the column “full_text”.</p><h4>In general.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JcXUBtczRQndTERlTNEpcQ.png" /></figure><h4>Hashtags.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uM8sz96E1oQjzOwAnhJaGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wPwhP1rLLx5XdQUf7T0LCw.png" /></figure><h4>Likes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D0hv63svtZjenMH9dDHZOA.png" /></figure><h3>Now, let’s go deeper….</h3><p>After finishing the analysis of the hashtags, arrobas and likes, the next thing we did was create a new column with the name of “cleanTweets”. In that column we applied some cleaning (with the help of regular expressions); the cleaning we did consisted of:</p><ul><li>Lemmatize.</li><li>Transform all the text to lowercase.</li></ul><p>Remove:</p><ul><li>Accents.</li><li>Stop words.</li><li>At symbols ( @ ).</li><li>Hashtag symbols (#).</li><li>Hypertexts.</li><li>Suspension points.</li><li>Isolated characters.</li><li>Isolated numbers.</li><li>Dates.</li><li>Possible laughs (ja, jaja, ha, haha, je, jeje, etc).</li><li>Certain symbols (symbols like {,}, [,], “ , ” , …)</li></ul><pre>def cleanTxt(text):     <br>    text = text.lower()     text = re.sub(r&#39;@&#39;, &#39;&#39;, text) <br>    text = re.sub(r&#39;#&#39;, &#39;&#39;, text) <br>    text = re.sub(&#39;(?:(?:https?|ftp):\/\/)?[\w/\-?=%.]+\.[\w/\-?    =%.]+&#39;,&#39;&#39;,text)     <br>    text = re.sub(&#39;[ (){}\[\]\|,;&quot;\“\”\‘\’&#39;\«\»!¡?¿]&#39;, &#39;&#39;, text) <br>    text = re.sub(&#39;\.[\.]+&#39;, &#39;&#39;, text)     <br>    text = re.sub(&#39;\s.\s&#39;, &#39;&#39;, text)     <br>    text = re.sub(&#39;\s[0-9]+\s&#39;, &#39;&#39;, text)    <br>    text = re.sub(&#39;\d\d/\d\d/\d\d|\d\d/\d\d/\d\d\d\d&#39;, &#39;&#39;, text)          <br>    text = re.sub(r&#39;\s([jJ][AaEeIi])+\s&#39;, r&#39; &lt;risa&gt; &#39;, text)    <br>    text = re.sub(r&#39;(ja|je|ji|JA|JE|JI|\s a|\s e|\s i)([jJ][AaEeIi])(\w?)&#39;, r&#39; &lt;risa&gt; &#39;, text)     <br>    text = re.sub(r&#39;\s((ha|Ha|he|HE)[hH][AaEeIi])+\w&#39;, r&#39; &lt;risa&gt; &#39;, text)     <br>    return text<br> <br>df[‘CleanTweets’] = df[‘full_text’].apply(cleanTxt)<br>df[&#39;CleanTweets&#39;].dropna(inplace=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mhqk9qN3RMhyCMT_gUOhLg.png" /></figure><p>Note: Before taking the next step, we made sure that there were no repeated tweets (in case there was more than one of the same, we would keep the first one and delete the others)</p><p>The next step we did was to create another column named “TokenizeTweetsTidy_text”, and that column contains the information of the column “cleanTweets” mentioned before BUT we applied tokenization to that column.</p><pre>#Tokenizar<br>df[&#39;TokenizeTweetsTidy_text&#39;] = df.apply(lambda row: nltk.word_tokenize(row[&#39;CleanTweets&#39;]), axis=1)<br>#comillas a las palabras<br>df[&#39;TokenizeTweetsTidy_text&#39;] = df.TokenizeTweetsTidy_text.astype(str)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dfcs_-8sQ1ZkCQQEmR8TOg.png" /></figure><p>Next, we took (from the column “clasification”) those tweets that have clasification=0 and clasification=1 and we created a new dataframe with those two together.</p><pre>liberales = df[df[&quot;clasification&quot;]==0] <br>conservadores  = df[df[&quot;clasification&quot;]==1]</pre><pre>libCon = pd.concat([liberales,conservadores]).reset_index()</pre><p>In the dataFrame libCon are 1639 Tweets of the class 0 (Liberales) and 3164 Tweets of the class 1 (Conservadores), giving us a total of 4803 Tweets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rygEIF1bV1mzZ5uECImGkg.png" /></figure><h3>Then, we vectorize the tweets that are in the column “TokenizeTweetsTidy_text” with Tf-idf.</h3><p>Unigrams, bigrams and trigrams are used, the minimum frequency of a word has to be repeated 5 times in the documents to be taken into account and if a term is repeated more than 90% in the documents, it is eliminated.</p><p>To see the tf-idf data set we use TSNEVisualizer, this one creates an inner transformer pipeline that applies such a decomposition first (SVD with 50 components by default), then performs the t-SNE embedding. The visualizer then plots the scatter plot, coloring by cluster or by class, or neither if a structural analysis is required.</p><pre>def GraficaData(df,nombreCol):     <br>  with warnings.catch_warnings():         <br>#ignore all caught warnings  <br>  warnings.filterwarnings(&quot;ignore&quot;)         <br># execute code that will generate warnings         <br>  Train_X, Test_X, Train_Y, Test_Y = model_selection.train_test_split(df[nombreCol],df[&#39;clasification&#39;],test_size=1)         <br>  y = Train_Y         <br>  Encoder = LabelEncoder()         <br>  Train_Y = Encoder.fit_transform(Train_Y)         <br>  Test_Y = Encoder.fit_transform(Test_Y)          <br>  Tfidf_vect = TfidfVectorizer(max_features=50000)  <br>  Tfidf_vect.fit(df[nombreCol])          <br>  Train_X_Tfidf = Tfidf_vect.transform(Train_X)         <br>  Test_X_Tfidf = Tfidf_vect.transform(Test_X)  <br>#print(Tfidf_vect.vocabulary_)         <br>  len(Tfidf_vect.vocabulary_)         <br>#print(Train_X_Tfidf)         <br>  warnings.filterwarnings(&quot;ignore&quot;)        <br>  tsne = TSNEVisualizer(cmap=&#39;PuOr&#39;)   <br>  warnings.filterwarnings(&quot;ignore&quot;)         <br>  tsne.fit(Train_X_Tfidf, y)   <br>  warnings.filterwarnings(&quot;ignore&quot;)         <br>  tsne.show()</pre><pre>GraficaData(libCon,&#39;TokenizeTweetsTidy_text&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VBsu5YRfHgy2MogoNvM7uw.png" /></figure><h3>¡Here starts the clasification!</h3><p>To classify we used logistic regression and SVM to see which one classifies the best.</p><h4>Logistic regression.</h4><p>Why did we choose the logistic regression model? because it is the easiest model to train and also gives a good (and fast) classification.</p><h4>SVM (Support-Vector Machine)</h4><p>Why did we choose the SVM model? because is a fast and dependable classification algorithm that performs very well with a limited amount of data to analyze.</p><h4>Before going into the clasification… Using the GridSearchCV technique we obtained the best values ​​for the classification in the two models.</h4><ul><li>In the case of logistic regression: The results were using: RegLog = ‘C’: 5, ‘max_iter’: 100, ‘solver’: ‘saga’.</li><li>For the support vector machine the results were: ‘C’: 100, ‘gamma’: 0.01, ‘kernel’: ‘rbf&#39;</li></ul><p>Next, what happen if we want to know (passing specific percentage of dataset) the degree of success achieved of our chosen model?</p><blockquote>We can know that by doing the learning curve!</blockquote><pre># Creating CV training and test scores for various training set sizes<br>X, _, y, _, Tfidf_vect = separaData(libCon,&#39;TokenizeTweetsTidy_text&#39;,1)<br>cv = StratifiedKFold(n_splits=10)<br>estimator = LogisticRegression(solver=&#39;saga&#39;,max_iter=100 , C=5)<br>train_sizes, train_scores, test_scores = learning_curve(estimator,X, y, cv=cv, scoring=&#39;f1_weighted&#39;, n_jobs=-1,train_sizes=np.linspace(0.3, 1.0, 10))<br><br># Creating means and standard deviations of training set scores<br>train_mean = np.mean(train_scores, axis=1)<br>train_std = np.std(train_scores, axis=1)<br><br># Creating means and standard deviations of test set scores<br>test_mean = np.mean(test_scores, axis=1)<br>test_std = np.std(test_scores, axis=1)<br><br>fig = go.Figure()<br><br>layout = go.Layout(title=&#39;Learning Curve Logistic Regression&#39;,<br>                xaxis=dict(title=&#39;Training Set Size&#39;),<br>                yaxis=dict(title=&#39;Accuracy Score&#39;))<br><br>fig = go.Figure(layout=layout)<br><br><br>fig.add_trace(go.Scatter(<br>    x=train_sizes, y=train_mean,<br>    line_color=&#39;rgb(230,171,2)&#39;,<br>    line = dict(width=4, dash=&#39;dash&#39;),<br>    name=&#39;Training score&#39;,<br>))<br><br>fig.add_trace(go.Scatter(<br>    x=train_sizes, y=test_mean,<br>    line_color=&#39;rgb(117,112,179)&#39;,<br>    line = dict(width=4),<br>    mode=&#39;lines+markers&#39;,<br>    name=&#39;Cross-validation score&#39;,<br>))<br><br>fig.update_traces(mode=&#39;lines&#39;)<br>fig.write_html(&quot;LearningCurveRegLogistic.html&quot;)<br>fig.show()</pre><pre>fig.update_traces(mode=&#39;lines&#39;) <br>  fig.write_html(&quot;LearningCurveSVM.html&quot;) <br>  fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*mjzqo5rEPwaCMGW72dbxMQ.png" /></figure><pre> # Ploting Learning Curve<br>    # Creating CV training and test scores for various training set sizes<br>    X, _, y, _, Tfidf_vect = separaData(libCon,&#39;TokenizeTweetsTidy_text&#39;,1)<br>    cv = StratifiedKFold(n_splits=10)<br>    estimator = SVC(gamma=0.01, C=100,probability=True, kernel= &#39;rbf&#39;)<br>    train_sizes, train_scores, test_scores = learning_curve(estimator,X, y, cv=cv, scoring=&#39;f1_weighted&#39;, n_jobs=-1,train_sizes=np.linspace(0.3, 1.0, 10))<br><br>    # Creating means and standard deviations of training set scores<br>    train_mean = np.mean(train_scores, axis=1)<br>    train_std = np.std(train_scores, axis=1)<br><br>    # Creating means and standard deviations of test set scores<br>    test_mean = np.mean(test_scores, axis=1)<br>    test_std = np.std(test_scores, axis=1)<br><br>    fig = go.Figure()<br><br>    layout = go.Layout(title=&#39;Learning Curve SVM&#39;,<br>                    xaxis=dict(title=&#39;Training Set Size&#39;),<br>                    yaxis=dict(title=&#39;Accuracy Score&#39;))<br><br>    fig = go.Figure(layout=layout)<br><br><br>    fig.add_trace(go.Scatter(<br>        x=train_sizes, y=train_mean,<br>        line_color=&#39;rgb(230,171,2)&#39;,<br>        line = dict(width=4, dash=&#39;dash&#39;),<br>        name=&#39;Training score&#39;,<br>    ))<br><br>    fig.add_trace(go.Scatter(<br>        x=train_sizes, y=test_mean,<br>        line_color=&#39;rgb(117,112,179)&#39;,<br>        line = dict(width=4),<br>        mode=&#39;lines+markers&#39;,<br>        name=&#39;Cross-validation score&#39;,<br><br>    ))<br>.<br>    fig.update_traces(mode=&#39;lines&#39;)<br>    fig.write_html(&quot;LearningCurveSVM.html&quot;)<br>    fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*5KMlIEOuvwaEgMSwIRbBFg.png" /></figure><p>According to the learning curves of the two models, the two graphs tell us the same thing: more training data is needed to be able to have a smaller error.</p><h3>This is the function that we are going to use to train</h3><pre>def ModeloClasificadorKFoldCV(modeloToTrain, df,nombre):<br>    X, _, y, _, Tfidf_vect = separaData(df,&#39;TokenizeTweetsTidy_text&#39;,1)<br><br>    if modeloToTrain == LogisticRegression:<br>        myModelo = modeloToTrain(solver=&#39;saga&#39;,max_iter=100 , C=5)<br>    elif modeloToTrain == SVC:<br>        myModelo = modeloToTrain(gamma=0.01, C=100,probability=True, kernel= &#39;rbf&#39;) <br><br>    cv = KFold(n_splits=10, random_state=None, shuffle=False)<br>    scores = []<br><br>        for train_index, test_index in cv.split(X):<br>            Train_X_Tfidf, Test_X_Tfidf, Train_Y, Test_Y = X[train_index], X[test_index], y[train_index], y[test_index]<br>            myModelo.fit(Train_X_Tfidf, Train_Y)<br>            scores.append(myModelo.score(Test_X_Tfidf, Test_Y))</pre><p>The first thing we did was define our k-fold and after that we trained! Then we got the predictions, the confusion matrix and the Roc curve of the model.</p><h3>Now, we have two options, to test our data without stemming and stemming.</h3><h3>First the results without stemming:</h3><p>Logistic regression:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*8h-OJidp-WpQoNbVfqdn-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*I5VG6sRNcl8kkmup0kNnlQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*EhTlAmI6k1R8HgVx_o-3VA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*vgh0lXAtXCymbIObjqrLAA.png" /></figure><p>SVM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*shZ8MkuPqE04csxc03hWPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*GyjRV8zRLSkbkd4OQ9A1JA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*PXKOYUbOrxDNQzvVnMvHIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*GGCPdBoGmCOYhL4qj9yZuQ.png" /></figure><h3>Now, we will see if changing to the best threshold improves the f1 score:</h3><h4>Logistic regression:</h4><p>Best Threshold=0.681871, G-Mean=0.708</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*o6KUOmtjZjUqbEbl0ikr7Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*EeD_QNm73XEl-ITf0Euqzw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*emH1diGPpJ_W7R9TIPdm7w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*b73zDSpmwlNqOInIYVAuqg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*CgT16WYWlQc6ho1D1bM77Q.png" /></figure><h4>SVM:</h4><p>Best Threshold=0.691281, G-Mean=0.659</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*CCIbT3F46fqDM7wCWlpXGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*wLMteJfBnoqiLIxkVmtpbw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*hmW3G7osWMrbcTaQ5yRN-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*8nQktkCX5l3BTKuL79lVeA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*hpXF2nDH59FdGsyiIfkngg.png" /></figure><p>What can be seen by observing the f1 score of the logistic regression is that changing the threshold is negatively affected and the same can also be observed with the support vector machine. So we will discard the option to modify the threshold.</p><h3>Now the results using stemming:</h3><h4>Logistic regression:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*BeQ_ouo0eW9LX_Lqlt44Dw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*bgUTigpIXE7oyE3DEP6RdQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*ux0aapSF6PhR_5g8-Vkmqg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*YrTgduFHSRgb6gPFo4u2Ow.png" /></figure><p>SVM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*ZfBlGe_H7_0lG6iDylOIMQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*psOSgbk9E8PmVwTKbjDAbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*RQOH73yBYJ1A4o_WPqvbbQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*c_WZjowsEFJoMZXhVtWEuQ.png" /></figure><p>Now looking at the results of when we lemmatize to when we do not, we can realize that if we do so we have a small loss in the f1 score both in the logistic regression and in the svm, that is why we will make the decision to train without lemmatizing.</p><p>As can be seen in the graphs, we can conclude that the logistic regression classifies conservative tweets very well, but liberal tweets very poorly, this is observed from the confusion matrix and the f1 score. From the roc curve, we can say that our classes, especially, the classification of the “liberal” class is very bad. For this we will try to solve our problems using another model which would be the SVM, since we want to improve the f1 score of the “liberal” class.</p><p>The results of the SVM do not differ much from the logistic regression model, the good thing here is that it increased what we wanted; the f1 score of the “liberal” class.</p><p>So after looking at the SVM and logistic regression results, we will use all the available data to classify a final model using SVM:</p><pre>def ModeloClasificadorFinal(modeloToTrain, df,nombre):<br><br>    if modeloToTrain == LogisticRegression:<br>        myModelo = modeloToTrain(solver=&#39;saga&#39;,max_iter=100 , C=5)<br>    elif modeloToTrain == SVC:<br>        myModelo = modeloToTrain(gamma=0.01, C=100,probability=True, kernel= &#39;rbf&#39;) <br><br>    myModelo.fit(Train_X_Tfidf,Train_Y)<br><br>    return myModelo<br><br>modelSVCFinal = ModeloClasificadorFinal(SVC, libCon, &quot;Maquina de vector de soportes&quot;)</pre><h3>Conclusion</h3><p>After seeing the results of the two models, we can conclude that to improve our model we must have more “liberal” tweets, since it can be seen in the two models that due to this problem our f1 score of the “liberal” class was extremely decreased and this makes our classifiers not 100% reliable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bdb015959503" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lcc-unison/what-happen-if-you-mix-twitter-the-president-of-m%C3%A9xico-and-natural-language-processing-bdb015959503">What happen if you mix Twitter, the president of México and natural language processing?</a> was originally published in <a href="https://medium.com/lcc-unison">LCC-Unison</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>