<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marc Bolle on Medium]]></title>
        <description><![CDATA[Stories by Marc Bolle on Medium]]></description>
        <link>https://medium.com/@marc.bolle?source=rss-759e12b6c3c8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tu0NNXIAisPtho-3RheM0g.jpeg</url>
            <title>Stories by Marc Bolle on Medium</title>
            <link>https://medium.com/@marc.bolle?source=rss-759e12b6c3c8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:37:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marc.bolle/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Datasaurus Dozen Visualization using Python]]></title>
            <link>https://medium.com/@marc.bolle/datasaurus-dozen-visualization-using-python-d328dad64d20?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d328dad64d20</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Sat, 18 Nov 2023 18:02:17 GMT</pubDate>
            <atom:updated>2023-11-19T09:49:13.890Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xW0QBfivl3NyED2hDLFjOQ.jpeg" /><figcaption>Image by <a href="https://www.pexels.com/fr-fr/@cup-of-couple/">Cup of Couple</a> on <a href="https://www.pexels.com/">Pexels</a></figcaption></figure><p>When conducting data analysis, relying solely on summary statistics such as the mean, variance, and correlation might result in missing import insights. In this post, I’ll demonstrate the importance of data visualization by introducing the Datasaurus Dozen.</p><h3>What is the Datasaurus Dozen?</h3><p>The Datasaurus was initially introduced by Professor Alberto Cairo in a <a href="http://www.thefunctionalart.com/2016/08/download-datasaurus-never-trust-summary.html">blog post</a> in 2016. It’s a usual dataset composed of 142 observations with a bivariate normal distribution. Yet, when visualized on a scatterplot, these observations collectively form the shape of a dinosaur:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*neL2olvh_B3GPwiUhuYvdg.jpeg" /><figcaption>Image by Alberto Cairo on <a href="http://www.thefunctionalart.com/">thefunctionalart.com</a></figcaption></figure><p>Inspired by this dinosaur, two researchers at Autodesk Research, Justin Matejka and George Fitzmaurice, created 12 datasets that have similar summary statistics as the dinosaur, but with different distributions and visual appearance when plotted on scatterplots. The scatterplots reveal very different shapes such as a star and a cross.</p><p>The authors developed a new statistical method for generating such datasets. As they explain in their <a href="https://www.autodesk.com/research/publications/same-stats-different-graphs">paper</a>, “new datasets are iteratively generated from a seed dataset through random perturbations of individual data points, and can be directed towards a desired outcome through a simulated annealing optimization strategy”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*vM-8cEQwp2XlBZnbHdVQng.png" /></figure><p>The Datasaurus Dozen serves as a fresh approach to underscore the importance of data visualization. While summary statistics aid in understanding data, they fall short in revealing the whole story. It’s a notion that is often taught in statistics lectures with Anscombe’s Quartet: four scatterplots that, although being visually distinct, all have the same mean, variance and correlation coefficient.</p><h3>Download the Datasaurus Dozen Dataset</h3><ul><li>Download the Datasaurus Dozen dataset from the Autodesk Research source <a href="https://www.autodesk.com/content/dam/autodesk/www/autodesk-reasearch/Publications/pdf/SameStatsCode.zip">here</a></li><li>Unpack the ‘DatasaurusDozen.tsv’ file from the SameStatsCode.zip archive.</li><li>Once retrieved, you’re all set to delve into its exploration using Python!</li></ul><h3><strong>Exploring the Datasaurus Dozen’s Summary Statistics</strong></h3><p>Summary statistics are statistics that summarize the characteristics of the data and the relationship between its variables.</p><p>They are very useful since they reduce large number of observations down to a single numbers that are easily interpretable. Mean, standard deviation and correlation coefficient are example of summary statistics.</p><p>● Let’s analyze the summary statistics of the the Datasaurus Dozen using Python. To begin, we’ll import the dataset into a DataFrame using Pandas.</p><pre>import pandas as pd<br><br>data = pd.read_csv(&#39;DatasaurusDozen.tsv&#39;, sep=&#39;\t&#39;)<br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*1loWSBV1FzuHX-BGkZmMZg.jpeg" /></figure><p>As observed, the dataset comprises a column labeled ‘dataset’ which contains the names of the datasets. Additionally, it includes two columns, ‘x’ and ‘y,’ representing the coordinates of individual data points.</p><p>● Let’s check how many datasets there are using the len() and unique() functions</p><pre>len(data[&#39;dataset&#39;].unique())<br><br>__<br>Ouput<br>13</pre><p>There are indeed 13 datasets. The dinosaur and 12 another datasets.</p><p>● Let’s now retrieve their name:</p><pre>data[&#39;dataset&#39;].unique()<br><br>__<br>Ouput<br>array([&#39;dino&#39;, &#39;away&#39;, &#39;h_lines&#39;, &#39;v_lines&#39;, &#39;x_shape&#39;, &#39;star&#39;,<br>       &#39;high_lines&#39;, &#39;dots&#39;, &#39;circle&#39;, &#39;bullseye&#39;, &#39;slant_up&#39;,<br>       &#39;slant_down&#39;, &#39;wide_lines&#39;], dtype=object)</pre><p>● Now, let’s calculate the summary statistics (mean, variance and standard deviation) of each dataset using Pandas’ functions groupby() and agg() :</p><pre>datasets_grouped = data.groupby(&#39;dataset&#39;)<br>datasets_grouped.agg([&#39;count&#39;, &#39;mean&#39;, &#39;var&#39;, &#39;std&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*vDHnwAcl6cz3aL-Jq7zRUg.jpeg" /></figure><p>As we can see, all datasets display similar summary statistics, each matching to two decimal places.</p><p>● Let’s now compute the correlation coefficients between X and Y in each dataset using the corr() function:</p><pre>corr = data.groupby(&#39;dataset&#39;)[[&#39;x&#39;,&#39;y&#39;]].corr().iloc[0::2,-1]<br>corr</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*mSir_GdIRqr1Cw_5mJuQ4Q.jpeg" /></figure><p>Once more, the datasets demonstrate analogous correlation coefficients, matching to two decimal places. Given the proximity of this coefficient to zero, we can infer a lack of substantial relationship between the X and Y variables.</p><p>● Similarly, we can calculate the covariance matrix for each dataset using the cov() function:</p><pre>cov_matrix = data.groupby(&#39;dataset&#39;).cov()<br>cov_matrix</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/1*5ezyTHDUKPjglqBzbxgOZg.jpeg" /></figure><p>Once more, the thirteen datasets display the exact same covariance matrix, matching to two decimal places.</p><p>● Let’s now try to compute a linear regression equations for each of these datasets using the <em>scipy.stats</em> module:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*8k5zWgclyAf3zSRcNaPd2A.jpeg" /></figure><p>Once more, they all have the exact same linear regression equation;</p><p>At this point, anyone looking at all of these similar statistical measures should conclude that all thirteen datasets have exactly the same distribution and appearance when displayed on scatter plots. Isn’t it?</p><h3><strong>Visualizing the Datasaurus Dozen in Python</strong></h3><p>●Let’s now visualize the 13 datasets in scatter plots using Python and Seaborn:</p><pre>import seaborn as sns <br><br>grid_scatterplots = sns.FacetGrid(data, col=&quot;dataset&quot;, hue=&quot;dataset&quot;, col_wrap=4)<br>grid_scatterplots.map_dataframe(sns.scatterplot, x=&quot;x&quot;, y=&quot;y&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*RRcSunzh4R8bijUMm9rMMA.png" /></figure><p>Here we are, 13 datasets with similar statistics but with dramatically different appearance. What a fascinating and curious phenomenon!</p><p>● Let’s fit a linear regression line on each dataset using seaborn’s lmfit() function:</p><pre>sns.lmplot(data=data, x=&quot;x&quot;, y=&quot;y&quot;, col=&quot;dataset&quot;, hue=&quot;dataset&quot;, line_kws={&#39;color&#39;: &#39;#003f5c&#39;}, col_wrap=4, ci=None, height=3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*CKRCSF2DZcrBkZx_F80KIQ.png" /></figure><p>We can observe that all the datasets exhibit the same linear regression line as well.</p><p>● Even more interesting, we can see that, as the data points change shapes, they retain the same summary statistical values to two decimal places throughout the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*uzVmvgn7I-t9x5MTByQg_A.gif" /><figcaption>GIF by <a href="https://www.research.autodesk.com/">Autodesk Research</a></figcaption></figure><h3><strong>Conclusion</strong></h3><p>The playful revelation of a dinosaur shape in the Datasaurus Dozen serves as a whimsical reminder of the limitations inherent in relying solely on summary statistics. It underscores the crucial role of data visualization for a more profound understanding of data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d328dad64d20" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Geolocating an IP Address with Python and Geocoder]]></title>
            <link>https://medium.com/@marc.bolle/geolocating-an-ip-address-with-python-and-geocoder-02e0c0f178ad?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/02e0c0f178ad</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Sun, 08 Oct 2023 13:09:57 GMT</pubDate>
            <atom:updated>2023-10-08T13:13:51.902Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*P-wWmc_5eV9Q5ULmQvMEqw.jpeg" /></figure><p>Obtaining the geographical location of an IP address can be valuable for various applications, from web analytics to cybersecurity. In this tutorial, we will explore how to determine the location of an IP address using Python, and Geocoder and Folium libraries.</p><h3>IP Geolocation with Geocoder</h3><p>Geocoder is a Python library that simplifies working with geocoding provider such as Google, Bing and OSM using Python. This Python library provides a straightforward method for IP geolocation.</p><h4>Geocoder Installation</h4><p>To get started, you’ll need to install Geocoder using <a href="https://pypi.org/project/geocoder/">PIP</a>:</p><pre>pip install geocoder</pre><p>You can also install Geocoder using <a href="https://anaconda.org/conda-forge/geocoder">Conda</a>:</p><pre>conda install -c conda-forge geocoder</pre><h4>IP Geolocation</h4><p>Once Geocoder is installed, you must import the library into Python and assign an IP address to a geocoder.ip variable. <br>You can then obtain the city and the coordinates (latitude and longitude) of the IP address:</p><pre>#Import Geocoder <br>import Geocoder<br><br>#Assign IP address to a variable<br>ip = geocoder.ip(&quot;161.185.160.93&quot;)<br><br>#Obtain the city<br>print(ip.city)<br><br>#Obtain the coordinates: <br>print(ip.latlng)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/1*WmMmEtia7JMf2Z2rvleiFA.png" /></figure><p>Alternatively, you can also retrieve your own IP address by specifying “me” as the input of geocoder.ip :</p><pre>#Assign your own IP address to a variable<br>ip = geocoder.ip(&quot;me&quot;)</pre><h3>Geolocate an IP Address on a Map with Folium</h3><p>Folium is a Python library that allows you to create interactive leaflet maps in Python. It makes it very easy to track an IP address on a map.</p><h4>Folium Installation</h4><p>To get started, you’ll need to install Folium using <a href="https://pypi.org/project/folium/">PIP</a>:</p><pre>pip install folium</pre><p>You can also install Folium using <a href="https://anaconda.org/conda-forge/folium">Conda</a>:</p><pre>conda install folium -c conda-forge</pre><h4>IP Geolocation on a Map</h4><p>Now, you can create a map with Folium based on the coordinates obtained earlier. <br>The following code creates a map centered around the IP address’s coordinates and adds a red circle and a marker for visualization:</p><pre>#Import Folium<br>import folium<br><br>#Extract the coordinates from the &#39;ip&#39; variable, which was previously obtained<br>location = ip.latlng<br><br>#Create a new Folium map centered around the extracted location coordinates<br>map = folium.Map(location=location, zoom_start=10)<br><br>#Add a red circle marker to the map at the specified &#39;location&#39; <br>folium.CircleMarker(location=location, radius=50, color=&quot;red&quot;).add_to(map)<br><br>#Add a standard marker (pin) to the map at the same &#39;location&#39; coordinates<br>folium.Marker(location).add_to(map)<br><br>#Render the map<br>map</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdvMg2pcqZdwuSUgHbARSw.png" /></figure><h4>Export the Map as HTML File</h4><p>Exporting the map as an HTML file is also possible as follow:</p><pre>map.save(&quot;map.html&quot;)</pre><p>You can then open it in a web browser for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*SQvt5N3YaulUN4B7kjhQeg.jpeg" /></figure><h3>Conclusion</h3><p>This tutorial has shown you how to use the Python as well as Geocoder and Folium libraries to geolocate IP addresses. With this knowledge, you may improve your Python projects and acquire insights from geographic data!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02e0c0f178ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ChatGPT vs. Stack Overflow: Which Is Better at Answering Software Engineering Questions?]]></title>
            <link>https://medium.com/@marc.bolle/chatgpt-vs-stack-overflow-which-is-better-at-answering-software-engineering-questions-5bba21569361?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bba21569361</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Tue, 22 Aug 2023 21:42:49 GMT</pubDate>
            <atom:updated>2023-08-22T21:42:49.462Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ez2K0l_iLMEZbnd7jJBrwg.jpeg" /></figure><p>Is ChatGPT better than Stack Overflow for accelerating software development? Researchers from Purdue University in Indiana, USA, explored this question in their study titled <a href="https://arxiv.org/pdf/2308.02312.pdf">“Who Answers It Better? An In-Depth Analysis of ChatGPT and Stack Overflow Answers to Software Engineering Questions.”</a></p><h3>Methodology</h3><p>To conduct their study, the Purdue researchers analyzed ChatGPT’s responses to 2,000 questions from the programming-focused website Stack Overflow.</p><p>The academics then assessed the code suggestions produced by ChatGPT for accuracy, consistency, comprehensiveness, and conciseness.</p><p>Twelve volunteers were also granted access to the same responses, enabling them to compare them with those posted by the Stack Overflow community for the corresponding questions</p><p>Finally, the researchers performed a linguistic analysis on ChatGPT’s answers to understand the language rules and processes that the tool use.</p><h3>Half of ChatGPT’s Answers are Erroneous</h3><p>The research demonstrates that ChatGPT provides erroneous answers to software programming questions half the time (52%). These ‘hallucinations’ — the term used for incorrect responses from LLMs — are convincingly misleading, fooling a third of the study participants.</p><p>According to the researchers, users only manage to identify errors from ChatGPT when they are obvious. “When the error isn’t easily verifiable or requires a programming environment or external documentation, users often fail to identify it or underestimate the degree of error in the response,” note the authors.</p><h3>Users Prefer ChatGPT’s Verbose Answers</h3><p>The linguistic analysis also demonstrates that 77% of ChatGPT’s answers are verbose. In other words, these responses use more words than are necessary to effectively communicate the information.</p><p>Participants, however, prefer ChatGPT’s responses to Stack Overflow’s users responses due to their comprehensiveness and articulate language style.</p><p>“In nearly 40% of cases, participants preferred ChatGPT’s answers over those from Stack Overflow, even though in 77% of cases, these suggestions were simply wrong!” noted the authors.</p><p>The researchers find that ChatGPT tends to reassure users with its polite language and well-articulated, comprehensive responses. In essence, ChatGPT’s displays confidence and apparent relevance of its responses (even when incorrect), which contribute to gaining users’ trust.</p><p>This is confirmed by the linguistic study and sentiment analysis. ChatGPT’s responses are more formal and analytical, conveying fewer negative sentiments than the comments posted on Stack Overflow.</p><p>“On many occasions, we observed that ChatGPT inserted words and phrases like ‘of course, I can help you,’ ‘this will certainly solve the problem,’ etc.,” the authors write. While users of Stack Overflow are more direct and concise in their answers.</p><h3>ChatGPT Accelerates Software Development</h3><p>The researchers acknowledged the potential of generative AI to revolutionize the way that programmers access information.</p><p>Platforms like Stack Overflow offer valuable insights from a community of experts but might involve prolonged waiting times for solutions.</p><p>In contrast, ChatGPT offers quick responses to complex coding queries, engaging in conversation-like interactions to delve deep into questions. Even if ChatGPT provides incorrect codes, the user can guide the model towards the correct response by providing it with more information.</p><h3>Maintaining Awareness about the Potential Risks</h3><p>The researchers expressed apprehensions regarding the possibility of chatbots spreading inaccurate information and code.</p><p>Indeed, the code generated by ChatGPT has the potential to damage information systems, pose security risks, or even alter the data in a database. As a result, programmers who use ChatGPT need to exercise caution.</p><p>This concern led Stack Overflow to impose a complete ban on answers originating from ChatGPT early this year.</p><p>Researchers recommend that ChatGPT go beyond its concise disclaimer concerning potential inaccuracies and provide a clear indication of the extent of incorrectness and ambiguity in each response.</p><p>“Our study highlights the need for careful scrutiny and correction of ChatGPT’s errors, as well as awareness among users of the risks associated with seemingly correct responses provided by the chatbot,” write the Purdue researchers.</p><p>Purdue’s academics also encourage other research teams to replicate and extend their study, offering their datasets for this purpose.</p><h3>Conclusion</h3><p>In conclusion, the research by Purdue University researchers emphasizes the dual nature of ChatGPT’s software development capabilities. While the AI tool accelerates the process with quick responses, the research underscores the critical need for caution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bba21569361" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WormGPT, FraudGPT: How Cybercriminals are Taking Over Generative AI]]></title>
            <link>https://medium.com/@marc.bolle/wormgpt-fraudgpt-how-cybercriminals-are-taking-over-generative-ai-eea8c6adf7fa?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/eea8c6adf7fa</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Sun, 20 Aug 2023 09:38:59 GMT</pubDate>
            <atom:updated>2023-08-20T16:02:39.298Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*12O86p4WcSAdI1CDHonBvg.jpeg" /><figcaption>Image by <a href="https://www.pexels.com/@tima-miroshnichenko/">Tima Miroshnichenko</a> on <a href="https://www.pexels.com/">Pexels</a></figcaption></figure><p>The increasing use of generative AI has opened up new opportunities while also introducing new cybersecurity dangers. As recent activity on the Dark Web Forum show, cybercriminals have quickly adopted generative AI to improve their attacks, including tools like WormGPT and FraudGPT.</p><h3>WormGPT: the Dark Side of ChatGPT</h3><p>On July 13th, researchers from the cybersecurity company <a href="https://slashnext.com/blog/wormgpt-the-generative-ai-tool-cybercriminals-are-using-to-launch-business-email-compromise-attacks/">SlashNext uncovered new tool</a>, known as WormGPT, which is presented as “similar to ChatGPT but without ethical boundaries.”</p><p>WormGPT has been developed on GPT-J open-source language model (Generative Pre-Trained Transformer with JAX built by EleutherAI in 2021), using various data sources, including information about cyberattacks and malicious software.</p><p>While ChatGPT has usage limitations, WormGPT can assist users in engaging in various malicious acts and illegal activities. WormGPT is essentially ChatGPT without moral restrictions.</p><h4>Boundless Potential for Harm</h4><p>For example, WormGPT can be used to automate the creation of highly convincing deceptive emails. Because these emails are highly personalized for the recipients, they can be used to deceive someone into sending money or sensitive information.</p><p>WormGPT can also be used to craft malicious code that steals sensitive information from a user’s device, or ransomware that encrypts files and demands payment to unlock them.</p><p>The researchers at SlashNext were able to employ WormGPT to “compose an email aimed at convincing an unsuspecting account manager to pay a fraudulent invoice.” They were astonished by how effectively the language model accomplished this task, labeling the outcome as “remarkably persuasive and strategically clever as well.”</p><h4>Monetizing Malice</h4><p>The developer of WormGPT is crafting a subscription model for access, ranging from $60 to $700.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nnEhZtQBSqrLSUfLV0j7Og.jpeg" /><figcaption>Screenshot of WormGPT showing a user request asking the model to produce a Python malware.</figcaption></figure><h3>FraudGPT: The Malicious Alternative to ChatGPT</h3><p>Netenrich, a security and operations analytics business, also <a href="https://netenrich.com/blog/fraudgpt-the-villain-avatar-of-chatgpt">found the presence of another malicious malware called FraudGPT</a>.</p><p>Since July 2023, this AI model has been circulating on Telegram Channels, allowing any cybercriminal to undertake complex attacks that were previously reserved for coding and hacking experts.</p><p>On June 23, 2023, the person behind this tool, identified as “CanadianKingpin,” launched a Telegram channel to promote their model. He argues, “Look no further if you are looking for a ChatGPT alternative designed to offer an expansive array of tools, features, and exclusive capabilities tailored for boundary-pushing individuals!”</p><h4>Easy Access through Subscription</h4><p>FraudGPT can be found on dark web marketplaces such as Alphabay, Empire, Torrez, Versus, WHM, and World. It is sold on a subscription basis, with costs starting at $200 per month, $1000 for six months, and $1700 annually.</p><h4>Empowering Cybercriminals with Unprecedented AI Capabilities</h4><p>Among the features of FraudGPT is the ability to generate malicious code to exploit weaknesses in computer systems, applications, and websites. It creates malware that evades conventional security measures.</p><p>Furthermore, FraudGPT may generate convincing phishing pages that confuse users by replicating authentic websites, increasing the success of phishing attempts.</p><p>FraudGPT also allows for the discovery of Non-Verified By Visa bins (Non-VBV), which simplifies unauthorized transactions by eliminating the need for additional security checks.</p><p>FraudGPT can also be used to identify weaknesses and data leaks within target infrastructures, facilitating security breaches.</p><p>It also helps cybercriminals find hidden hacker groups, covert websites, and black marketplaces where stolen data is traded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*TvggByJW5NgrYcKqtMtE_g.png" /><figcaption>Screenshot of FraudGPT showing a user request asking the model to produce a SMS spam.</figcaption></figure><h3>Conclusion</h3><p>The emergence of these tools raises concerns about their potential misuse and their implications for online security. It highlights the need of remaining vigilant in the face of evolving cyber threats and the importance of staying informed about such developments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eea8c6adf7fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jupyter AI — A generative AI extension for JupyterLab]]></title>
            <link>https://medium.com/@marc.bolle/jupyter-ai-a-generative-ai-extension-for-jupyterlab-9f072b00e266?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f072b00e266</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Thu, 13 Jul 2023 20:21:51 GMT</pubDate>
            <atom:updated>2023-07-13T20:21:51.366Z</atom:updated>
            <content:encoded><![CDATA[<h3>Jupyter AI — A generative AI extension for JupyterLab</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*e3JY7ni_7Eeo_cqcC0-1fg.jpeg" /></figure><p>Discover Jupyter AI, a new Jupyter Lab extension that adds a generative AI-powered conversational assistant!</p><p>Jupyter AI was developed by David L. Qiu. It enhances your JupyterLab experience by providing a user-friendly interface for exploring generative AI models in your notebooks.</p><p>Jupyter AI supports a variety of language models, including OpenAI, Hugging Face, SageMaker, and Cohere.</p><p>This article provides a practical introduction to the chat interface of Jupyter AI</p><h4>Table of Contents</h4><p>◦ Installing Jupyter AI <br>◦ Setting up the Chat Interface<br>◦ Chatting<br>◦ Generating code <br>◦ Asking about something in your notebook<br>◦ Generating a new notebook<br>◦ Learning about local data</p><h3>Installing Jupyter AI</h3><p>In order to unleash the power of Jupyter AI, the first step is to install this extension into your JupyterLab environment. In this section, I will guide you through the step-by-step installation process.</p><h4>Prerequisites</h4><p>Before we proceed with the installation of Jupyter AI, let’s ensure that you have all the prerequisites in place:</p><ol><li>Python version from 3.8 to 3.10 (Python 3.11 is not supported yet)</li><li>JupyterLab 3 installed (JupyterLab 4 is not supported yet)</li><li>Jupyter Server 2.x</li></ol><h4>Jupyter AI installation</h4><p>We are now ready to install Jupyter AI. You can install it seamlessly with PIP:</p><pre>pip install jupyter_ai</pre><h3>Setting up the Chat Interface</h3><p>The chat interface is the simplest method to use Jupyter AI. You can engage with a generative AI model from within Jupyter Lab.</p><p>After you’ve launched JupyterLab, open the chat interface by clicking the new “chat” icon in the left side panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*_DN2uBa0w1YCaE91BMhEOg.jpeg" /></figure><p>The first time you open the chat interface, Jupyter AI will ask you which models you wish to use as a language model and an embedding model. Jupyter AI supports the following models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*sttbJta2Iw_tfeZ2yCPlKA.jpeg" /></figure><p>After you’ve made your choices, the UI shows fields for authenticating yourself with API keys. In the following example, I pick OpenAI’s <em>gpt-3.5-turbo</em> as language model, and OpenAI’s <em>text-embedding-ada-002</em> as embedding model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*DxVLD82XujjlfjqFSjxbAA.jpeg" /></figure><p>Once you’ve configured all of the required keys, click the “back” (left arrow) button in the upper-left corner of the Jupyter AI side panel. The chat interface appears and you can ask anything.</p><p>⚠️ The chat interface sends data to the third-party model hosted you have selected. Read the terms of use of this third-party hoster to find out what types of data are transmitted and stored.</p><h3>Chatting</h3><p>You can chat with Jupyternaut, the Jupyter AI chatbot. Once you have sent a message, Jupyternaut should respond.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*xIHrTq85s23wv5TwlP-Ryg.jpeg" /></figure><p>The last two exchanges in your conversation are remembered by the chat backend and passed to the language model. Thus, you can ask follow-up questions.</p><h3>Generating code</h3><p>You can ask the Jupyternaut to generate code. In the following example, I ask the model to generate a code that reverses the words of a string:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*26JlWjmQIQN6oDxcIqFktg.jpeg" /></figure><h3>Asking about something in your notebook</h3><p>The chat interface of Jupyter AI can incorporate a section of your notebook in your prompt and send it to the language model.</p><p>To do that, you need to hightlight a portion of your notebook and click the “Include selection” box in the chat window. Then, you can enter your message and send it to the model.</p><p>In the following example, I include a cell and ask the model to explain what does it do:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*VVOrax62B3xOPKVjGTQX0g.jpeg" /></figure><p>⚠️ Adding code from your notebook to your prompt will rise the number of tokens, causing your request to cost more money. Review your model provider’s cost policy to avoid undesired bill.</p><h3>Generating a new notebook</h3><p>Here is one of the most interesting features of Jupyter AI: the generation of new notebooks from scratch. You can use the command /generate to ask Jupyternaut to write a new notebook.</p><p>It may take several minutes to generate your notebook. In the meantime, you can continue to use JupterLab and Jupyter AI as usual.</p><p>When Jupyter AI has finished creating your notebook, it will send you another message containing the location of the freshly created notebook.</p><p>In the following example, I ask the model the create a new notebook that demonstrates how to use Matplotlib:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*dv0wYgVbA-AZPwLMXL7AvQ.jpeg" /></figure><p>And here is the notebook, which contains a very-well introduction to Matplotlib with an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PDR8einSNMN8_3Td9zJr3Q.jpeg" /></figure><p>⚠️ Generated notebooks may contain errors and may have unintended side effects when you run the code in them. Before running any generated code, review it thoroughly.</p><p>⚠️ This type of request might considerably rise the number of tokens too, review your model provider’s cost policy to avoid undesired bill.</p><h3>Learning about local data</h3><p>You can teach Jupyter AI about local data so that it can be used to answer your questions.<br>This local data is embedded using the embedding model you selected in the settings panel.</p><p>You have two options: <br>- The /learn command to teach Jupyter AI about a file, <br>- The /lean docs/ command to teach Jupyter AI about a folder full of documentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*FET74GKkcadqk6sxDQRelA.jpeg" /></figure><p>You can then use /ask command to ask a question specifically about the data that you taught Jupyter AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*Kfrcyf0UcNfs3fXAd773yA.png" /></figure><p>You can make Jupyter AI forget all it learn with the /learn -d command. This will erase the local vector database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*bkrej0_pIdVMQ23VKsQyrw.png" /></figure><p>⚠️This functionality may send data to the third-party model provider that host the embedding model you selected. Review carefully their policies before using this functionality.</p><p>That’s it. We have seen how to use the chat interface of Jupyter AI to enhance our experience in Jupyter Lab. For more info about Jupyter AI, visit the <a href="https://jupyter-ai.readthedocs.io/en/latest/index.html">official documentation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f072b00e266" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenAI APIs with Python — Complete Guide]]></title>
            <link>https://medium.com/@marc.bolle/openai-apis-with-python-complete-guide-d933fb770f95?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d933fb770f95</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Tue, 04 Jul 2023 20:48:35 GMT</pubDate>
            <atom:updated>2023-11-30T09:12:49.828Z</atom:updated>
            <content:encoded><![CDATA[<h3>OpenAI APIs with Python — Complete Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fwSo2TJdFSHePC7UgKQR_w.png" /></figure><p>Are you having trouble understanding how to utilize the OpenAI API in Python?</p><p>Learn how to use the OpenAI API in Python to execute tasks like text classification, chat completion, code generation, moderation, language translation, text to speech, and image generation with ease. Watch the GPT-3, ChatGPT, Whisper and GPT-4 models in action.</p><p>Whether you’re a newbie or an experienced developer, this tutorial will help you integrate the OpenAI API into your Python projects.</p><h3>What is the OpenAI API ?</h3><p>OpenAI has released an API for accessing its AI models. It gives users access to all the models, such as GPT-3, GPT-4, DALL-E and Whisper.</p><p>With the API, developers can easily add AI capabilities to their applications using a variety of programming languages such as Python, Curl and Node.JS.</p><p>The current endpoints of the API are: <br>- /completions<br>- /chat/completions<br>- /edits <br>- /images/generations<br>- /images/edits<br>- /images/variations<br>- /embeddings<br>- /audio/transcriptions<br>- /audio/translations<br>- /moderations<br>- /files (not included in this article)<br>- /fine-tunes (not included in this article)</p><h3>Table of Contents</h3><p>◦ <strong>Setting up OpenAI API</strong></p><p><strong>◦ Completion</strong><br>Text completion<br>Text generation <br>Language translation <br>Sentiment analysis <br>Text classification <br>Code generation <br>Summarization <br>Text insertion <br>Text to emojis</p><p><strong>◦ Chat Completions</strong><br>What’s the difference between Completions and Chat Completions APIs? <br>Role and Content definition<br>Chat completion for non-chat request<br>Chat completion with instructions<br>Chat completion with few shot learning <br>Conversation loop like ChatGPT</p><p><strong>◦ Edits</strong><br>Text edition<br>Code edition</p><p><strong>◦ Embeddings </strong><br>Obtaining the embeddings <br>Advanced use cases</p><p><strong>◦ Moderations </strong><br>Content moderation example</p><p><strong>◦Images </strong><br>Images Generations <br>Images Edits <br>Images Variations</p><p><strong>◦ Speech to Text</strong><br>Transcription<br>Translation</p><h3>Setting up OpenAI API</h3><p>Let’s get started! In this part, I will walk you through the process of setting the OpenAI API.</p><h4>Step 1: Get an API key</h4><p>To obtain an API key for the OpenAI API, you must first create an OpenAI account on the <a href="https://openai.com/">OpenAI website</a>.</p><p>Once you have an account, you can generate an API key by performing the following steps:</p><ol><li>Log into your OpenAI account on <a href="https://platform.openai.com/">platform.openai.com</a> website</li><li>Click on the “<em>View API Keys</em>” button in the top-right corner of the page.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*koEGCoFV8fPT9As0kKhGMA.jpeg" /></figure><p>3. Click the “<em>+ Create new secret key</em>” button to generate a new API key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*2Nd321E2hdLwcNqoU8Lr0A.jpeg" /></figure><p>4. Once the API key is generated, you can copy it and use it in your code to authenticate with the OpenAI API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*1XE1-UPpEVYJGC9FX2zcQw.jpeg" /></figure><h4>Step 2: Install OpenAI Library</h4><p>To connect to ChatGPT via the OpenAI API in Python, you will need to install the <a href="https://github.com/openai/openai-python">openai library</a> via <a href="https://pypi.org/project/openai/">PyPI</a> as follow:</p><pre>pip install --upgrade openai</pre><h4>Step 3: <strong>Authenticate</strong> by using <strong>API</strong> keys</h4><p>You can now authenticate by using your API key:</p><pre>import openai  <br><br>openai.api_key = &quot;YOUR_API_KEY&quot;  # Set your OpenAI API key</pre><p>✅ You are now fully ready to send requests to the APIs of OpenAI!</p><h3>Completion</h3><p><em>OpenAI.Completion</em> is an endpoint of the OpenAI API that allows you to interact with GPT language models (<em>text-davinci</em>, <em>davinci</em>, <em>curie</em>, <em>babbage</em>, <em>ada</em>) to generate human-like text completions based on a given prompt.</p><p>These models have been trained on vast amounts of text data and are capable of understanding context and generating coherent responses across various domains.</p><p>The <em>OpenAI.Completion</em> API endpoint is located at: https://api.openai.com/v1/completions.</p><p>The create() method from the <strong><em>openai.Completion</em></strong> module allows you to perform completion. The parameter values are as follows:<br>- <strong>model</strong>: the type of the GPT model, which can be one of the Davinci, Curie, Babbage, or Ada models. <br>- <strong>prompt</strong>: the prompt’s text guides the model to produce an output.<br>- <strong>max_tokens</strong>: the maximum number of tokens to consume (four tokens typically constitute one English word).<br>- <strong>temperature</strong>: specifies the creativity level of your model. Setting the temperature to a lower value will return more precise and straightforward answers.<br>- <strong>n</strong>: the number of completions to generate for each prompt.</p><h4>Text completion</h4><p>You can ask the API to add a credible continuation to a text you provide in the prompt. It generates a completion that strives to align with the given context or pattern.</p><p>The following code shows a text completion of “The quick brown fox” using the latest Davinci model:</p><pre>prompt = &quot;The quick brown fox&quot;<br><br>response = openai.Completion.create(<br>  engine=&quot;text-davinci-003&quot;,<br>  prompt=prompt,<br>  max_tokens=50<br>)<br><br>generated_text = response.choices[0].text.strip()<br>print(generated_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*MkzkYzulSCUuLLcHtmXE4Q.jpeg" /></figure><h4><strong>Text generation</strong></h4><p>You can ask the API to generate written content. Add the text in the prompt and guide the model about the precise type of text you want.</p><p>The following script generates 3 “beautiful rhyming poem on any subject” with a limit of 30 tokens each using the lastest Davinci model:</p><pre>prompt=&quot;write you a beautiful rhyming poem on any subject&quot;<br><br>response = openai.Completion.create(<br>  model=&quot;text-davinci-003&quot;,<br>  prompt=prompt,<br>  max_tokens=30,<br>  temperature=1,<br>  n=3<br>)</pre><p>You can print the different versions the text that have been produced as follow:</p><pre>for choice in response[&#39;choices&#39;]:<br>    print(choice[&#39;text&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*OscQyJu2Ye5qAvUkGIOdaA.jpeg" /></figure><h4>Language translation</h4><p>You can ask the API to translate text. Add the text to be translated in the prompt and guide the model about the type of translation you expect.</p><p>The following script translates “Hello, how are you” in French with the latest Davinci model:</p><pre>text=&quot;Hello, how are you?&quot;<br>prompt = f&quot;Translate from English to French: {text}&quot;<br><br>response = openai.Completion.create(<br>  engine=&quot;text-davinci-003&quot;,<br>  prompt=prompt,<br>)<br><br>translation = response.choices[0].text.strip()<br>print(translation)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*j3ollM3mj-28iaU-LuYxyw.jpeg" /></figure><h4>Sentiment analysis</h4><p>GPT-3 models are “smart” enough to carry out sentiment analysis on textual content.</p><p>The following script returns a sentiment analysis of “I love ice cream!” using the latest Davinci model:</p><pre>text=&quot;I love ice cream!&quot;<br>prompt=f&quot;Sentiment analysis: {text}&quot;<br><br>response = openai.Completion.create(<br>  engine=&quot;text-davinci-003&quot;,<br>  prompt=prompt<br>)<br><br>sentiment = response.choices[0].text.strip()<br>print(sentiment)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/89/1*rhjl3OOPTHOMYM4lhjgc9g.jpeg" /></figure><h4>Text classification</h4><p>You can easily perform text classification on a textual input with GPT models. Text classification is the practice of categorizing text based on its content or features.</p><p>In the following example, we use the latest Davinci model to classify the provided text as offensive or non-offensive content:</p><pre>text=&quot;f*** y**!&quot;<br>prompt=f&quot;Is this inappropriate or offensive content: {text}&quot;<br><br>response = openai.Completion.create(<br>  engine=&quot;text-davinci-003&quot;,<br>  prompt=prompt<br>)<br><br>classification = response.choices[0].text.strip()<br>print(classification)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*LDZz70ono7tKVGLjjlhzwg.jpeg" /></figure><h4>Code generation</h4><p>The Completion module is capable of code generation. You can easily generate various sorts of code.</p><p>In the following example, we use the latest Davinci model to produce a Python function that sorts letters in a word:</p><pre>prompt=&quot;Create a Python function that sorts letters in a word&quot;<br><br>response = openai.Completion.create(<br>  engine=&quot;text-davinci-003&quot;,<br>  prompt=prompt,<br>  max_tokens=100<br>)<br><br>code = response.choices[0].text.strip()<br>print(code)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*85z3nG9jKpr8rRPMIH90DQ.jpeg" /></figure><p>And we can see that the code works perfectly:</p><pre>def sortLetters(word):<br>  sortedLetters = sorted(word) # uses built-in Python sorting algorithm<br>  return &quot;&quot;.join(sortedLetters) # joining all sorted letters <br><br>word = input(&quot;Enter a word: &quot;)<br>print(sortLetters(word))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*rJ50qMuzeSS3eSELgfw5zQ.jpeg" /></figure><h4>Summarization</h4><p>GPT models have the capacity to generate concise summaries of text as well.</p><p>The following script generates summarizes a description of Dubai using the latest curie model:</p><pre>text = &quot;Dubai is the most populous city in the United Arab Emirates (UAE) and the capital of the Emirate of Dubai, the most populated of the 7 emirates of the United Arab Emirates.&quot;<br><br>response = openai.Completion.create(<br>  engine=&quot;text-curie-001&quot;,<br>  prompt=f&quot;Summarize this text:{text}&quot;,<br>  max_tokens=30<br>)<br><br>summary = response.choices[0].text.strip()<br>print(summary)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*xL4fbhPBueIpzCMZno5QLA.jpeg" /></figure><h4><strong>Text insertion</strong></h4><p>GPT-3 models let you to insert text within a body of text.<br>Pass the text that comes before the text you want to insert as the prompt to the model. Then, using the create() method’s suffix attribute, add the text that will trail. The model will incorporate text in between.</p><pre># insert the beginning of the text to be completed<br>prompt_text = &quot;&quot;&quot; List of the latest 5 Presidents of the United States:<br>1. Joe Biden<br>&quot;&quot;&quot;<br><br>response = openai.Completion.create(<br>  model=&quot;text-davinci-003&quot;,<br>  prompt= prompt_text,<br>  max_tokens=100,<br>  n = 1,<br>  suffix = &quot;5. Bill Clinton&quot; #insert the text that comes after the insertion point<br>)<br><br>insertion = response.choices[0].text.strip()<br>print(insertion)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/1*LSd8z2O9Ob1ZXF_wMKU8gg.jpeg" /></figure><h4>Text to emojis</h4><p>Another text conversion capability of GPT-3 models is the text-to-emojis. For example, the following script converts text to emojis with the latest <em>davinci</em> model:</p><pre>prompt_text = &quot;&quot;&quot; Convert the following list of emotions to emojis:<br>1. Happy<br>2. Sad<br>3. Cry<br>&quot;&quot;&quot;<br><br>response = openai.Completion.create(<br>  model=&quot;text-davinci-003&quot;,<br>  prompt= prompt_text,<br>  max_tokens=100,<br>  temperature = 0,<br>  n = 1<br>)<br><br>emojis = response.choices[0].text.strip()<br>print(emojis)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/70/1*-uFKm2I8ffxdfedr-z5Sxg.jpeg" /></figure><h3>Chat Completions</h3><p><em>OpenAI.Chat.Completion</em> is a new OpenAI API endpoint for interacting with the latest and most capable language models (<em>gpt-4</em> and <em>gpt-3.5-turbo</em>) to generate human-like text completions based on a dialogue.</p><p>These models have been trained on vast amounts of text data and are capable of understanding context and generating coherent responses across various domains.</p><p>You can perform all the tasks that can be performed by the <em>Completions</em> endpoint, including text completion, text generation, language translation, sentiment analysis, text classification, code generation, summarization, text insertion and text to emojis.</p><p>The <em>OpenAI.Chat.Completion</em> API endpoint is located at: https://api.openai.com/v1/chat/completions.</p><h4><strong>What’s the difference between Completions and Chat Completions APIs?</strong></h4><p>The Completions endpoint (<em>v1/Completions</em>) completes a single prompt and takes a single text as input, while the Chat Completions<em> </em>endpoint (<em>v1/Chat/Completions</em>) responds to a specified dialogue and requires input in a specific format with the message history.</p><h4>Role and Content definition</h4><p>Unlike the ChatGPT website, which saves and remembers your messages to make a dialog, the API does not keep prior messages. As a result, you will need to adopt a new prompt style to have an interactive and dynamic conversation, as detailed below:</p><ul><li>Instead of sending a single string as your prompt, you have to send a <strong>list of messages</strong> as your input.</li><li>Each message in the list has two properties: <strong>role</strong> and <strong>content</strong></li><li>The <strong>role</strong> parameter is used to specify the role or identity of the message sender, it can takes three values: <br>- ‘<strong>user</strong>’ : it means you, or who is chatting, or who is asking to the model.<br>- ‘<strong>assistant</strong>’ : it means the model that is replying to the user’s questions. <br>- ‘<strong>system</strong>’: it means the system, and it’s used to set the context, give instructions to the assistant, or guide the conversation.</li><li>The <strong>content</strong> parameter is where the message of the corresponding role is provided. The content can include questions, statements, commands, or any other text-based input relevant to the conversation.</li><li>The messages are processed in the order they appear in the list, and the assistant responds accordingly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*0F6ZK33G5kUtTkZ_dGvFWA.jpeg" /></figure><h4>Chat completion for non-chat request</h4><p>The Chat Completion API is intended for chat, but it also works well in non-chat circumstances. We may pass a simple user request, and the model will respond as a helpful assistant.</p><p>In the following example, we ask <em>pt-3.5-turbo</em> model to tell a joke:</p><pre>completion = openai.ChatCompletion.create(<br>  model=&quot;gpt-3.5-turbo&quot;,<br>  messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;tell me a joke&quot;}]<br>)<br><br>chat_response = completion.choices[0].message.content<br>print(chat_response)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*b6NjRU0tXs1qGnhnch9Ikw.jpeg" /></figure><h4>Chat completion with instructions</h4><p>We can pass a request to the model and ask it to take on a specific role in its completion. To do that, we must define the system role.</p><p>The system role, also known as the system message, is included at the beginning of the array. You can provide various information in the system role including: <br>- A brief description of the assistant, <br>- Personality traits of the assistant, <br>- Instructions or rules you would like the assistant to follow, <br>- Data or information needed for the model.</p><p>The system role is optional but it’s recommended to at least include a basic one to get the best results.</p><p>In the following example, we ask <em>gpt-3.5-turbo</em> to explain the Normandy landings in one sentence like Winston Churchill would:</p><pre>completion = openai.ChatCompletion.create(<br>  model=&quot;gpt-3.5-turbo&quot;,<br>  messages=[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;you are Winston Churchill&quot;},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;explain the Normandy landings in one sentence&quot;}]<br>)<br><br>chat_response = completion.choices[0].message.content<br>print(chat_response)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*j4j8zFDGm9n0vxeDoKa5KQ.jpeg" /></figure><h4>Chat completion with few shot learning</h4><p>You can ask the model to complete a series of messages between the user and the assistant. This set of messages in the prompt will act as a few shot examples that can be used to seed answers to typical questions or teach the model specific behaviors.</p><p>In the following example, we ask <em>gpt-3.5-turbo </em>to complete a dialogue where the user ask for a trick question to the assistant:</p><pre>completion = openai.ChatCompletion.create(<br>  model=&quot;gpt-3.5-turbo&quot;,<br>  messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Tell me a joke&quot;},<br>            {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;What goes up but never ever comes down?&quot;},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;I don&#39;t know, tell me&quot;}]<br>)<br><br>chat_response = completion.choices[0].message.content<br>print(chat_response)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/84/1*1uR4jqp8YQFXAVSf2MPIfA.jpeg" /></figure><p>You can also include relevant data or information in the system message to give the model extra context for the conversation.</p><pre>completion = openai.ChatCompletion.create(<br>  model=&quot;gpt-3.5-turbo&quot;,<br>  messages=[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;Annie is 35 years old&quot;},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;How old is Annie?&quot;}]<br>)<br><br>chat_response = completion.choices[0].message.content<br>print(chat_response)<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*PHFDnQlz81ZTFHIDpkebwQ.jpeg" /></figure><h4>Conversation loop like ChatGPT</h4><p>The models have no memory. That’s why in the basic examples so far shown, the assistant can’t recall the previous messages if you make a new API request.</p><p>We can, however, establish a conversation loop to have a kind of “ChatGPT capability”. We do that by storing all previous queries and responses in arrays and sending them with each new query. As a result, the model retains the context of the prior queries and responses.</p><p>When you run the following code, you will see a blank console window. Enter your first question in the box and press enter. After receiving an answer, you can repeat the process and continue to ask questions.</p><pre>conversation=[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful assistant.&quot;}]<br><br>while(True):<br>    user_input = input()      <br>    conversation.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_input})<br><br>    response = openai.ChatCompletion.create(<br>        model=&quot;gpt-3.5-turbo&quot;, <br>        messages = conversation<br>    )<br><br>    conversation.append({&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: response[&#39;choices&#39;][0][&#39;message&#39;][&#39;content&#39;]})<br>    print(&quot;\n&quot; + response[&#39;choices&#39;][0][&#39;message&#39;][&#39;content&#39;] + &quot;\n&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*UyGfKWfXadtfzkZHdAay2Q.jpeg" /></figure><h3>Edits</h3><p><em>OpenAI.Edits</em> is another API endpoint of the OpenAI API that is useful for editing text and code, translating text, and tweaking text.</p><p>The <em>Edits</em> endpoint is located at https://api.openai.com/v1/edits</p><p>The openai.Edit.create() method has the following parameters: <br>- <strong>model</strong>: model to use. <br>- <strong>input</strong>: the input text to use as a starting point for the edit <br>- <strong>instruction</strong>: the instruction that tells the model how to edit the prompt <br>- <strong>n</strong>: how many edits to generate for input and instruction <br>- <strong>temperature</strong>: sampling temperature (higher values will make the output more random, while lower values will make it more focused and deterministic.</p><h4>Text edition</h4><p>You can edit a text using the openai.Edit module with the <em>text-davinci-edit-001</em> model.</p><p>For example, the following script converts input text from passive to active voice:</p><pre>text = &quot;My homework is being done&quot;<br><br>response = openai.Edit.create(<br>  model=&quot;text-davinci-edit-001&quot;,<br>  input = text,<br>  instruction=&quot;Convert the sentences to active voice.&quot;<br>)<br><br>text_edit = response.choices[0].text.strip()<br>print(text_edit)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*veKXZg9uNAs6BdHD-DOaTw.jpeg" /></figure><h4>Code edition</h4><p>You can also edit code using the openai.Edit module with the <em>code-davinci-edit-001</em> model.</p><p>For example, the following script asks the model to add more elements to a simple print(‘Hello World!’)</p><pre>input = &quot;print(&#39;Hello World!&#39;)&quot;<br><br>text_edit = openai.Edit.create(<br>  model=&quot;code-davinci-edit-001&quot;,<br>  instruction=&quot;Edit this Python code so that it prints: Hello World, I&#39;m Marc! &quot;,<br>    temperature=0.1<br>)<br><br>edited_code = text_edit.choices[0].text.strip()<br>print(edited_code)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*fUPm-iJCLnOSoXzz2AivDQ.jpeg" /></figure><p>Even though the <em>code-davinci-edit-001</em> works well at editing code, I would recommend using Chat Completion models for this task since OpenAI deprecated <em>Codex</em> and recommended that all users switch to <em>gpt-3.5-turbo </em>for code-related tasks.</p><h3>Embeddings</h3><p>Simply put, an embedding is a condensed representation of the meaning of a text. It is constructed as a numerical vector made up of floating point numbers. The distance between two embeddings in this vector space indicates how semantically similar the source texts are. Text embeddings, in essence, serve as a measure of how related different text strings are to one another. Small distances suggest high relatedness and large distances suggest low relatedness.</p><p>Embeddings are commonly used in machine learning for search, clustering, recommendations, anomaly detection, diversity measurement or classification.</p><p><em>OpenAI.Embedding </em>is an API of OpenAI gives access to both second-generation (-002) and first-generation (-001) embedding models. These models create an embedding vector representing the input text.</p><p>OpenAI embeddings are normalized to length 1, which means that you can obtain the distance by computing Cosine similarity or Euclidean distance. They will result in the identical rankings.</p><p><em>OpenAI.Embeddings</em> API endpoint is located at https://api.openai.com/v1/embedding.</p><p>The create() method from the <strong><em>openai.Embedding</em></strong> module allows you to create embedding vectors. The parameter values are as follows:<br>- <strong>model</strong>: the type of the embedding model. OpenAI recommends using <em>text-embedding-ada-002</em>,<em> </em>which is better and cheaper.<br>- <strong>input: </strong>the text string.</p><h4>Obtaining the embeddings</h4><p>To get an embedding, send your text string to the embeddings endpoint along with a choice of model. The response will contain an embedding, which you can extract, save, and use to develop a machine learning model.</p><p>In the following example, we generate an embedding vector from a poem about a turtle using the latest embedding <em>ada</em> model:</p><pre>response = openai.Embedding.create(<br>    input=&quot;I am a little turtle, I crawl so slow, I carry my house, wherever I go. When I get tired, I put in my head my legs and my tail, and I go to bed&quot;,<br>    model=&quot;text-embedding-ada-002&quot;<br>)<br>embeddings = response[&#39;data&#39;][0][&#39;embedding&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/206/1*L6lcFAcvRpEUMx9yYOswyw.jpeg" /></figure><h4>Advanced use cases</h4><p>OpenAI demonstrates advanced embedding use cases in its documentation, using the Amazon fine-food reviews dataset.</p><p>You can find their code samples on GitHub: <br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Obtain_dataset.ipynb">Obtaining the embeddings </a><br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Visualizing_embeddings_in_2D.ipynb">Data visualization in 2D</a><br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Regression_using_embeddings.ipynb">Embedding as a text feature encoder for ML algorithms </a><br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Classification_using_embeddings.ipynb">Classification using the embedding features</a><br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Zero-shot_classification_with_embeddings.ipynb">Zero-shot classification</a><br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/User_and_product_embeddings.ipynb">Obtaining user and product embeddings for cold-start recommendation</a><br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Clustering.ipynb">Clustering </a><br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Semantic_text_search_using_embeddings.ipynb">Text search using embeddings </a><br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Code_search.ipynb">Code search using embeddings </a><br>- <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Recommendation_using_embeddings.ipynb">Recommendations using embeddings</a></p><h3>Moderations</h3><p><em>OpenAI.Moderations </em>is an endpoint of OpenAI API that allows you to check whether textual content complies with <a href="https://openai.com/policies/usage-policies">OpenAI’s usage policies</a>. It is meant to assist developers in protecting their applications from abuse by providing access to GPT-based classifiers that detect undesired content.</p><p>OpenAI has released a <a href="https://arxiv.org/abs/2208.03274">technical paper </a>describing their methodology for developing the classifiers and the dataset used for evaluation.</p><p><em>OpenAI.Moderations</em> API endpoint is located at https://api.openai.com/v1/moderations.</p><p>The models classifies the following categories:<br>- <strong>hate</strong> : Hateful content based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste. <br>- <strong>hate/threatening </strong>: Hateful content that entails violence or serious harm to a certain group.<br>- <strong>self-harm</strong> : Content that encourages, promotes, or shows self-harming behaviors such as suicide, cutting, and eating disorders.<br>- <strong>sexual</strong> : Content intended to generate sexual arousal, such as sexual activity descriptions, or that promotes sexual services.<br>- <strong>sexual/minors</strong> : Sexual content that contains a minor under the age of 18.<br>- <strong>violence</strong> : Content that encourages or glorifies violence, or that celebrates the pain or humiliation of others.<br>- <strong>violence/graphic</strong>: Violent content with graphic depictions of death, violence, or major physical injury.</p><p><strong>Moderations output <br></strong>The output of the endpoint gives the following information: <br>- <strong>flagged</strong> : if the content is deemed to violate OpenAI’s usage policies, set to true ; otherwise, set to false.<br>- <strong>categories</strong>: contains a dictionnary of violation flags for each category of usage policies. The value is set to true if that category is being violated, and false otherwise. <br>- <strong>category_scores</strong>: contains a dictionnary of each category of usage policies with raw scores, which indicate how certain the model is that the input violates the given category. The values range from 0 to 1, with higher values signifying greater confidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*nczos4r-JSZhp_gDZFTMAw.jpeg" /></figure><h4>Content moderation example</h4><p>Let’s make a test of the <em>Moderations</em> API with a textual input that comes from the <a href="https://github.com/openai/moderation-api-release">evaluation dataset</a> used by OpenAI:</p><pre>response = openai.Moderation.create(<br>    input=&quot;I&#39;m a teenager and I&#39;m thinking about suic***&quot;<br>)<br>output = response[&quot;results&quot;][0]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*kV42ttiq-K9qiacMRcK8nw.jpeg" /></figure><p>The model accurately determines that this content violates the OpenAI usage policies (flagged: true), and we can see that it accurately identifies the self-harm (self-harm: 0.99).</p><h3>Images</h3><p><em>Images </em>is an API of OpenAI that provides access to the <a href="https://openai.com/research/dall-e">DALL·E model</a>s, which allow you to make images from text. DALL·E is a 12-billion parameter version of GPT-3 that was trained on a dataset of text-image pairs to produce images from a description in natural language. It is capable of creating realistic images and abstract art.</p><p>The <em>Images</em> API offers three endpoints to interact with images: <br>1) <em>Images.Generations</em>: making images from a text prompt,<br>2) <em>Images.Edits</em>: modifying an existing image with a new text prompt,<br>3) <em>Images.Variations</em>: creating variations of an existing image.</p><p>You can experiment DALL·E on <a href="https://labs.openai.com/">OpenAI labs</a>.</p><h4>Images Generations</h4><p><em>Images.Generations</em> API endpoint allows you to generate a new image from scratch from a text prompt.</p><p>A prompt is a request in natural language where you describe to the AI what type of image you desire. You must be as precise as possible so that the image generated by DALL·E matches your expectations.</p><p><em>Images.Generations</em> endpoint is located at: https://api.openai.com/v1/images/generations</p><p>The create() method of the openai.Image module has the following parameters: <br>- prompt: a text description of the desired image with a max length of 1000 characters. <br>- n: the number of images to generate, must be between 1 and 10. <br>- size: the size of the generated images. It caan be 256x256, 512x512, or 1024x1024. <br>- response_format: the format in which the generated images are returned. Must be one of url or b64_json.</p><p>Let’s ask DALL·E to generate a photo of Pinocchio:</p><pre>response = openai.Image.create(<br>  prompt=&quot;Photo of Pinocchio riding a bike on Central Park&quot;,<br>  n=1,<br>  size=&quot;1024x1024&quot;<br>)<br>image_url = response[&#39;data&#39;][0][&#39;url&#39;]<br>image_url #the URL of the generated image</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_tcYOnH9ZXUp0ph4vRcJQ.png" /></figure><h4>Images Edits</h4><p><em>Images.Edits</em> API endpoint allows you to edit or extend an image.</p><p>To do so, you must provide DALL·E with the original image as well as a mask with transparent areas indicating where the image should be edited.</p><p><em>Images.Edits</em> endpoint is located at: https://api.openai.com/v1/images/edits</p><p>The create_edit() method of the openai.Image module has the following parameters: <br>- image: the original image that you want to edit. It must be a square (aspect ration 1:1) PNG image that does not exceed 4MB in size. <br>- mask: the original image, with transparent areas indicating where you want edits. It must be in the a square PNG image with the same dimension as the original image. <br>- prompt: <br>- n: the number of images to generate, must be between 1 and 10. <br>- size: the size of the edited images. It can be 256x256, 512x512, or 1024x1024. <br>- response_format: the format in which the generated images are returned. Must be one of url or b64_json.</p><p>💡If you’re not sure how to make a mask, this website will assist you in doing it easily: <a href="https://ai-image-editor.netlify.app/">ai-mask-creator</a> (download both original and mask images at the end).</p><p>Let’s ask DALL·E to edit our image of Pinocchio:</p><pre>response = openai.Image.create_edit(<br>  image = open(&quot;Pinnochio_original.png&quot;, &quot;rb&quot;),<br>  mask = open(&quot;Pinnochio_mask.png&quot;,&quot;rb&quot;),<br>  prompt = &quot;Photo of Pinocchio riding a bike on Central Park with a hump on his back&quot;,<br>  n=1,<br>  size=&quot;1024x1024&quot;<br>)<br>image_url = response.data[0][&#39;url&#39;]<br>image_url</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GC2dpADcKD49Ln_ZpvILqg.png" /></figure><h4>Images Variations</h4><p><em>Images.Variations </em>is an OpenAI API that allows you to generate a variation of an image.</p><p>Similar to the <em>Images.Edits</em> endpoint, it accepts a square PNG image that does not exceed 4MB as input and generates a variant of it without a textual prompt.</p><p><em>Images.Variations</em> endpoint is located at: https://api.openai.com/v1/images/variations</p><p>Let’s make a variation of the Pinnochio image with this API:</p><pre>response = openai.Image.create_variation(<br>  image=open(&quot;Pinnochio_original.png&quot;, &quot;rb&quot;),<br>  n=1,<br>  size=&quot;1024x1024&quot;<br>)<br>image_url = response[&#39;data&#39;][0][&#39;url&#39;]<br>image_url</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PUi0m8jB0G9m3dJRePrDRA.png" /></figure><h3>Speech to Text</h3><p>The <em>Speech to Text</em> API, also known as the <em>Audio</em> API, allows you to access to OpenAI’s <a href="https://openai.com/research/whisper">Whisper model</a>.</p><p>Whisper is an <a href="https://github.com/openai/whisper">open-source</a> automatic speech recognition (ASR) system trained by OpenAI on multilingual and multitask supervised data collected from the web.</p><p>This Speech to text API offers two endpoints: <br>- <em>Audio.Transcriptions</em>: transcribe audio in multiple languages, <br>- <em>Audio.Translations</em>: translate and transcribe the audio into English.</p><p>File uploads are currently limited to 25 MB and the following input file types are supported: mp3, mp4, mpeg, mpga, m4a, wav, and webm.</p><p>The API supports <a href="https://platform.openai.com/docs/guides/speech-to-text/supported-languages">98 languages</a>, however only about 60 have a greater 50% word error rate (WER). For languages below this rate, the model may deliver low quality results.</p><h4>Transcription</h4><p>The <em>Audio.Transcriptions</em> endpoint allows you to transcribe audio. The API takes the audio file as input and outputs the transcription.</p><p><em>Audio.Transcriptions</em> endpoint is located at: https://api.openai.com/v1/images/variations</p><p>The transcribe() method of the openai.Audio module has the following parameters:<br>- file: used to specify the audio file that you want to transcribe.<br>- model : used to select the transcription model to be used (<em>whisper-1</em> by default). <br>- response_format: used to define the format in which you want the transcription response to be returned. It can be simple text (<em>text</em>), subrip subtitles (<em>srt</em>), video text track subtitles (<em>vtt</em>) or metadata (<em>verbose_json</em>). <br>- language: used to specify the language of the audio file you want to transcribe. It helps the model understand and interpret the audio accurately but you can leave it blank. <br>- prompt: an optional text in English to guide the model. <br>- temperature: used to define the sampling temperature, which should be between 0 and 1. Higher values will provide more random results while lower values would produce more concentrated and deterministic results.</p><p>Let’s try the transcription endpoint with <a href="https://commons.wikimedia.org/wiki/File:Armstrong_Small_Step.ogg">Neil Armstrong’s famous first words on the moon</a>:</p><pre>audio_file= open(&quot;Armstrong_Small_Step.mp3&quot;, &quot;rb&quot;)<br><br>transcript = openai.Audio.transcribe(<br>    file=audio_file,<br>    model=&quot;whisper-1&quot;,<br>    response_format=&quot;text&quot;,<br>)<br><br>transcript</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*na8UqOJW-teIb0SyE-SWVQ.jpeg" /></figure><p>In the previous example, we have set the output response format to be simple text. Let’s try the subrip subtitles (<em>srt</em>):</p><pre>audio_file= open(&quot;Armstrong_Small_Step.mp3&quot;, &quot;rb&quot;)<br><br>transcript = openai.Audio.transcribe(<br>    file=audio_file,<br>    model=&quot;whisper-1&quot;,<br>    response_format=&quot;srt&quot;,<br>)<br><br>transcript</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*vWbOKODLk4dM0HbQeaMjLQ.jpeg" /></figure><h4>Translation</h4><p>The <em>Audio.Translations </em>accepts audio files in any languages and transcribes them into English. It is distinct from the <em>Audio.Transcriptions</em> endpoint since the transcription is not provided in the original input language but is translated into English.</p><p><em>Audio.Translations</em> endpoint is located at: https://api.openai.com/v1/images/variations</p><p>The translate() method of the openai.Audio module has the following parameters:<br>- file: used to specify the audio file that you want to transcribe.<br>- model : used to select the transcription model to be used (<em>whisper-1</em> by default). <br>- response_format: used to define the format in which you want the transcription response to be returned. It can be simple text (<em>text</em>), subrip subtitles (<em>srt</em>), video text track subtitles (<em>vtt</em>) or metadata (<em>verbose_json</em>). <br>- prompt: an optional text in English to guide the model. <br>- temperature: used to define the sampling temperature, which should be between 0 and 1. Higher values will provide more random results while lower values would produce more concentrated and deterministic results.</p><p>Let’s try the <em>Translation</em> endpoint with <a href="https://commons.wikimedia.org/wiki/File:Xabier_paya.wav">Xabier Paya audio file sample</a> (Basque language):</p><pre>audio_file= open(&quot;Xabier_paya.wav&quot;, &quot;rb&quot;)<br><br>translation = openai.Audio.translate(<br>    file=audio_file,<br>    model=&quot;whisper-1&quot;,<br>    response_format=&quot;text&quot;<br>)<br><br>translation</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*97lr-nWvhLzpjvaBWgqvyA.jpeg" /></figure><h3>Conclusion</h3><p>✅ This article provides a complete review of the OpenAI API’s capabilities by delving into its features and functionalities. Without a question, the OpenAI API has enormous potential, providing new capabilities for businesses, researchers, and developers. And as OpenAI progresses, we should expect to see even more functionalities in the future.</p><p>💡May this post be a beneficial resource for you, allowing you to fully utilize the OpenAI API and catapult your projects to new heights of possibilities!</p><p>➡️Your input is invaluable in helping me improve this article. If you spot any errors or have any suggestions, please don’t hesitate to contact me!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d933fb770f95" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fetching and Formatting Phone Numbers in Python]]></title>
            <link>https://medium.com/@marc.bolle/fetching-and-formatting-phone-numbers-in-python-794ae16ab198?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/794ae16ab198</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Sun, 11 Jun 2023 16:47:42 GMT</pubDate>
            <atom:updated>2023-07-09T09:35:24.710Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*aFDy7Cyfzho50x9neejzcw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/fr/@anniespratt">Annie Spratt</a> on <a href="https://unsplash.com/">Unsplash</a>.</figcaption></figure><p>When it comes to handling phone numbers in Python, data practitioners face significant challenges. It can be hard to extract phone numbers from text, standardize them into a specific format and check their accuracy. This can be particularly daunting due to the varying formats and country codes.</p><p>However, there’s good news! A handy Python library called <em>phonenumbers</em> simplifies the process of handling phone numbers, making the task considerably easier.</p><p>In this article, I will demonstrate how you can use <em>phonenumbers</em> to manipulate phone numbers effectively using Python.</p><h4>Table of Contents</h4><p>◦ Converting Python String to PhoneNumber Object<br>◦ Extracting Phone Numbers from Text<br>◦ Obtaining Information from Phone Number<br>◦ Validating Phone Numbers<br>◦ Formatting Phone Numbers</p><h4>The phonenumbers Python Library</h4><p>The Python library <a href="https://github.com/daviddrysdale/python-phonenumbers"><em>phonenumbers</em></a> provides functionality for parsing, formatting, and validating phone numbers. It is used to handle phone numbers in a standardized way, regardless of their initial formatting or international dialing codes.</p><p><em>phonenumbers</em> is based on Google’s <a href="https://github.com/google/libphonenumber">libphonenumber</a> library, which is written in Java. It acts as a port of the original library and provides a Pythonic interface to work with phone numbers.</p><p>In this tutorial, we’ll learn how to extract, validate and format phone numbers in Python.</p><h4>Installing phonenumbers</h4><p>First things first, we have to install the <em>phonenumbers</em> library.</p><p>You can install it from <a href="https://pypi.org/project/phonenumbers/">PyPI</a> as follow:</p><pre>pip install phonenumbers</pre><p>You can also install it over Conda:</p><pre>conda install -c conda-forge phonenumbers</pre><h3>Converting Python String to PhoneNumber Object</h3><p>Phone numbers are typically collected from web forms as text, which are then represented as strings in Python.</p><p>The first step, with the <em>phonenumbers</em> module, is to parse the number. This parsing operation transforms the Python string into a PhoneNumber object, allowing us to use it for formatting, validation, and other functionalies.</p><p>To do that, we use the phonenumbers.parse() method. It takes two arguments: <br>- A phone number as a string: The number can be in the international format (with +1 in the US for instance) or in a domestic format.<br>– A country code in ISO 3166–1 alpha-2 format (US for USA for instance).</p><p>To parse a phone number in the domestic format, specify the country code as the second augment of the method as follow:</p><pre>phone_number_string = &quot;(209) 921-6581&quot;<br>parsed_number = phonenumbers.parse(phone_number_string, &quot;US&quot;)<br>print(parsed_number)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*6AV94Jbhlgy_XHaN9u1jww.jpeg" /></figure><p>To parse a phone number in the international format, just add the phone number:</p><pre>phone_number_string = &quot;+1 209-921-6581&quot;<br>parsed_number = phonenumbers.parse(phone_number_string)<br>print(parsed_number)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*6AV94Jbhlgy_XHaN9u1jww.jpeg" /></figure><h3>Extracting Phone Numbers from Text</h3><p>Businesses often collect phone numbers from text, such as emails. In this case, we can use the phonenumbers library to extract phone numbers from a text.</p><p>The methodphonenubers.PhoneNumberMatcher() iterates the text and extract any phone number from it.</p><p>This method accurately retrieves the phone numbers, whether they are in domestic or international format, or when there are special characters or digits attached to them.</p><p>We must specify a country code so the method detects phone numbers from a particular country:</p><pre>text = &quot;Hi Jane, thank you for contacting us to learn more about our services!<br>Please feel free to reach us at 020 8366 1177 or 020 8785 7465.<br>We want to assure you that there will be no obligation to sign up for our newsletter following the call!<br>Best regards.&quot;<br><br># Use the PhoneNumberMatcher class to find phone numbers in the &#39;text&#39; string<br># The &quot;GB&quot; argument specifies that it should look for British phone numbers <br>numbers = phonenumbers.PhoneNumberMatcher(text, &quot;GB&quot;)<br><br># Iterate over the &#39;numbers&#39; iterable, which contains the phone numbers found in the &#39;text&#39;, and print each phone number<br>for number in numbers:<br>    print(number)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*xpUk_9e6VzqdYctrBiOmiA.jpeg" /></figure><p>To extract phone numbers from different countries within the text, we can utilize a for loop that includes several PhoneNumberMatcher() methods as follow:</p><pre>text = &quot;Hi Josh, thank you for contacting us to learn more about our services!<br>The best way to learn more about us is by giving us a call.<br>Feel free to reach out to us at 020 8366 1177 if you are in the UK or at (484) 445-8165 if you are in the USA.<br>We want to assure you that there will be no obligation to sign up for our newsletter after the call!<br>Best regards.&quot;<br><br># Iterate over each character in the &#39;text&#39; string and find GB and US <br>for numbers in text:<br>    numbers_GB = phonenumbers.PhoneNumberMatcher(text, &quot;GB&quot;)<br>    numbers_USA = phonenumbers.PhoneNumberMatcher(text, &quot;US&quot;)<br><br># Print the GB phone number found<br>for number in numbers_GB:<br>    print(&quot;GB number: &quot;, number)<br><br># Print the US phone number found<br>for number in numbers_USA:<br>    print(&quot;US number: &quot;, number)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*I7Mvt8CXFyLvru8zXtU0ZQ.jpeg" /></figure><h3>Obtaining Information from Phone Number</h3><p>A phone number holds valuable information for businesses, including details such as the country, region, time zone and carrier associated with it. This information can prove very useful for marketers.</p><h4><strong>Country Code and National Number</strong></h4><p>Printing the PhoneNumber object will return the country code as well as the number in its domestic format:</p><pre>phone_number = phonenumbers.parse(&quot;+1 209-921-6581&quot;)<br>print(phone_number)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*w3CHreQO4YZHxuPIWoNVhw.jpeg" /></figure><h4><strong>Location of the Phone Number</strong></h4><p>The geocoder.description_for_number() method returns the country, the region or the city of the phone number, depending on what is available:</p><pre>from phonenumbers import geocoder<br><br>phone_number = phonenumbers.parse(&quot;955 967 7649&quot;,&quot;BR&quot;)<br>print(geocoder.description_for_number(phone_number,&#39;en&#39;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/70/1*Hef8VMNEcbZERM7c_OqvAA.jpeg" /></figure><p>You can choose the language in which the location will be shown. Let’s try with Russian:</p><pre>from phonenumbers import geocoder<br><br>phone_number = phonenumbers.parse(&quot;8127774566&quot;,&quot;RU&quot;)<br>print(geocoder.description_for_number(phone_number,&#39;ru&#39;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*ygCRo2ujVlnCX7eiJMmGxg.jpeg" /></figure><h4><strong>Time Zone of a Phone Number</strong></h4><p>The timezone.time_zones_for_number() method returns the time zone of the number number, which can prove very useful for marketers:</p><pre>from phonenumbers import timezone<br><br>phone_number = phonenumbers.parse(&quot;+1 646-754-3010&quot;)<br>print(timezone.time_zones_for_number(phone_number))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*b1TyjDjhcoosoJTNeJtQkw.jpeg" /></figure><h4><strong>Carrier of a Phone Number</strong></h4><p>The carrier.name_for_number() method returns the mobile carrier of the phone number, if available:</p><pre>from phonenumbers import carrier<br><br>my_number = phonenumbers.parse(&quot;0685746520&quot;, &quot;FR&quot;)<br>carrier = carrier.name_for_number(my_number, &#39;en&#39;)<br>print(carrier)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/72/1*EEdYl3cEnphsFYjFqdPyiw.jpeg" /></figure><h3>Validating Phone Numbers</h3><p>Collecting active phone numbers capable of receiving SMS or calls is crucial for businesses. That’s why you may need to validate phone numbers before utilizing them.</p><p>The <em>phonenumbers</em> library has two convenient methods that do the job for you:</p><h4><strong>Possible phone number</strong></h4><p>phonenumbers.is_possible_number() method checks if the given number can be a real telephone number by looking at its length. It returns either True or False:</p><pre>my_number1 = phonenumbers.parse(&quot;6467543010&quot;, &quot;US&quot;)<br>print(phonenumbers.is_possible_number(my_number1))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/50/1*PyNsEE_ToXRM3TGr2hoZvA.jpeg" /></figure><pre>my_number2 = phonenumbers.parse(&quot;64675430100&quot;,&quot;US&quot;)<br>print(phonenumbers.is_possible_number(my_number2))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/50/1*Y2Sx9Pkw_ySnvjWcTt1VkQ.jpeg" /></figure><h4>Valid phone number</h4><p>phonenumbers.is_valid_number() method checks if the phone number matches with the specified country code:</p><pre>my_number = phonenumbers.parse(&quot;0672874731&quot;,&quot;US&quot;)<br>print(&quot;Possible number: &quot;,phonenumbers.is_possible_number(my_number))<br>print(&quot;Valid number: &quot;, phonenumbers.is_valid_number(my_number))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/1*l17T1LW1ShRDeQ2gMuqW7w.jpeg" /></figure><h3>Formatting Phone Numbers</h3><p>Phone numbers are recorded in diverse formats by individuals, leading to challenges in manipulating them effectively using Python.</p><p>Fortunately, the <em>phonenumbers</em> library has got us covered with thephonenumbers.format_number() function that makes it simple to format phone numbers to a desired format.</p><p>It offers three types of format:<br>– Domestic format,<br>– International format,<br>– E164 format (international format with 15 digits).</p><h4>Domestic Formatting</h4><p>Let’s start by formatting a phone number into a domestic format using the phonenumbers.PhoneNumberFormat.NATIONAL constant:</p><pre>import phonenumbers<br><br># Defining the US phone number in international format <br>international_number = phonenumbers.parse(&quot;+1 646 980 6705&quot;, &quot;US&quot;)<br><br># Formatting the phone number into the national format <br>national_number = phonenumbers.format_number(international_number, phonenumbers.PhoneNumberFormat.NATIONAL)<br><br>print(national_number)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*j-HmySIefXeWIAmeYRAtfw.jpeg" /></figure><h4>International Formatting</h4><p>We can also format a phone number into an international format using the phonenumbers.PhoneNumberFormat.INTERNATIONAL constant:</p><pre>import phonenumbers<br><br># Defining the US phone number in national format<br>national_number = phonenumbers.parse(&quot;(646) 980-6705&quot;, &quot;US&quot;)<br><br># Formatting the phone number into the international format<br>international_number = phonenumbers.format_number(national_number, phonenumbers.PhoneNumberFormat.INTERNATIONAL)<br><br>print(international_number)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/138/1*uN8B4RagQbR8nZpbnftiVA.jpeg" /></figure><h4>E164 Formatting</h4><p>The E164 phone format is a international standard for phone numbers. It consists of the country code followed by the national number, with no special characters or punctuation.</p><p>Let’s format a phone number into E164 format using the phonenumbers.PhoneNumberFormat.E164 constant:</p><pre>import phonenumbers<br><br># Defining the US phone number in national format<br>national_number = phonenumbers.parse(&quot;(646) 980-6705&quot;, &quot;US&quot;)<br><br># Formatting the phone number into E164 format<br>E164_number = phonenumbers.format_number(national_number, phonenumbers.PhoneNumberFormat.E164)<br><br>print(E164_number)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/104/1*ieP6Sv4Nlgewd2Skbg_Tug.jpeg" /></figure><h4>Real-time Formatting</h4><p>The phonenumbers.AsYouTypeFormatter() that allows you to format a phone number as the user types it. It automatically ajdusts the number based on the input.</p><p>This feature is useful if you have a website or an application where users type their phone numbers.</p><pre>import phonenumbers<br><br>formatter = phonenumbers.AsYouTypeFormatter(&quot;US&quot;)<br><br>&gt;&gt;&gt; formatter.input_digit(&quot;6&quot;)<br>&#39;6&#39;<br>&gt;&gt;&gt; formatter.input_digit(&quot;5&quot;)<br>&#39;65&#39;<br>&gt;&gt;&gt; formatter.input_digit(&quot;0&quot;)<br>&#39;650&#39;<br>&gt;&gt;&gt; formatter.input_digit(&quot;4&quot;)<br>&#39;650 4&#39;<br>&gt;&gt;&gt; formatter.input_digit(&quot;1&quot;)<br>&#39;650 41&#39;<br>&gt;&gt;&gt; formatter.input_digit(&quot;6&quot;)<br>&#39;650 416&#39;<br>&gt;&gt;&gt; formatter.input_digit(&quot;4&quot;)<br>&#39;650-4164&#39;<br>&gt;&gt;&gt; formatter.input_digit(&quot;8&quot;)<br>&#39;(650) 416-48&#39;<br>&gt;&gt;&gt; formatter.input_digit(&quot;7&quot;)<br>&#39;(650) 416-487&#39;<br>&gt;&gt;&gt; formatter.input_digit(&quot;9&quot;)<br>&#39;(650) 416-4879&#39;</pre><h3>Conclusion</h3><p>I hope you have enjoyed reading this blog. By diving into the features of the <em>phonenumbers</em> module, we have explored a range of techniques to handle phone numbers efficiently in Python. Thank you for joining me on this journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=794ae16ab198" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Augmentation in Python with NLPAUG]]></title>
            <link>https://medium.com/@marc.bolle/text-augmentation-in-python-with-nlpaug-48c3eebacf46?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/48c3eebacf46</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Sat, 03 Jun 2023 16:58:10 GMT</pubDate>
            <atom:updated>2023-07-09T09:36:02.014Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*NbVtnz-Yi3fQsLuZE_H1Yg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/fr/@sincerelymedia">Sincerely Media</a> on <a href="https://unsplash.com/">Unsplash</a>.</figcaption></figure><h3>What is Text Augmentation?</h3><p>In machine learning, the quality of a training dataset mostly determines how effective a model will be. But the reality, we often don’t have enough diverse data, which affects the model’s accuracy. That’s where data augmentation comes to the rescue.</p><blockquote>“Data augmentation in data analysis are techniques used to increase the amount of data by adding slightly modified copies of already existing data or newly created synthetic data from existing data.”</blockquote><p>As this <a href="https://en.wikipedia.org/wiki/Data_augmentation">Wikipedia definition</a> suggests, data augmentation refers to the process of generating new, synthetic data samples by applying various transformations to the existing data.</p><p>The same goes for text data. It is typical to have too little text input to train a language model. For example, in text classification, one of the most common tasks in natural language processing (NLP), a large amount of data is required to train a model. That’s why text augmentation techniques are commonly used to improve the performance of NLP models.</p><p>In this article, I’ll show you how to apply textual augmentation using the Python library NLPAUG.</p><h3>NLPAUG Text Augmentation Library</h3><p><a href="https://github.com/makcedward/nlpaug">NLPAUG</a> is an open-source Python library developed by Edward Ma that makes text augmentation very simple. The library offers a range of text augmenters for character, word and sentence that you can apply in just 3 lines of code.</p><h4>Installation</h4><p>NLPAUG can be installed from <a href="https://pypi.org/project/nlpaug/">PyPI </a>as follow:</p><pre>pip install nlpaug</pre><p>It can also be installed over <a href="https://anaconda.org/conda-forge/nlpaug">Conda</a>:</p><pre>conda install -c makecedward nlpaug</pre><p>If you plan to perform synonym or antonym augmentation, you need to have the <a href="https://www.nltk.org/">NLTK library</a> installed. You can install it from <a href="https://pypi.org/project/nltk/">PyPI</a> as follow:</p><pre>pip install nltk</pre><p>If you plan to perform word embeddings augmentation, you need to download a pre-trained model. You can download <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a>, <a href="https://en.wikipedia.org/wiki/GloVe">glove</a> or <a href="https://fasttext.cc/">fasttext</a> as follow:</p><pre>from nlpaug.util.file.download import DownloadUtil<br>DownloadUtil.download_word2vec(dest_dir=&#39;.&#39;) # Download word2vec model<br>DownloadUtil.download_glove(model_name=&#39;glove.6B&#39;, dest_dir=&#39;.&#39;) # Download GloVe model<br>DownloadUtil.download_fasttext(model_name=&#39;wiki-news-300d-1M&#39;, dest_dir=&#39;.&#39;) # Download fasttext model<br><br>pip install gensim</pre><p>If you plan to perform back translation augmentation on sentences, you must install the <a href="https://github.com/alvations/sacremoses">SacreMoses library</a>. You can install it as follow:</p><pre>pip install sacremoses</pre><h4>Augmentation operations</h4><p>Each NLPAUG augmentater employs one or more operations :</p><ul><li><em>Insert</em>: adding new characters, words or sentences to the original text.</li><li><em>Substitute</em>: replacing characters, words or sentences to the original text.</li><li><em>Swap</em>: exchanging the positions of adjacent characters, words or sentences to the original text.</li><li><em>Delete</em>: removing characters, words or sentences from the original text.</li><li><em>Split</em>: dividing words into two or more parts.</li><li><em>Crop</em>: removing a portion of sentences.</li></ul><h4>Table of Contents</h4><ol><li><strong>Character Augmenter </strong><br>1.1 Keyboard Augmenter <br>1.2 Optical Character Recognition Augmenter (OCR)<br>1.3 Random Augmenter</li><li><strong>Word Augmenter</strong><br>2.1 Synonym Augmenter<br>2.2 Antonym Augmenter<br>2.3 Split Augmenter<br>2.4 Spelling Augmenter<br>2.5 Reserved Word Augmenter<br>2.6 Word Embeddings Augmenter<br>2.7 Contextual Word Embeddings Augmenter<br>2.8 Back Translation Augmenter<br>2.9 TF-IDF Augmenter<br>2.10 Random Word Augmenter</li><li><strong>Sentence Augmenter </strong><br>3.1 Abstractive Summarization Augmenter<br>3.2 Contextual Word Embeddings Augmenter<br>3.3 LAMBADA Augmenter<br>3.4 Random Augmenter</li></ol><h3>1) Character Augmentrer</h3><p>A character augmenter is a technique used in text augmentation to generate variations of text data by applying specified changes at the character level. Character augmenters introduce changes such as character replacements, deletions, insertions, or other adjustments.</p><p>Character augmentation uses rules or probabilistic approaches to determine which characters to change and how to change them.</p><p>NLPAUG’s nlpaug.augmenter.charmodule provides three character augmentation techniques: Keyboard augmenter, Optical character recognition augmenter, and Random augmenter.</p><h4>1.1 Keyboard Augmenter</h4><p>Keyboard augmentation is a text augmentation technique that augments text data by inserting typing errors. At the character-level, it substitutes characters in words with ones at a close distance on a keyboard, mimicking the likelihood of making keyboard mistakes while typing.</p><p>As such, this method generates augmented text data that resembles typographical errors typically encountered in real-world circumstances and improves the quality of training data.</p><p>NLPAUG’s char.KeywordAug() function applies typo error simulation to textual input:</p><pre>import nlpaug.augmenter.char as nac<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Keyboard <br>aug = nac.KeyboardAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*1Ol7Vvw0JJX0De5_ODqT7g.jpeg" /></figure><h4>1.2 Optical Character Recognition Augmenter (OCR)</h4><p>Character Recognition (OCR) is a technology that converts images or documents into text that can be read by machines. It does that by analyzing the image’s visual patterns, identifiying individual characters or words, and converting them into text.</p><p>OCR models commonly make mistakes, such as confusing ‘0’ with ‘o’ or ‘I’ with ‘1’. That’s why, when training an OCR model, simulating the errors and inconsistencies that can arise throughout the OCR process is essential.</p><p>NLPAUG’s char.OcrAug() augmenter introduces OCR errors at the character level in the text by substituting characters:</p><pre>import nlpaug.augmenter.char as nac<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># OCR<br>aug = nac.OcrAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*kyWH4aZ3aHiHB6wsnxe_mw.jpeg" /></figure><h4>1.3 Random Augmenter</h4><p>The random augmenter introduces random variations to the text data. It simulates different types of errors that can occur naturally in language.</p><p>NLPAUG’s RandomCharAug() function may replace characters with similar ones, swap adjacent characters randomly, or delete or insert random characters inside the text. These variations contribute to a more diversified training dataset and improve the robustness of your model when dealing with real-life data.</p><p>You can choose a single type of operation using the action parameter: <em>insert</em>, <em>substitute</em>, <em>swap</em>, <em>delete</em>.</p><pre>import nlpaug.augmenter.char as nac<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Random - Character level - insert<br>aug = nac.RandomCharAug(action=&quot;insert&quot;)<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*yeKyGmXgAhcDlwznW1Wvhw.jpeg" /></figure><h3>2) Word Augmenter</h3><p>Text augmentation can also be performed at the word level, by introducing variations or modifications to words.</p><p>Word augmentation techniques include replacing words with synonyms, inserting or deleting words or even changing the order of words in a sentences.</p><p>Word augmentation can be useful for training models for NLP tasks, such as text classification and sentiment analysis, where a diverse training dataset can enhance the model performance on real-world text.</p><p>NLPAUG’s nlpaug.augmenter.word module provides ten word augmentation techniques: synonym augmenter, antonym augmenter, split augmenter, spelling augmenter, reserved word augmenter, word embeddings augmenter, contextual word embeddings augmenter, back-translation augmenter, TF-IDF augmenter, random word augmenter.</p><h4>2.1 Synonym Augmenter</h4><p>Synonym augmentation is a text augmentation technique that substitutes words with their synonyms. It contributes to the diversification of the vocabulary utilized in training your language model.</p><p>NLPAUG’s SynonymAug() substitutes similar words according to WordNet/PPDB synonyms. You can choose which model to use using the aug_src parameter:</p><pre>import nlpaug.augmenter.word as naw<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Synonym<br>aug = naw.SynonymAug(aug_src=&#39;wordnet&#39;)<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*FUoTT67EXg_nhBZbVcryEw.jpeg" /></figure><h4>2.2 Antonym Augmenter</h4><p>Antonym augmentation is a text augmentation technique that replaces words in a text with their antonyms, which are words with opposite meanings. Antonym augmentation contributes to the creation of a diversified training dataset for your language model.</p><p>NLPAUG’s AntonymAug() function substitutes opposite meaning words according to WordNet antonyms:</p><pre>import nlpaug.augmenter.word as naw<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Antonym<br>aug = naw.AntonymAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*NHJDbGPod_YMUevBLjAAhA.jpeg" /></figure><h4>2.3 Split Augmenter</h4><p>Split augmentation is a text augmentation technique that changes the word structure by randomly dividing words into two or more parts. Thiw word augmentation method results in variations in the original text and contributes to making a diversified text for training a language model.</p><p>NLPAUG’s SplitAug() function applies word splitting operation to textual input:</p><pre>import nlpaug.augmenter.word as naw<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Split<br>aug = naw.SplitAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*-cekufeIgv5HpvSK11fcQw.jpeg" /></figure><h4>2.4 Spelling Augmenter</h4><p>Spelling augmentation is a text augmentation technique that introduces mistakes in the spelling of words. Introducing alternative spellings can be useful for training language models.</p><p>NLPAUG’s SpellingAug() substitutes word according to spelling mistake dictionary:</p><pre>import nlpaug.augmenter.word as naw<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Spelling<br>aug = naw.SpellingAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*4qAPX6YrNh-KwhkqVRMnsA.jpeg" /></figure><h4>2.5 Reserved Word Augmenter</h4><p>The reserved word augmentation technique applies target word replacement operation to the text.</p><p>This technique allows you to choose which words have to remain unchanged during the augmentation process. This technique is useful when you want to apply augmentation on specific parts of the text while preserving others.</p><p>NLPAUG’s ReservedAug() function replaces the words that aren’t defined in a list which is passed as an argument to the reserved_token parameter:</p><pre>import nlpaug.augmenter.word as naw<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Reserved words<br>reserved_tokens = [&#39;lion&#39;, &#39;windows&#39;]  # Specify the reserved tokens here<br>aug = naw.ReservedAug(reserved_tokens=reserved_tokens)<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*Bi_iGAHl1iwVovsJEpoDYw.jpeg" /></figure><h4>2.6 Word Embeddings Augmenter</h4><p>Word embedding augmentation is a text augmentation technique that replaces certain words with semantically comparable alternatives. This technique helps to diversity the text’s language.</p><p>NLPAUG’s WordEmbsAug() function applies operation to textual input based on word embeddings. You can specify one type of operation to apply using the action parameter: <em>insert</em> or <em>substitute. </em>You can also specify the word embedding model to be used, such as Word2Vec, GloVe, or fastText.</p><pre>import nlpaug.augmenter.word as naw<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Word embeddings<br>aug = naw.WordEmbsAug(model_type=&#39;word2vec&#39;, model_path=gn_vec_path, action=&#39;insert&#39;)<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*yYpFpobUsZP0BSXXtM97XQ.jpeg" /></figure><h4>2.7 Contextual Word Embeddings Augmenter</h4><p>Contextual word embeddings augmentation is a text augmentation technique that understands the meaning and context of words within a sentence and either feed their surroundings or replace them with top n similar words from pre-trained language models such as BERT, DistilBERT, RoBERTa or XLNet.</p><p>NLPAUG’s context_word_embs() function leverages contextual word embeddings to find top n similar word for augmentation. You can specify one type of operation to apply using the action parameter: <em>insert</em> or <em>substitute.</em></p><pre>import nlpaug.augmenter.word as naw<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Contextual Word Embeddings - Word level<br>aug = naw.ContextualWordEmbsAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*zKz1qSgE2Zneb9QDKsGg4w.jpeg" /></figure><h4>2.8 Back Translation Augmenter</h4><p>Back translation augmentation is a text augmentation technique that uses a pre-trained machine learning model to translate the text from one language to another and then back to the original language.</p><p>The back translation augmenter adds diversity into the text by doing this round-trip translation. This text augmentation technique can be useful when you need to enhance the size of your training data.</p><p>NLPAUG’s Back_TranslationAug() function leverages two translation models for augmentation:</p><pre>import nlpaug.augmenter.word as naw<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Back translation<br>aug = naw.BackTranslationAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*nQXRYAvoJKZ14utr4Lo5ag.jpeg" /></figure><h4>2.9 TF-IDF Augmenter</h4><p>Another text augmentation technique is term frequency-inverse document frequency (TF-IDF).</p><p>This augmenter computes the TF-IDF scores of the words in a text by multiplying the Term Frequency (TF) by the Inverse Document Frequency (IDF). These scores indicate the significance of a word within a text.</p><p>Words having a low TF-IDF score are thought to be less relevant to the meaning of the text and will be more likely to be substituted by other low-scoring words. This technique creates variations that do not change the meaning of the original text.</p><p>A TF-IDF model has to be trained on a dataset, after which you can use NLPAUG’s TfIdfAug() function to insert or substitute words based on the scores.</p><p>In the following example, I train a TF-IDF model on Scikit-Learn’s 20 News Groups dataset and uses it to augment our sentence using NLPAUG TF-IDF augmenter:</p><pre>import sklearn.datasets<br>import re<br>import nlpaug.augmenter.word as naw<br>import nlpaug.model.word_stats as nmw<br><br># Defining a tokenizer function to extract word tokens<br>def _tokenizer(text, token_pattern=r&quot;(?u)\b\w\w+\b&quot;):<br>    token_pattern = re.compile(token_pattern)<br>    return token_pattern.findall(text)<br><br># Load sample data (Scikit Learn 20 News Groups)<br>train_data = sklearn.datasets.fetch_20newsgroups(subset=&#39;train&#39;, remove=(&#39;headers&#39;, &#39;footers&#39;, &#39;quotes&#39;))<br>train_x = train_data.data<br><br># Tokenize input<br>train_x_tokens = [_tokenizer(x) for x in train_x]<br><br># Train TF-IDF model<br>tfidf_model = nmw.TfIdf()<br>tfidf_model.train(train_x_tokens)<br>tfidf_model.save(&#39;.&#39;)<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Augment the text with TFIDF augmenter<br>aug = naw.TfIdfAug(model_path=&#39;.&#39;, tokenizer=_tokenizer)<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*K2ozpTQ8bXZRi2nLnriayg.jpeg" /></figure><h4>2.10 Random Word Augmenter</h4><p>The random word augmenter applies random word-level augmentation behaviors to the input text. It introduces diversity in the text by using multiple text augmentation techniques randomly.</p><p>NLPAUG’s RandomWordAug() function may swap, crop or delete words randomly. These variations contribute to a more diversified training dataset.</p><p>You can choose a single type of operation using the action parameter: <em>swap</em>, <em>crop</em>, <em>delete</em>.</p><pre>import nlpaug.augmenter.word as naw<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Random<br>aug = naw.RandomWordAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*WhvZ-HfEdzgFT_5m7XZCrg.jpeg" /></figure><h3>3) Sentence Augmenter</h3><p>Text augmentation can also be done at the sentence level, which involves introducing variations or adjustments to sentences while maintaining the general context and meaning of the text.</p><p>Examples of sentence augmentation techniques include inserting words depending on context or rearranging the order of words within a sentence while maintaining grammatical accuracy.</p><p>Sentence augmentation, like character and word augmentations, can help train models for NLP tasks like text classification and sentiment analysis, where a diversified training dataset can improve the model’s performance on real-world text.</p><p>NLPAUG’s nlpaug.augmenter.sentence module provides four sentence augmenters:</p><pre>nlpaug.augmenter.sentence</pre><h4>3.1 Abstractive Summarization Augmenter</h4><p>The abstractive summarization augmenter is a text augmentation technique that uses abstractive summarization to introduce variations to textual input.</p><p>Abstractive summarization is the task of producing a succinct summary of a given text that captures the key concepts and meaning of the original content is known. The summary may include new phrases and sentences that do not appear in the source text.</p><p>Text augmentation with abstractive summarization can bring diversity and variances in phrase structures and content, which might be useful for training an NLP model.</p><p>NLPAUG’s AbstSummAug() Summarize article by abstractive summarization method:</p><pre>import nlpaug.augmenter.sentence as nas<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Abstractive Summarization<br>aug = nas.AbstSummAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*5xR0JW7af6CbWKAge3-1Xw.jpeg" /></figure><h4>3.2 Contextual Word Embeddings Augmenter</h4><p>Contextual word embeddings augmentation is a text augmentation techniques that understand the meaning and context of words within a sentence in order to insert new words.</p><p>In the context of sentence-level augmentation, contextual word embeddings add new sentences with top n similar words from pre-trained models such as XLNet, GPT2 or DistilGPT2.</p><p>NLPAUG’s ContextualWordEmbsForSentenceAug()function leverages contextual word embeddings to insert new sentences:</p><pre>import nlpaug.augmenter.sentence as nas<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Contextual Word Embeddings - Sentence level<br>aug = nas.ContextualWordEmbsForSentenceAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*8yizam08vLIVbOzcGjTDkw.jpeg" /></figure><h4>3.3 LAMBADA Augmenter</h4><p>Language-model-based data augmentation (LAMBADA) is a text augmentation technique that uses a pre-trained language model to predict missing words in a sentence and generate alternative sentence variations.</p><p>The LAMBADA technique (<a href="https://arxiv.org/abs/1911.03118">Anaby-Tavor et al., 2019</a>) is inspired by the LAMBADA dataset, which consists of passages from books where the last word is removed. The goal is to predict the missing word, requiring a deep understanding of the context. LAMBADA text augmentation leverages language models, such as GPT or BERT, to generate new sentences by predicting the missing word given the context.</p><p>Using LAMBADA augmenter is a excellent way to introduce diversity in the sentence structure and improve the quality of your training data for your NLP model.</p><p>A LAMBADA model has to be trained on a dataset, after which you can use NLPAUG’s LambadaAug() function to apply sentence-level augmentation. Refer to NLPAUG’s <a href="https://github.com/makcedward/nlpaug/blob/master/example/lambada-train_model.ipynb">GitHUb repository example on LAMBADA</a>.</p><h4>3.4 Random Augmenter</h4><p>The random augmenter applies random sentence-level augmentation behaviors to the input text.</p><p>NLPAUG’s RandomSentAug()function may swap, crop or delete sentences randomly. These variations contribute to a more diversified training dataset.</p><pre>import nlpaug.augmenter.sentence as nas<br><br>text = &quot;It was a dark and stormy night. I was alone at home when I saw a lion&#39;s face followed by a scary thunderous roar at the windows.&quot;<br><br># Random Augmenter - Sentence level <br>aug = nas.RandomSentAug()<br>augmented_text = aug.augment(text)<br>print(augmented_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*Z9ppaW32ZGZho_od0f-Lcg.jpeg" /></figure><h3>Conclusion</h3><p>In conclusion, the NLPAUG Python library provides a diverse set of text augmentation techniques that can significantly improve the quality and diversity of textual data for NLP tasks. From simple operations like synonym replacement to advanced techniques such as contextual word embedding, NLPAUG provides a broad toolkit for scholars and practitioners in the discipline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48c3eebacf46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Easy Ways to Detect Outliers in Python]]></title>
            <link>https://medium.com/@marc.bolle/5-easy-ways-to-detect-outliers-in-python-b07639ba32d5?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b07639ba32d5</guid>
            <category><![CDATA[outliers]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Sun, 28 May 2023 16:36:47 GMT</pubDate>
            <atom:updated>2023-07-09T09:36:48.775Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*i9iiFUkbSuj2iuAVR4vieA.jpeg" /></figure><h3>What is an Outlier?</h3><p>An outlier is a data point that differs substantially from the rest of the data. In other words, it’s an observation that lies an abnormal distance from other values. An outlier may be due to the inherent variability of the data or it may indicate an error.</p><h4>Univariate vs. multivariate outliers</h4><ul><li><strong>A univariate outlier</strong> is a data point that consists of an extreme value on one variable.</li><li><strong>A multivariate outlier</strong> is a combination of unusual values on two variables or more. This case is harder to detect.</li></ul><h4>Why do we care about outliers?</h4><ul><li><strong>Impact on statistical analysis</strong>: outliers influence statistical measures that are sensitive to extreme values, such as the mean and standard deviation.</li><li><strong>Skewness and assumptions</strong>: outliers introduce skewness in the data distribution, breaching the normality assumptions of many statistical models such as linear regression and hypothesis testing.</li><li><strong>Machine learning performance</strong>: outliers have an impact on predictive models, which frequently attach disproportionate importance to outliers during training, resulting in decreased generalizability of the models.</li><li><strong>Data quality</strong>: outliers may indicate data quality concerns or errors caused by measurement errors, input errors, or other data gathering issues.</li></ul><p>In this post, we will explore 5 simple methods for identifying outliers with Python code examples.</p><h4><strong>Table of Contents</strong></h4><p>1. Histogram <br>2. Scatter Plot and Pairs Plot<br>3. Boxplot<br>4. Interquartile Range (IQR)<br>5. Z-score</p><h3>1. Histogram</h3><p>A histogram is a graphical depiction of a dataset’s distribution. It shows the frequency of observations in distinct intervals (or “bins”) of data. Histograms can help you discover probable outliers by studying the form and dispersion of the distribution.</p><p>A histogram’s x-axis displays the range of values in the feature, divided into bins, while the y-axis displays the frequency of observations in each bin. Outliers can be identified as solitary bins in the tails of your histogram.</p><p>Let’s plot an histogram of each feature of Scikit Learn’s California Housing dataset:</p><pre>import matplotlib.pyplot as plt<br>import pandas as pd<br>from sklearn.datasets import fetch_california_housing<br><br># Load the California Housing dataset<br>data = fetch_california_housing(as_frame=True)<br>df = data[&#39;data&#39;]<br>df[&#39;MedHouseVal&#39;] = data[&#39;target&#39;]<br><br># Create a 4x2 grid of subplots<br>fig, ax = plt.subplots(4, 2,figsize=(10, 10))<br><br># Plot histograms on each subplot<br>ax[0, 0].hist(df[&#39;MedInc&#39;], bins=&quot;auto&quot;)<br>ax[0, 1].hist(df[&#39;HouseAge&#39;], bins=&quot;auto&quot;)<br>ax[1, 0].hist(df[&#39;AveRooms&#39;], bins=&quot;auto&quot;)<br>ax[1, 1].hist(df[&#39;AveBedrms&#39;], bins=&quot;auto&quot;)<br>ax[2, 0].hist(df[&#39;Population&#39;], bins=&quot;auto&quot;)<br>ax[2, 1].hist(df[&#39;AveOccup&#39;], bins=&quot;auto&quot;)<br>ax[3, 0].hist(df[&#39;MedHouseVal&#39;], bins=&quot;auto&quot;)<br><br># Set titles for each subplot<br>ax[0, 0].set_title(&#39;MedInc&#39;)<br>ax[0, 1].set_title(&#39;HouseAge&#39;)<br>ax[1, 0].set_title(&#39;AveRooms&#39;)<br>ax[1, 1].set_title(&#39;AveBedrms&#39;)<br>ax[2, 0].set_title(&#39;Population&#39;)<br>ax[2, 1].set_title(&#39;AveOccup&#39;)<br>ax[3, 0].set_title(&#39;MedHouseVal&#39;)<br><br># Remove the empty subplot in the last row and last column<br>fig.delaxes(ax[3, 1])<br><br># Adjust the spacing between subplots<br>fig.tight_layout()<br><br># Display the plot<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bLc0zTMHs6sZTPQqfaBmYg.jpeg" /></figure><p>Outliers can be easily identified visually here. Some bins are noticeably different from the others and deviate from the expected pattern. However, it is important to analyze the data further and evaluate additional elements.</p><h3>2. Scatter Plot and Pairs Plot</h3><p>Scatter plot is another type of plot for identifying outliers in a dataset. This plot visually depicts the values of a dataset across two variables. Each data point is represented on the plot as a distinct point, generating a collection of points that helps to identify patterns and relationships.</p><p>We can quickly identify outliers by inspecting a scatter plot. Outliers are points that are unusually far out from the majority, and their distinguishing location allows for easy identification.</p><p>Let’s plot two scatter plots of the California Housing dataset as an example:</p><pre>import matplotlib.pyplot as plt<br>from sklearn.datasets import fetch_california_housing<br><br># Load the California Housing dataset<br>data = fetch_california_housing(as_frame=True)<br>df = data[&#39;data&#39;]<br><br># Create a grid with 1 row and 2 columns<br>fig, axs = plt.subplots(1, 2, figsize=(10, 5))<br><br># Plot scatter plot of house age against average rooms<br>axs[0].scatter(df[&#39;HouseAge&#39;], df[&#39;AveRooms&#39;])<br>axs[0].set_xlabel(&#39;House Age&#39;)<br>axs[0].set_ylabel(&#39;Average Rooms&#39;)<br>axs[0].set_title(&#39;House Age vs. Average Bedrooms&#39;)<br><br># Plot scatter plot of house age against average bedrooms<br>axs[1].scatter(df[&#39;HouseAge&#39;], df[&#39;AveBedrms&#39;])<br>axs[1].set_xlabel(&#39;House Age&#39;)<br>axs[1].set_ylabel(&#39;Average Bedrooms&#39;)<br>axs[1].set_title(&#39;House Age vs. Average Bedrooms&#39;)<br><br># Adjust spacing between subplots<br>plt.tight_layout()<br><br># Display the plots<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*bGlKohHrqGFNzvzPRJkBkw.png" /></figure><p>Data points that differ greatly from the main clusters are easily identified.<br>Creating scatter plots for each variable combination in a dataset, on the other hand, can be tedious and time-consuming. To circumvent this difficulty, we can use a technique called as a pairs plot.</p><p><strong>Pairs plot</strong> consists of a grid of scatter plots, where each variable in the dataset is plotted against every other variable. The distribution of each variable is commonly shown on the diagonal of the grid, while the scatter plots of the pairwise associations are shown on the off-diagonal plots.</p><p>Let’s create a pairs plot of the California Housing dataset to identify outliers:</p><pre>import seaborn as sns<br>import matplotlib.pyplot as plt<br>from sklearn.datasets import fetch_california_housing<br><br># Load the California Housing dataset<br>data = fetch_california_housing(as_frame=True)<br>df = data[&#39;data&#39;]<br>df[&#39;MedHouseVal&#39;] = data[&#39;target&#39;]<br><br># Create a pairs plot using seaborn<br>pairplot = sns.pairplot(df)<br><br># Display the plot<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qWWazpu_LiyvUW1SuolrCA.png" /></figure><p>We can easily spot the outliers in this pairs plot. The data points that are far from the central clusters can be considered as outliers. As a result, pairs plots give us a quick visual way to identify outliers in a dataset across all variables.</p><h3>3. Boxplot</h3><p>A boxplot, or box-and-whisker plot, is a graphical representation of the distribution of a dataset. It shows at the same time the median, quartiles, and range of the data. A boxplot is particularly useful for identifying outliers.</p><p>In a boxplot: <br>- The box represents the interquartile range (IQR), which is the range between the first quartile (25th percentile) and the third quartile (75th percentile) of the data. <br>- The line inside the box represents the median (50th percentile) of the data. <br>- The whiskers of the plot extend from the top and bottom of the box to the minimum and maximum values within 1.5 times the IQR from the first and third quartiles.</p><p>Any data points outside the whiskers can be considered potential outliers and are plotted as individual points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*4xvJ9VCtu96MKHI2zLw8vQ.png" /></figure><p>There are several ways to plot boxplots in Python. Let’s plot a bse scikit-learn’s California housing dataset as an example. It contains aggregated data about demography and real estate regarding each district in California:</p><pre><br>from sklearn.datasets import fetch_california_housing  <br>import pandas as pd <br><br># Load the California Housing dataset<br>data = fetch_california_housing(as_frame=True)  <br>df = data[&#39;data&#39;] <br>df[&#39;MedHouseVal&#39;] = data[&#39;target&#39;]  <br><br># Plotting box plots for each feature in the dataset<br>df.plot(kind=&quot;box&quot;, subplots=True, layout=(3,3), figsize=(10,10))  </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*E4ckD_bM7D4X07QcxZPfzg.jpeg" /></figure><p>We can see very clearly that the variables MedInc, AveRooms, AveBedrms, Population, AveOccup and MedHouseVal contain outliers.</p><p>We can even combine a histogram and a boxplot using Plotly package. This feature enhances our ability to visualize the data distribution of a feature and identify outliers. Let’s plot the variable <em>MedInc</em> of the California Housing dataset as an example:</p><pre>from sklearn.datasets import fetch_california_housing  <br>import pandas as pd  <br>import plotly.express as px  <br><br># Fetch the California housing dataset as a pandas DataFrame<br>data = fetch_california_housing(as_frame=True)<br>df = data[&#39;data&#39;]<br>df[&#39;MedHouseVal&#39;] = data[&#39;target&#39;]<br><br># Load the California Housing dataset<br>fig = px.histogram(df, x=&quot;MedInc&quot;, marginal=&quot;box&quot;, title=&quot;MedInc&quot;)<br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*pdU7pFzi3OIRCSXYe8krcQ.png" /></figure><p>Furthermore, if the features of your dataset have the same length, you can plot them in a side-by-side boxplot as follow:</p><pre># Import necessary libraries<br>from sklearn.datasets import fetch_california_housing<br>import pandas as pd  <br>import matplotlib.pyplot as plt  <br><br># Fetch the California housing dataset as a pandas DataFrame<br>data = fetch_california_housing(as_frame=True)<br>df = data[&#39;data&#39;]<br>df[&#39;MedHouseVal&#39;] = data[&#39;target&#39;]<br><br># Create a figure and axes objects<br>fig, ax = plt.subplots()<br><br># Create vertical boxplots for &#39;AveBdrms&#39; and &#39;AveRooms&#39;<br>ax.boxplot([df[&#39;AveBedrms&#39;], df[&#39;AveRooms&#39;]], labels=[&#39;AveBdrms&#39;, &#39;AveRooms&#39;])<br><br># Set the title and labels for the axes<br>ax.set_title(&#39;Boxplots of AveBedrms and AveRooms Boxplot&#39;)<br><br># Display the boxplots<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*Jxipx73ubBOP2-yYh-M1Bw.jpeg" /></figure><h3>4. Interquartile Range (IQR)</h3><p>The interquartile range (IQR) is a measure of variability in a dataset. It’s the difference between the 75th percentile (Q3) and the 25th percentile (Q1) in a sorted dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*VEUa88-chnLnZQlViI5bVg.png" /></figure><p>We can find outliers using the IQR by calculating upper and lower bounds, also known as the “Tukey’s fences”. Values outside these bounds are considered potential outliers: <br>- <strong>Lower bound</strong>: (Q1–1.5 * IQR)<br>- <strong>Upper bound</strong>: (Q3 + 1.5 * IQR)</p><p>Let’s identify the outliers of the California Housing dataset using the IQR as follow:</p><pre>from sklearn.datasets import fetch_california_housing  <br>import pandas as pd  <br><br># Fetch the California housing dataset as a pandas DataFrame<br>data = fetch_california_housing(as_frame=True)<br>df = data[&#39;data&#39;]<br>df[&#39;MedHouseVal&#39;] = data[&#39;target&#39;]<br><br># Calculate Q1 and Q3<br>Q1 = df.quantile(0.25)<br>Q3 = df.quantile(0.75)<br><br># Calculate the IQR<br>IQR = Q3 - Q1<br><br># Make a dataset with the outliers<br>IQR_outliers = df[((df &lt; (Q1 - 1.5 * IQR)) |(df &gt; (Q3 + 1.5 * IQR))).any(axis=1)]<br>IQR_outliers</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*WKDRNgalsF1LwxA96Fx7ZQ.jpeg" /></figure><p>Using the IQR method, we find 4328 outliers. We can remove them as follow:</p><pre># Remove outliers<br>df_without_outliers = df[~((df &lt; (Q1 - 1.5 * IQR)) |(df &gt; (Q3 + 1.5 * IQR))).any(axis=1)]</pre><h3>5. Z-score</h3><p>The Z-score is a statistical measure that represents the number of standard deviations a data point is away from the mean. In other words, a z-score measures the distance between the data point and the mean using standard deviations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*oeoPcLRCtE7C_G3I_993dg.png" /></figure><p>Z-score can be used to detect outliers. We determine a threshold value for the Z-score, and data points with Z-scores beyond this threshold are considered potential outliers.</p><p>It is common to use -3 and +3 as threshold. Any data point with a z-score below -3 or above +3 are treated as outliers. The reason for this common threshold is that 99.7% of the values in a standard normal distribution fall between -3 and +3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KvvqkYON2TpRaCNH9kwqSw.png" /></figure><p>The suitability of Z-score as a method for detecting outliers depends on the specific characteristics of your data: <br>- Z-score assumes that the data follow a normal distribution. If your data is not normally distributed, Z-score may not be the most appropriate method for detecting outliers. <br>- Z-score is sensitive to extreme values, which can influence the mean and standard deviation, leading to misleading outlier detection. <br>- Choosing a threshold for identifying outliers is subjective. A threshold of 3 is commonly used, but the optimal threshold may depend on your dataset and the context.</p><p>The easiest way to calculate z-scores using Python is the stats.zscore() function of the SciPy library. Let’s apply it to the California Housing dataset:</p><pre>from scipy import stats<br>import numpy as np<br><br># Calculate the z-scores<br>z_scores = stats.zscore(df)<br><br># Print the z-scores<br>z_scores</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*0JIBpLqyTtdfH_NspjMTWQ.jpeg" /></figure><pre># Convert the z-scores to absolute values<br>abs_z_scores = np.abs(z_scores)<br><br># Select data points with a z-scores above or below 3<br>filtered_entries = (abs_z_scores &lt; 3).all(axis=1)<br><br># Filter the dataset<br>filtered_df = df[filtered_entries]</pre><p>Using Z-score, we find 846 outliers in the California Housing dataset.</p><h3>Conclusion</h3><p>We have explored five methods for detecting outliers in Python. I hope that this post has given you the tools you need to recognize outliers in your datasets!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b07639ba32d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Strengths and Weaknesses of ChatGPT: Insights from The French Academy of Technologies]]></title>
            <link>https://medium.com/@marc.bolle/strengths-and-weaknesses-of-chatgpt-insights-from-the-french-academy-of-technologies-1680bffd6e1?source=rss-759e12b6c3c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/1680bffd6e1</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Marc Bolle]]></dc:creator>
            <pubDate>Tue, 23 May 2023 14:08:34 GMT</pubDate>
            <atom:updated>2023-07-09T09:37:40.888Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*elrce97X_dZGuhPbe9cPsw.jpeg" /></figure><p>The French Academy of Technologies, a learned society dedicated to harnessing technology for the benefit of humanity, provides clarity on emerging technologies to foster informed public debates.</p><p>Having already shared their perspectives on cloud computing, quantum computing, and trustworthy AI, the <a href="https://www.academie-technologies.fr/en/">Academy</a> has now delved into the realm of generative AI, with a special focus on <a href="https://openai.com/blog/chatgpt">ChatGPT</a>.</p><p>In their report titled “<a href="https://www.academie-technologies.fr/wp-content/uploads/2023/05/20230504-Avis-AT-agents-conversationnels-ChatGPT.pdf"><em>Achievements and Limitations of Artificial Language Imitation: Intelligent Conversational Agents including ChatGPT</em></a><em>,</em>” the Academy, comprising of 300 members from diverse backgrounds, presents their inaugural formal opinion on conversational AI.</p><p>The report is the result of a collective opinion formed through a democratic voting process during a plenary assembly. Its purpose, as highlighted by Gérard Roucairol, the President of the Academy, is to establish reliable reference points on a subject prone to buzz and conflicting opinions.</p><p><strong>In this article, I will provide a summary of the key findings from this report, giving you an accessible overview of the Academy’s insights into conversational AI.</strong></p><h4>Table of Contents</h4><p><strong>SWOT Analysis of ChatGPT</strong><br>◦ ChatGPT Strenghts <br>◦ ChatGPT Weaknesses <br>◦ ChatGPT Opportunities <br>◦ ChatGPT Threats</p><h3>SWOT Analysis of ChatGPT</h3><p>The Academy of Technologies has provided a SWOT analysis of ChatGPT.</p><p>This strategic planning tool assesses an emerging technology’s strengths and weaknesses; and opportunities and threats presented by the market, competitors, and external factors.</p><h4>ChatGPT Strengths</h4><ul><li>Traditional search engines provide webpages where users have to find the answer to their question; whereas ChatGPT offers a <strong>direct, synthesized answer</strong>.</li><li>Users can <strong>customize the length and complexity of the text generated</strong> by ChatGPT based on their preferences.</li><li>ChatGPT <strong>considers previous sentences in a conversation</strong>, taking them into account when generating new responses.</li><li>ChatGPT has <strong>some control over its responses</strong> and can refuse requests related to weapon manufacturing or offensive language for example.</li></ul><h4>ChatGPT Weaknesses</h4><ul><li>ChatGPT generates <strong>answers based on statistical patterns</strong> rather than relying on truth, logic, or calculations. It aims to provide the most statistically plausible responses to user queries.</li><li>ChatGPT is capable of generating “<strong>hallucinations</strong>,” which are confident but entirely fabricated responses. These hallucinations can closely resemble real data, posing a challenge for human interlocutors to differentiate between truth and fabrication.</li><li><strong>Inconsistencies and indeterminism</strong> may occur, especially during dialogues.</li><li><strong>Biases and offensive language </strong>can be be obtained in responses through subtle wording of requests.</li><li>The <strong>responsibility for determining undesirable interactions lies with the language model’s owning company</strong>, which also influences the political, economic, or philosophical “tone” of ChatGPT’s responses.</li><li>ChatGPT’s <strong>training corpus does not adhere to the European General Data Protection Regulation</strong> (GDPR), resulting in temporary limitations imposed by the Italian Data Protection Authority.</li><li>The <strong>compatibility of the generated texts with current legislation</strong>, particularly in terms of copyright and intellectual property, remains uncertain.</li></ul><h4>ChatGPT Opportunities</h4><ul><li>We can expect an <strong>abundance of innovations</strong> based on large language models such as ChatGPT.</li><li>ChatGPT significantly <strong>accelerates text production</strong> in fields such as marketing, sales, and customer relations.</li><li>ChatGPT operates in an interactive loop of question, answer, and correction. As ChatGPT’s responses are often comprehensive and structured, it serves as a <strong>useful tool for exploring a topic</strong> by providing insights that users can choose to delve deeper into with additional questions.</li><li>ChatGPT can also generate non-natural language texts, such as programming languages, structured documents (spreadsheets, presentations, data visualization tables), or industrial nomenclatures. These tools will enable a <strong>significant boost in productivity</strong>.</li><li>ChatGPT can be <strong>enhanced with reasoning capabilities</strong> to improve coherence, credibility, and move towards a certain notion of common sense. However, the hybridization of statistical methods with logical or mathematical proof methods remains a challenging problem.</li></ul><h4><strong>ChatGPT Threats</strong></h4><ul><li>An important concern arises from the user-friendly nature of ChatGPT, as it carries the risk of transforming into an <strong>oracle that delivers default responses to any query</strong>.</li><li>ChatGPT could <strong>create arbitrary beliefs or push people toward inappropriate actions</strong>. This threat is similar to that of fake news.</li><li>The <strong>roles of knowledge workers are expected to undergo changes</strong>, with generative AI models playing a more prominent role.</li><li>ChatGPT <strong>disrupts education</strong>. The current methods of assessing students should be deeply reconsidered to measure a student’s contribution compared to a response provided by a language model.</li><li>Large language models are <strong>excessively energy-intensive</strong> due to their data-intensive development, the construction of the necessary infrastructure for their operation, and their usage by users.</li></ul><h3>Conclusion</h3><p>In conclusion, the SWOT analysis of ChatGPT by the French Academy of Technologies reveals its strengths as a powerful tool with remarkable capabilities, allowing for enhanced productivity and innovation in various fields. However, it also highlights weaknesses such as potential biases and limitations in handling complex queries. While ChatGPT presents exciting opportunities for assisting in tasks like content generation, there are also significant threats to address, including the risk of misinformation and potential ethical concerns.</p><p>As ChatGPT continues to evolve, it is crucial to harness its strengths, mitigate weaknesses, capitalize on opportunities, and proactively address threats to ensure responsible and beneficial utilization of this transformative technology.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1680bffd6e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>