<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jozsef Meszaros on Medium]]></title>
        <description><![CDATA[Stories by Jozsef Meszaros on Medium]]></description>
        <link>https://medium.com/@neuraljojo?source=rss-71417ba6ebf5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*43TiNYq5yPk__PM6nbFUeg.jpeg</url>
            <title>Stories by Jozsef Meszaros on Medium</title>
            <link>https://medium.com/@neuraljojo?source=rss-71417ba6ebf5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:23:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@neuraljojo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Analyze Scientific Publications with E-utilities and Python]]></title>
            <link>https://medium.com/data-science/analyze-scientific-publications-with-e-utilities-and-python-56f76de22959?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/56f76de22959</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[text-mining]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Fri, 19 May 2023 04:13:53 GMT</pubDate>
            <atom:updated>2023-05-19T13:12:48.349Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to gather data about scientific literature and discover trends</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dyLU0htgzeBYD6L3" /><figcaption>Photo by <a href="https://unsplash.com/@alexblock?utm_source=medium&amp;utm_medium=referral">Alex Block</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Staying on top of scientific research trends is essential for any scientist, science writer, or aspiring start-up founder. When it comes to searching biomedical science literature, most people will turn to Google Scholar, PubMed, or their favorite reference manager. As you might expect, these public-facing search tools offer ease of use but sacrifice efficiency, control, and scalability. Thus, they are generally not readily useful when it comes to data science. Instead, you’ll want to use the E-utilities offered by the National Center of Biotechnology Information (NCBI).[1] The scientific articles available from NCBI are stored within the PubMed database, which primarily covers life science research, but also a few journals related to chemistry and physics.[2]</p><blockquote>For data science applications, web-based search engines and popular reference managers won’t cut it</blockquote><p>The information on how to efficiently use NCBI for data science is strewn about various governmental and academic websites. Many of the resources were uploaded in the 1990’s, focused on searching for genetic sequences rather than primary literature publications, and accompanied by code examples written only in Perl. There was once a course on using NCBI’s E-Utilities that was hosted at NIH during the early 2000’s, for which Powerpoint “slidesets” along with obligatory clip-art illustrations are available <a href="https://ftp.ncbi.nlm.nih.gov/pub/PowerTools/eutils/">here</a>. In this article, I will convey the lessons I have learned compiling information from these various sources and regular trial-and-error.</p><blockquote><strong>Note:</strong> There are many types of data searchable from within NCBI, including genetic sequences, phylogenetic trees, 3-D structures, and other information. This article will be focused exclusively on searching <strong>primary</strong> <strong>literature</strong>.</blockquote><h4>What kind of data science questions can be answered using NCBI?</h4><p>There are many kinds of science questions answerable using NCBI. You could, for instance, create a cluster-map of author-provided keywords for articles to examine publishing trends over time, as I demonstrate at the end of this article. Additionally, you could build LLMs and NLP models using abstracts and text from publications to assist with making connections between literature. Here are the three steps I will cover in the article:</p><p>(1) querying the databases,</p><p>(2) returning results for multiple publications, and</p><p>(3) retrieving similar articles and full text versions of articles.</p><p>At the end, I will provide a full documented code example to see how to carry out these three steps for a relatively large corpus (~10,000 articles) and an accompanying data visualization.</p><h3>Querying NCBI databases</h3><p>To query an NCBI database effectively, you’ll want to learn about certain E-utilities, define your search fields, and choose your search parameters — which control the way results are returned to your browser or in our case, we’ll use Python to query the databases.</p><h4>Four most useful E-utilities</h4><p>There are <strong>nine E-utilities</strong> available from NCBI, and they are all implemented as server-side fast CGI programs. This means you will access them by creating URLs which end in <strong>.cgi </strong>and specify query parameters after a question-mark, with parameters separated by ampersands. All of them, except for <strong>EFetch</strong>, will give you <strong>either XML or JSON </strong>outputs.</p><ul><li><strong>ESearch </strong>generates a list of ID numbers that meet your search query</li></ul><p>The following E-Utilities can be used with one or more ID numbers:</p><ul><li><strong>ESummary </strong>journal, author list, grants, dates, references, publication type</li><li><strong>EFetch </strong>**XML ONLY** all of what <strong>ESummary</strong> provides as well as an abstract, list of grants used in the research, institutions of authors, and MeSH keywords</li><li><strong>ELink </strong>provides a list of links to related citations using computed similarity score <strong><em>as well as providing a link to the published item </em></strong>[your gateway to the full-text of the article]</li></ul><p>The NCBI hosts 38 databases across their servers, related to a variety of data that goes beyond literature citations. To get a complete list of current databases, you can use <strong>EInfo </strong>without<strong> </strong>search terms:</p><p><a href="https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi">https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi</a></p><p>Each database will vary in how it can be accessed and the information it returns. For our purposes, we’ll focus on the <em>pubmed</em> and <em>pmc</em> databases because these are where scientific literature are searched and retrieved.</p><p>The two most important things to learn about searching NCBI are <strong>search fields </strong>and <strong>outputs</strong>. The <strong>search fields are numer</strong>ous and will depend on the database. The outputs are more straightforward and learning how to use the outputs is essential, especially for doing large searches.</p><h4>Search fields</h4><p>You won’t be able to truly harness the potential of E-utilities without knowing about the available search fields. You can find a full list of these search fields on the <a href="https://www.nlm.nih.gov/bsd/mms/medlineelements.html">NLM website</a> along with a description of each, but for the <strong><em>most accurate list of search terms specific to a database</em></strong>, you’ll want to parse your own XML list using this link:</p><p><a href="https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi?db=pubmed">https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi?db=pubmed</a></p><p>with the <strong>db</strong> flag set to the database (we will use <em>pubmed</em> for this article, but literature is also available through <em>pmc</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Dq5YTg-A2JtA7T7wzCR1A.png" /><figcaption>A list of search fields for querying PubMed MEDLINE records. (Source: <a href="https://www.nlm.nih.gov/bsd/mms/medlineelements.html">https://www.nlm.nih.gov/bsd/mms/medlineelements.html</a>)</figcaption></figure><p>One especially useful search field is the Medline Subject Headings (<strong>MeSH</strong>).[3] Indexers, who are experts in the field, maintain the PubMed database and use MeSH terms to reflect the subject matter of journal articles as they are published. Each indexed publication is typically described by 10 to 12 carefully selected MeSH terms by the indexers. If no search terms are specified, then queries will be executed against every search term available in the database queried.[4]</p><h4>Query parameters</h4><p>Each of the E-utilities accepts multiple query parameters through the URL line which you can use to control the type and amount of output returned from a query. This is where you can set the number of search results retrieved or the dates searched. Here are a list of the more important parameters:</p><p><strong>Database parameter:</strong></p><ul><li>db should be set to the database you are interested in searching — <em>pubmed</em> or <em>pmc</em> for scientific literature</li></ul><p><strong>Date parameters: </strong>You can get more control over the date by using search fields, [pdat] for example for the publication date, but date parameters provide a more convenient way to constrain results.</p><ul><li>reldate the days to be searched relative to the current date, set reldate=1 for the most recent day</li><li>mindate and maxdate specify date according to the format YYYY/MM/DD, YYYY, or YYYY/MM <strong>(a query must contain both </strong>mindate<strong> and </strong>maxdate<strong> parameters)</strong></li><li>datetype sets the type of date when you query by date — options are ‘mdat’ (modification date), ‘pdat’ (publication date) and ‘edat’ (Entrez date)</li></ul><p><strong>Retrieval parameters:</strong></p><ul><li>rettype the type of information to return (for literature searches, use the default setting)</li><li>retmode format of the output (XML is the default, though all E-utilities except fetch do support JSON)</li><li>retmax is the maximum number of records to return —<strong><em> the default is 20 and the maximum value is 10,000</em></strong> (ten thousand)</li><li>retstart given a list of hits for a query, retstart specifies the index (useful for when your search exceeds the ten thousand maximum)</li><li>cmd this is only relevant to <strong>ELink</strong> and is used to specify whether to return IDs of similar articles or URLs to full-texts</li></ul><h3>Use Python to execute queries and store results</h3><p>Once we know about the E-Utilities, have chosen our search fields, and decided upon query parameters, we’re ready to execute queries and store the results — even for multiple pages.</p><p>While you don’t specifically need to use Python to use the E-utilities, it does make it much easier to parse, store, and analyze the results of your queries. Here’s how to get started on your data science project.</p><p>Let’s say you want to search MeSH terms for the term “myoglobin” between 2022 and 2023. You’ll set your <strong>retmax</strong> to 50 for now, but remember <strong><em>the max is 10,000 and you can query at a rate of 3/s.</em></strong></p><pre>import urllib.request<br>search_url = f&#39;http://eutils.ncbi.nlm.nih.gov/entrez//eutils/esearch.fcgi/&#39; + \<br>              f&#39;?db=pubmed&#39; + \<br>              f&#39;&amp;term=myoglobin[mesh]&#39; + \<br>              f&#39;&amp;mindate=2022&#39; + \<br>              f&#39;&amp;maxdate=2023&#39; + \<br>              f&#39;&amp;retmode=json&#39; + \<br>              f&#39;&amp;retmax=50&#39;<br><br>link_list = urllib.request.urlopen(search_url).read().decode(&#39;utf-8&#39;)<br>link_list</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3xx-G4sFW0hG3quDYX4BmQ.png" /><figcaption>The output of the <strong>esearch </strong>query from above.</figcaption></figure><p>The results are returned as a list of IDs, which can be used in a subsequent search within the database you queried. Note that <strong>“count”</strong> shows there are 154 results for this query, which you could use if you wanted to get a total count of publications for a certain set of search terms. If you wanted to return IDs for all the publication, you’d set the <strong>retmax</strong> parameter to the count, or 154. In general, I set this to a very high number so I can retrieve all of the results and store them.</p><p><strong>Boolean searching</strong> is easy with PubMed and it only requires adding +OR+, +NOT+, or +AND+ to the URL between search terms. For example:</p><p><a href="http://eutils.ncbi.nlm.nih.gov/entrez//eutils/esearch.fcgi/?db=pubmed&amp;term=CEO[cois]+OR+CTO[cois]+OR+CSO[cois]&amp;mindate=2022&amp;maxdate=2023&amp;retmax=10000">http://eutils.ncbi.nlm.nih.gov/entrez//eutils/esearch.fcgi/?db=pubmed&amp;term=</a><a href="http://eutils.ncbi.nlm.nih.gov/entrez//eutils/esearch.fcgi/?db=pubmed&amp;term=CEO[cois]+OR+CTO[cois]+OR+CSO[cois]&amp;mindate=2022&amp;maxdate=2023&amp;retmax=10000"><strong>CEO[cois]+OR+CTO[cois]+OR+CSO[cois]</strong></a><a href="http://eutils.ncbi.nlm.nih.gov/entrez//eutils/esearch.fcgi/?db=pubmed&amp;term=CEO[cois]+OR+CTO[cois]+OR+CSO[cois]&amp;mindate=2022&amp;maxdate=2023&amp;retmax=10000">&amp;mindate=2022&amp;maxdate=2023&amp;retmax=10000</a></p><p>These search strings can constructed using Python. In the following steps, we’ll parse the results using Python’s <em>json package</em> to get the IDs for each of the publications returned. The IDs can then be used to create a string — this string of IDs can be used by the other E-Utilities to return information about the publications.</p><h4>Use ESummary to return information about publications</h4><p>The purpose of <strong>ESummary</strong> is to return data that you might expect to see in a paper’s citation (date of publication, page numbers, authors, etc). Once you have a result in the form of a list of IDs from <strong>ESearch </strong>(in the step above), you can join this list into a long URL.</p><blockquote>The limit for a URL is 2048 characters, and each publication’s ID is 8 characters long, so to be safe, you should split your list of links up into batches of 250 if you have a list larger than 250 IDs. See my notebook at the bottom of the article for an example.</blockquote><p>The results from an <strong>ESummary </strong>are returned in JSON format and can include a link to the paper’s full-text:</p><pre>import json<br>result = json.loads( link_list )<br>id_list = &#39;,&#39;.join( result[&#39;esearchresult&#39;][&#39;idlist&#39;] )<br><br>summary_url = f&#39;http://eutils.ncbi.nlm.nih.gov/entrez//eutils/esummary.fcgi?db=pubmed&amp;id={id_list}&amp;retmode=json&#39;<br><br>summary_list = urllib.request.urlopen(summary_url).read().decode(&#39;utf-8&#39;)</pre><p>We can again use json to parse <em>summary_list</em>. When using the json package, you can browse the fields of each individual article by using summary[‘result’][id as string], as in the example below:</p><pre>summary = json.loads( summary_list )<br>summary[&#39;result&#39;][&#39;37047528&#39;]</pre><p>We can create a dataframe to capture the ID for each article along with the name of the journal, the publication date, title of the article, a URL for retrieving the full text, as well as the first and last author.</p><pre>uid = [ x for x in summary[&#39;result&#39;] if x != &#39;uids&#39; ]<br>journals = [ summary[&#39;result&#39;][x][&#39;fulljournalname&#39;] for x in summary[&#39;result&#39;] if x != &#39;uids&#39; ]<br>titles = [ summary[&#39;result&#39;][x][&#39;title&#39;] for x in summary[&#39;result&#39;] if x != &#39;uids&#39; ]<br>first_authors = [ summary[&#39;result&#39;][x][&#39;sortfirstauthor&#39;] for x in summary[&#39;result&#39;] if x != &#39;uids&#39; ]<br>last_authors = [ summary[&#39;result&#39;][x][&#39;lastauthor&#39;] for x in summary[&#39;result&#39;] if x != &#39;uids&#39; ]<br>links = [ summary[&#39;result&#39;][x][&#39;elocationid&#39;] for x in summary[&#39;result&#39;] if x != &#39;uids&#39; ]<br>pubdates = [ summary[&#39;result&#39;][x][&#39;pubdate&#39;] for x in summary[&#39;result&#39;] if x != &#39;uids&#39; ]<br><br>links = [ re.sub(&#39;doi:\s&#39;,&#39;http://dx.doi.org/&#39;,x) for x in links ]<br>results_df = pd.DataFrame( {&#39;ID&#39;:uid,&#39;Journal&#39;:journals,&#39;PublicationDate&#39;:pubdates,&#39;Title&#39;:titles,&#39;URL&#39;:links,&#39;FirstAuthor&#39;:first_authors,&#39;LastAuthor&#39;:last_authors} )</pre><p>Below is a list of all the different fields that <strong>ESummary</strong> returns so you can make your own database:</p><pre>&#39;<strong>uid</strong>&#39;,&#39;<strong>pubdate</strong>&#39;,&#39;<strong>epubdate</strong>&#39;,&#39;<strong>source</strong>&#39;,&#39;<strong>authors</strong>&#39;,&#39;<strong>lastauthor</strong>&#39;,&#39;<strong>title</strong>&#39;,<br>&#39;<strong>sorttitle</strong>&#39;,&#39;<strong>volume</strong>&#39;,&#39;<strong>issue</strong>&#39;,&#39;<strong>pages</strong>&#39;,&#39;<strong>lang</strong>&#39;,&#39;<strong>nlmuniqueid</strong>&#39;,&#39;<strong>issn</strong>&#39;,<br>&#39;<strong>essn</strong>&#39;,&#39;<strong>pubtype</strong>&#39;,&#39;<strong>recordstatus</strong>&#39;,&#39;<strong>pubstatus</strong>&#39;,&#39;<strong>articleids</strong>&#39;,&#39;<strong>history</strong>&#39;,<br>&#39;<strong>references</strong>&#39;,&#39;<strong>attributes</strong>&#39;,&#39;<strong>pmcrefcount</strong>&#39;,&#39;<strong>fulljournalname</strong>&#39;,&#39;<strong>elocationid</strong>&#39;,<br>&#39;<strong>doctype</strong>&#39;,&#39;<strong>srccontriblist</strong>&#39;,&#39;<strong>booktitle</strong>&#39;,&#39;<strong>medium</strong>&#39;,&#39;<strong>edition</strong>&#39;,<br>&#39;<strong>publisherlocation</strong>&#39;,&#39;<strong>publishername</strong>&#39;,&#39;<strong>srcdate</strong>&#39;,&#39;<strong>reportnumber</strong>&#39;,<br>&#39;<strong>availablefromurl</strong>&#39;,&#39;<strong>locationlabel</strong>&#39;,&#39;<strong>doccontriblist</strong>&#39;,&#39;<strong>docdate</strong>&#39;,<br>&#39;<strong>bookname</strong>&#39;,&#39;<strong>chapter</strong>&#39;,&#39;<strong>sortpubdate</strong>&#39;,&#39;<strong>sortfirstauthor</strong>&#39;,&#39;<strong>vernaculartitle</strong>&#39;</pre><h4>Use EFetch when you want abstracts, keywords, and other details (XML output only)</h4><p>We can use <strong>EFetch</strong> to return similar fields as <strong>ESummary</strong>, with the caveat that the <strong><em>result is returned in XML only</em></strong>. There are several interesting additional fields in <strong>EFetch </strong>which include: the abstract, author-selected keywords, the Medline Subheadings (MeSH terms), grants that sponsored the research, conflict of interest statements, a list of chemicals used in the research, and a complete list of all the references cited by the paper. Here’s how you would use BeautifulSoup<strong> </strong>to obtain some of these items:</p><pre>from bs4 import BeautifulSoup<br>import lxml<br>import pandas as pd<br><br>abstract_url = f&#39;http://eutils.ncbi.nlm.nih.gov/entrez//eutils/efetch.fcgi?db=pubmed&amp;id={id_list}&#39;<br>abstract_ = urllib.request.urlopen(abstract_url).read().decode(&#39;utf-8&#39;)<br>abstract_bs = BeautifulSoup(abstract_,features=&quot;xml&quot;)<br><br>articles_iterable = abstract_bs.find_all(&#39;PubmedArticle&#39;)<br><br># Abstracts<br>abstract_texts = [ x.find(&#39;AbstractText&#39;).text for x in articles_iterable ]<br><br># Conflict of Interest statements<br>coi_texts = [ x.find(&#39;CoiStatement&#39;).text if x.find(&#39;CoiStatement&#39;) is not None else &#39;&#39; for x in articles_iterable ]<br><br># MeSH terms<br>meshheadings_all = list()<br>for article in articles_iterable:<br>  result = article.find(&#39;MeshHeadingList&#39;).find_all(&#39;MeshHeading&#39;)<br>  meshheadings_all.append( [ x.text for x in result ] )<br><br># ReferenceList<br>references_all = list()<br>for article in articles_:<br>  if article.find(&#39;ReferenceList&#39;) is not None:<br>    result = article.find(&#39;ReferenceList&#39;).find_all(&#39;Citation&#39;)<br>    references_all.append( [ x.text for x in result ] )<br>  else:<br>    references_all.append( [] )<br><br>results_table = pd.DataFrame( {&#39;COI&#39;:coi_texts, &#39;Abstract&#39;:abstract_texts, &#39;MeSH_Terms&#39;:meshheadings_all, &#39;References&#39;:references_all} )</pre><p>Now we can use this table to search abstracts, conflict of interest statements, or make visuals that connect different fields of research using MeSH headings and reference lists. There are of course many other tags that you could explore, returned by <strong>EFetch</strong>, here’s how you can see them all using BeautifulSoup:</p><pre>efetch_url = f&#39;http://eutils.ncbi.nlm.nih.gov/entrez//eutils/efetch.fcgi?db=pubmed&amp;id={id_list}&#39;<br>efetch_result = urllib.request.urlopen( efetch_url ).read().decode(&#39;utf-8&#39;)<br>efetch_bs = BeautifulSoup(efetch_result,features=&quot;xml&quot;)<br><br>tags = efetch_bs.find_all()<br><br>for tag in tags:<br>  print(tag)</pre><h4><strong>Using ELink to retrieve similar publications, and full-text links</strong></h4><p>You may want to find articles similar to the ones returned by your search query. These articles are grouped according to a similarity score using a probabilistic topic-based model.[5] To retrieve the similarity scores for a given ID, you must pass <strong>cmd=neighbor_score </strong>in your call to <strong>ELink</strong>. Here’s an example for one article:</p><pre>import urllib.request<br>import json<br><br>id_ = &#39;37055458&#39;<br>elink_url = f&#39;http://eutils.ncbi.nlm.nih.gov/entrez//eutils/elink.fcgi?db=pubmed&amp;id={id_}&amp;retmode=json&amp;cmd=neighbor_score&#39;<br>elinks = urllib.request.urlopen(elink_url).read().decode(&#39;utf-8&#39;)<br><br>elinks_json = json.loads( elinks )<br><br>ids_=[];score_=[];<br>all_links = elinks_json[&#39;linksets&#39;][0][&#39;linksetdbs&#39;][0][&#39;links&#39;]<br>for link in all_links:<br>  [ (ids_.append( link[&#39;id&#39;] ),score_.append( link[&#39;score&#39;] )) for id,s in link.items() ]<br><br>pd.DataFrame( {&#39;id&#39;:ids_,&#39;score&#39;:score_} ).drop_duplicates([&#39;id&#39;,&#39;score&#39;])</pre><p>The other function of <strong>ELink</strong> is to provide full-text links to an article based on its ID, which can be returned if you pass <strong>cmd=prlinks</strong> to <strong>ELink</strong> instead.</p><blockquote>If you wish to access only those full-text links that are free to the public, you will want to use links that contain “pmc” (PubMed Central). Accessing articles behind a paywall may require subscription through a University—<strong>before downloading a large corpus of full-text articles through a paywall, you should consult with your organization’s librarians.</strong></blockquote><p>Here is a code snippet of how you could retrieve the links for a publication:</p><pre>id_ = &#39;37055458&#39;<br>elink_url = f&#39;http://eutils.ncbi.nlm.nih.gov/entrez//eutils/elink.fcgi?db=pubmed&amp;id={id_}&amp;retmode=json&amp;cmd=prlinks&#39;<br>elinks = urllib.request.urlopen(elink_url).read().decode(&#39;utf-8&#39;)<br><br>elinks_json = json.loads( elinks )<br><br>[ x[&#39;url&#39;][&#39;value&#39;] for x in elinks_json[&#39;linksets&#39;][0][&#39;idurllist&#39;][0][&#39;objurls&#39;] ]</pre><p>You can also retrieve links for multiple publications in one call to <strong>ELink</strong>, as I show below:</p><pre>id_list = &#39;37055458,574140&#39;<br>elink_url = f&#39;http://eutils.ncbi.nlm.nih.gov/entrez//eutils/elink.fcgi?db=pubmed&amp;id={id_list}&amp;retmode=json&amp;cmd=prlinks&#39;<br>elinks = urllib.request.urlopen(elink_url).read().decode(&#39;utf-8&#39;)<br><br>elinks_json = json.loads( elinks )<br><br>elinks_json<br>urls_ = elinks_json[&#39;linksets&#39;][0][&#39;idurllist&#39;]<br>for url_ in urls_:<br>  [ print( url_[&#39;id&#39;], x[&#39;url&#39;][&#39;value&#39;] ) for x in url_[&#39;objurls&#39;] ]</pre><h3>Example data visualization: Scientific publications from C-suite authors</h3><p>Occasionally, a scientific publication will be authored by someone who is a CEO, CSO, or CTO of a company. With PubMed, we have the ability to analyze the latest life science industry trends. Conflict of interest statements, which were introduced as a search term in PubMed during 2017,[6] give a lens into which author-provided keywords are appearing in publications where a C-suite executive is disclosed as an author. In other words, the keywords chosen by the authors to describe their finding. To carry out this function, simply include <strong>CEO[cois]+OR+CSO[cois]+OR+CTO[cois] </strong>as<strong> </strong>search term in your URL, retrieve all of the results returned, and extract the keywords from the resulting XML output for each publication. Each publication contains between 4–8 keywords. Once the corpus is generated, you can quantify keyword frequency per year within the corpus as <strong><em>the number of publications in a year specifying a keyword, divided by the number of publications for that year.</em></strong></p><p>For example, if 10 publications list the keyword “cancer” and there are 1000 publications that year, the keyword frequency would be 0.001. Using the seaborn clustermap module with the keyword frequencies you can generate a visualization where darker bands indicate a larger value of keyword frequency/year (I have dropped COVID-19 and SARS-COV-2 from the visualization as they were both represented at frequencies far greater 0.05, predictably). Each year, approximately 1000–1500 papers were returned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*lKvH6uJ6hF1nOsf5HvnZqg.png" /><figcaption>Clustermap of author-specified keyword frequencies for publications with a C-suite author listed, generated by the author using Seaborn’s clustermap module.</figcaption></figure><p>From this visualization, several insights about the corpus of publications with C-suite authors listed becomes clear. First, one of the most distinct clusters (at the bottom) contains keywords that have been strongly represented in the corpus for the past five years: cancer, machine learning, biomarkers, artificial intelligence — just to name a few. Clearly, industry is heavily active and publishing in these areas. A second cluster, near the middle of the figure, shows keywords that disappeared from the corpus after 2018, including physical activity, public health, children, mass spectrometry, and mhealth (or mobile health). It’s not to say that these areas are not being developed in industry, just that the publication activity has slowed. Looking at the bottom right of the figure, you can extract terms which have appeared more recently in the corpus, including liquid biopsy and precision medicine — which are indeed two very “hot” areas of medicine at the moment. By examining the publications further, you could extract the names of the companies and other information of interest. Below is the code I wrote to generate this visual:</p><pre>import pandas as pd<br>import time<br>from bs4 import BeautifulSoup<br>import seaborn as sns<br>from matplotlib import pyplot as plt<br>import itertools<br>from collections import Counter<br>from numpy import array_split<br>from urllib.request import urlopen<br><br>class Searcher:<br>    # Any instance of searcher will search for the terms and return the number of results on a per year basis #<br>    def __init__(self, start_, end_, term_, **kwargs):<br>        self.raw_ = input<br>        self.name_ = &#39;searcher&#39;<br>        self.description_ = &#39;searcher&#39;<br>        self.duration_ = end_ - start_<br>        self.start_ = start_<br>        self.end_ = end_<br>        self.term_ = term_<br>        self.search_results = list()<br>        self.count_by_year = list()<br>        self.options = list()<br><br>        # Parse keyword arguments<br><br>        if &#39;count&#39; in kwargs and kwargs[&#39;count&#39;] == 1:<br>            self.options = &#39;rettype=count&#39;<br><br>        if &#39;retmax&#39; in kwargs:<br>            self.options = f&#39;retmax={kwargs[&quot;retmax&quot;]}&#39;<br><br>        if &#39;run&#39; in kwargs and kwargs[&#39;run&#39;] == 1:<br>            self.do_search()<br>            self.parse_results()<br><br>    def do_search(self):<br>        datestr_ = [self.start_ + x for x in range(self.duration_)]<br>        options = &quot;&quot;.join(self.options)<br>        for year in datestr_:<br>            this_url = f&#39;http://eutils.ncbi.nlm.nih.gov/entrez//eutils/esearch.fcgi/&#39; + \<br>                       f&#39;?db=pubmed&amp;term={self.term_}&#39; + \<br>                       f&#39;&amp;mindate={year}&amp;maxdate={year + 1}&amp;{options}&#39;<br>            print(this_url)<br>            self.search_results.append(<br>                urlopen(this_url).read().decode(&#39;utf-8&#39;))<br>            time.sleep(.33)<br><br>    def parse_results(self):<br>        for result in self.search_results:<br>            xml_ = BeautifulSoup(result, features=&quot;xml&quot;)<br>            self.count_by_year.append(xml_.find(&#39;Count&#39;).text)<br>            self.ids = [id.text for id in xml_.find_all(&#39;Id&#39;)]<br><br>    def __repr__(self):<br>        return repr(f&#39;Search PubMed from {self.start_} to {self.end_} with search terms {self.term_}&#39;)<br><br>    def __str__(self):<br>        return self.description<br><br># Create a list which will contain searchers, that retrieve results for each of the search queries<br>searchers = list()<br>searchers.append(Searcher(2022, 2023, &#39;CEO[cois]+OR+CTO[cois]+OR+CSO[cois]&#39;, run=1, retmax=10000))<br>searchers.append(Searcher(2021, 2022, &#39;CEO[cois]+OR+CTO[cois]+OR+CSO[cois]&#39;, run=1, retmax=10000))<br>searchers.append(Searcher(2020, 2021, &#39;CEO[cois]+OR+CTO[cois]+OR+CSO[cois]&#39;, run=1, retmax=10000))<br>searchers.append(Searcher(2019, 2020, &#39;CEO[cois]+OR+CTO[cois]+OR+CSO[cois]&#39;, run=1, retmax=10000))<br>searchers.append(Searcher(2018, 2019, &#39;CEO[cois]+OR+CTO[cois]+OR+CSO[cois]&#39;, run=1, retmax=10000))<br><br># Create a dictionary to store keywords for all articles from a particular year<br>keywords_dict = dict()<br><br># Each searcher obtained results for a particular start and end year<br># Iterate over searchers<br>for this_search in searchers:<br><br>    # Split the results from one search into batches for URL formatting<br>    chunk_size = 200<br>    batches = array_split(this_search.ids, len(this_search.ids) // chunk_size + 1)<br><br>    # Create a dict key for this searcher object based on the years of coverage<br>    this_dict_key = f&#39;{this_search.start_}to{this_search.end_}&#39;<br><br>    # Each value in the dictionary will be a list that gets appended with keywords for each article<br>    keywords_all = list()<br><br>    for this_batch in batches:<br>        ids_ = &#39;,&#39;.join(this_batch)<br><br>        # Pull down the website containing XML for all the results in a batch<br>        abstract_url = f&#39;http://eutils.ncbi.nlm.nih.gov/entrez//eutils/efetch.fcgi?db=pubmed&amp;id={ids_}&#39;<br><br>        abstract_ = urlopen(abstract_url).read().decode(&#39;utf-8&#39;)<br>        abstract_bs = BeautifulSoup(abstract_, features=&quot;xml&quot;)<br>        articles_iterable = abstract_bs.find_all(&#39;PubmedArticle&#39;)<br><br>        # Iterate over all of the articles from the website<br>        for article in articles_iterable:<br>            result = article.find_all(&#39;Keyword&#39;)<br>            if result is not None:<br>                keywords_all.append([x.text for x in result])<br>            else:<br>                keywords_all.append([])<br><br>        # Take a break between batches!<br>        time.sleep(1)<br><br>    # Once all the keywords are assembled for a searcher, add them to the dictionary<br>    keywords_dict[this_dict_key] = keywords_all<br><br>    # Print the key once it&#39;s been dumped to the pickle<br>    print(this_dict_key)<br><br># Limit to words that appeared approx five times or more in any given year<br><br>mapping_ = {&#39;2018to2019&#39;:2018,&#39;2019to2020&#39;:2019,&#39;2020to2021&#39;:2020,&#39;2021to2022&#39;:2021,&#39;2022to2023&#39;:2022}<br>global_word_list = list()<br><br>for key_,value_ in keywords_dict.items():<br>  Ntitles = len( value_ )<br>  flattened_list = list( itertools.chain(*value_) )<br><br>  flattened_list = [ x.lower() for x in flattened_list ]<br>  counter_ = Counter( flattened_list )<br>  words_this_year = [ ( item , frequency/Ntitles , mapping_[key_] ) for item, frequency in counter_.items() if frequency/Ntitles &gt;= .005 ]<br>  global_word_list.extend(words_this_year)<br><br># Plot results as clustermap<br><br>global_word_df = pd.DataFrame(global_word_list)<br>global_word_df.columns = [&#39;word&#39;, &#39;frequency&#39;, &#39;year&#39;]<br>pivot_df = global_word_df.loc[:, [&#39;word&#39;, &#39;year&#39;, &#39;frequency&#39;]].pivot(index=&quot;word&quot;, columns=&quot;year&quot;,<br>                                                                    values=&quot;frequency&quot;).fillna(0)<br><br>pivot_df.drop(&#39;covid-19&#39;, axis=0, inplace=True)<br>pivot_df.drop(&#39;sars-cov-2&#39;, axis=0, inplace=True)<br><br>sns.set(font_scale=0.7)<br>plt.figure(figsize=(22, 2))<br>res = sns.clustermap(pivot_df, col_cluster=False, yticklabels=True, cbar=True)</pre><h3>Conclusion</h3><p>After reading this article, you should be ready to go from crafting highly tailored search queries of the scientific literature all the way to generating data visualizations for closer scrutiny. While there are other more complex ways to access and store articles using additional features of the various E-utilities, I have tried to present the most straightforward set of operations that should apply to most use cases for a data scientist interested in scientific publishing trends. By familiarizing yourself with the E-utilities as I have presented here, you will go far toward understanding the trends and connections within scientific literature. As mentioned, there are many items beyond publications that can be unlocked through mastering the E-utilities and how they operate within the larger universe of NCBI databases.</p><p>To learn more about accessing NCBI, <a href="https://ftp.ncbi.nlm.nih.gov/pub/PowerTools/eutils/">you can download course materials for a set of NIH courses that were held up until 2008</a> or check out the references below. To stay up-to-date about changes to the E-Utilities, including a potentially new API, you may want to sign up for the NCBI’s very 1990’s looking mailing list, <a href="https://www.ncbi.nlm.nih.gov/mailman/listinfo/utilities-announce/">at this website</a>. Lastly, searching arxiv.org for “PubMed” will return a handful of results that can motivate research projects using this data.</p><h4>References</h4><p>[1] <a href="https://www.nlm.nih.gov/bsd/difference.html#:~:text=MEDLINE%20is%20the%20largest%20subset,Journal%20Categories%20filter%20called%20MEDLINE">https://www.nlm.nih.gov/bsd/difference.html#:~:text=MEDLINE%20is%20the%20largest%20subset,Journal%20Categories%20filter%20called%20MEDLINE</a></p><p>[2] Chapman D. Advanced search features of PubMed. J Can Acad Child Adolesc Psychiatry. 2009 Feb;18(1):58–9. PMID: 19270851; PMCID: PMC2651214.</p><p>[3] Sayers E. The E-utilities In-Depth: Parameters, Syntax and More. 2009 May 29 [Updated 2022 Nov 30]. In: Entrez Programming Utilities Help [Internet]. Bethesda (MD): National Center for Biotechnology Information (US); 2010-. Available from: <a href="https://www.ncbi.nlm.nih.gov/books/NBK25499/">https://www.ncbi.nlm.nih.gov/books/NBK25499/</a></p><p>[4] <a href="https://ftp.ncbi.nlm.nih.gov/pub/PowerTools/eutils/Oct.2007/slides/Lecture1.pdf">https://ftp.ncbi.nlm.nih.gov/pub/PowerTools/eutils/Oct.2007/slides/Lecture1.pdf</a></p><p>[5] Lin J, Wilbur WJ. PubMed related articles: a probabilistic topic-based model for content similarity. BMC Bioinformatics. 2007 Oct 30;8:423. doi: 10.1186/1471–2105–8–423. PMID: 17971238; PMCID: PMC2212667. Available from: <a href="https://pubmed.ncbi.nlm.nih.gov/17971238/">https://pubmed.ncbi.nlm.nih.gov/17971238/</a></p><p>[6] <a href="https://library.mskcc.org/blog/2019/07/conflict-of-interest-statement-field-in-pubmed/">https://library.mskcc.org/blog/2019/07/conflict-of-interest-statement-field-in-pubmed/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56f76de22959" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/analyze-scientific-publications-with-e-utilities-and-python-56f76de22959">Analyze Scientific Publications with E-utilities and Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comprehensive Guide for Scientists Using ChatGPT3]]></title>
            <link>https://ai.plainenglish.io/how-i-use-chatgpt3-as-a-scientist-12832caf5048?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/12832caf5048</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Tue, 28 Feb 2023 21:33:01 GMT</pubDate>
            <atom:updated>2023-08-22T10:11:27.882Z</atom:updated>
            <content:encoded><![CDATA[<h4>A discussion of this technology’s promise, as well as pitfalls and perils</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2sPbborchumaRl2n" /><figcaption>Photo by <a href="https://unsplash.com/@siva_photography?utm_source=medium&amp;utm_medium=referral">Levart_Photographer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Science work, whether taking place at the bench, screen, class room or conference room, can all benefit from artificial intelligence. It is already making an impact on how we cite papers,¹ with examples like <a href="https://www.semanticscholar.org/">SemanticScholar</a>. For writing there are AI-powered assistants like <a href="https://www.grammarly.com/">Grammarly</a>, and tools like <a href="https://github.com/features/copilot">Github Copilot</a> for coding. ChatGPT3 is entirely free (for now), so I spent a few weeks experimenting with it. Similar to other language models, ChatGPT3 uses statistical regularities in petabytes of text that it has processed to both create a knowledge base and generate replies to user inputs.² I found myself amazed at first, then mortified, and later reassured that the computers won’t be taking our jobs anytime soon. Like with any new (and imperfect) technology, learning about ChatGPT3 early will give you the best chance to realize its full potential as this novel technology rapidly evolves. Note: for the rest of the article, I’ll refer to it as ChatGPT for reading ease.</p><blockquote>For background, I completed a PhD in <strong>neuroscience</strong> five years ago. Since then, I have primarily been working as a Postdoc doing data analysis alongside experimentalists in a <strong>cell biology</strong> lab, where we focus on single-molecule imaging. Most of my work has involved coding in MATLAB or Python. My time as a scientist has been split across the lab bench, teaching podium, and computer screen.</blockquote><p>Below I have outlined some of the ways I have used ChatGPT to assist me with basic lab tasks. For the <strong>“Too Long/Didn’t Read summary”: ChatGPT’s knowledge is miles wide, but only inches deep — and unlike Google, if it doesn’t have information you’re looking for, it will make things up</strong>, leading to potentially a lot of wasted time. From my experience, I believe ChatGPT is a big advance above Wikipedia or Google (with the major disadvantage that <strong>it lies</strong>), but it is far from anything like intelligence. You can read on to learn where I think ChatGPT excels over our current approaches, and where you’re better off sticking with the status quo for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M0boXbCWZdJXkJ1KkGbrpw.png" /></figure><p>I have attempted to aid myself in completing various science tasks using ChatGPT3, and I describe them below.</p><h3>Scientific Writing: Huge time saver</h3><p>Need a way to phrase something that you feel like you already wrote a million times? Or you want someone to complete your thought for you? ChatGPT is there for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eyPQoWGsdmaEc3-eD6aOzA.png" /></figure><p>And if you don’t like that response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5_WIe-vEdh0j6zWuLxS4Sw.png" /></figure><p>And ChatGPT can generate infinitely many such responses. Just keep asking it to try again. Remember, ChatGPT is not a search engine, it is a generator of text: recombining words with similar meanings and completing sentences in ways that make sense given what it knows.² Thus, you can rest assured that the sentence it wrote won’t already have been written (which is why ChatGPT can escape plaigarism detectors quite well). Unlike a writing partner, ChatGPT will never lose its patience with you. This way, you can turn writing into something more like a multiple-choice problem, which creates less cognitive load (very useful for those late nights before a deadline).</p><blockquote><strong><em>“LaTeX? Oh, yeah, that’s something our collaborators use…”</em></strong></blockquote><p>ChatGPT is phenomenal at helping you write basic documents in LaTeX/Overleaf, too. Writing with ChatGPT open in a nearby tab helps you generate rapid and concise LaTeX code. If you find yourself needing to use LaTeX for writing, especially equations, ChatGPT will provide wonderful guidance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dk2xyfuOdscqSpUNRdWXaw.png" /></figure><p>ChatGPT knows many standard equations, so you can simply ask it to write them out in LaTeX. I asked for LaTeX code to write out the Hodgkin-Huxley equation, an important equation in neuroscience and took the output to LaTeX:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ctKl_JxVxZ6qrsILwLWLvA.png" /><figcaption>Beautiful equation created by ChatGPT in LaTeX. It even describes each variable and constant.</figcaption></figure><p>ChatGPT knows its way around Microsoft Word and Google Docs, too, and can help you navigate the menu items and complete word processing tasks. In short, if you use ChatGPT like an assistant for taking care of the mechanical aspects of writing, it works nicely.</p><h3>Literature search: Don’t believe everything you read</h3><p>ChatGPT was good at synthesizing background information about different scientific fields and compiling it in a readable way, but when it came to asking for details about the literature, <strong>ChatGPT is inaccurate and confident about it</strong>. You can ask it about past collaborations between different types of labs and it will return a long list:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QnW7GdcByTESG9U4Yd9nrQ.png" /></figure><p>The problem is most of these papers don’t actually exist, or they aren’t published by who ChatGPT says they are. Remember, ChatGPT does not treat documents like research articles as individual objects,² thus it returns information based on statistical regularities in language, but it doesn’t know that research articles are a thing that cannot just be invented from just the relationships between words and names in a corpus. I am certain this is something that will be addressed by a major publisher like Elsevier when they license AI software.</p><blockquote>But it gets worse!</blockquote><p>I asked ChatGPT to summarize a specific scientific article published by my peers using chemogenetics to study thalamic activity in a mouse model.³ This is a publicly available article that was published in 2013. <strong>Here is where things get ugly:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*Z767ibOomRD7cIZvH6Sg2w.png" /></figure><p>The research in that paper was carried out <strong>in mice</strong>, not monkeys. That would make a huge difference to anyone in neuroscience, and would make you look pretty silly in a lab meeting or discussion with your peers. But it gets worse:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*eW_MJ7jOd2SqLHBwmA63MA.png" /></figure><p>The paper didn’t use optogenetics. It used chemogenetics, a substantially different method. Now, a project that used chemogenetics in rodents has become a project that used optogenetics in monkeys, which wasn’t even possible at the time of the publication. Optogenetics as a technique has rarely been used successfully in primates, so I asked ChatGPT for example references where it believes it has:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*Zzw2J-OuhyOSlzLvrhzfNw.png" /></figure><p>The first two papers literally <strong>do not exist. </strong>ChatGPT made them up. The second paper is about rodents.</p><blockquote>You can type “links” and ChatGPT3 will give you links and those links can be entirely made-up.</blockquote><p>Only the last two papers are about optogenetics in primates. Thus, I would not trust ChatGPT to summarize an individual paper or even to summarize newer developments in a field of science.</p><p>But maybe ChatGPT has learned how to structure PubMed searches — and you could use it to make more efficient use of PubMed? I asked it to create a search query for use in PubMed, of all articles where the last author was Gordon. It couldn’t even do this correctly. Here is what it returned:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8UVDd1nvSla0HI-T4gDLWQ.png" /></figure><p>The actual query should be “Gordon”[Author — Last].</p><p>While ChatGPT can summarize basic areas of science like “molecular biology” or even “protein biology”, it has no deep knowledge of science and it has no ability to reason about what it knows, <em>o</em>r <em>doesn’t know.</em> Thus, for literature searches, I would stick with getting good at PubMed and using citation tools. ChatGPT’s disregard for facticity and its comfort making up facts make it a highly unreliable source of information about scientific literature.</p><h3>Proprietary Software: Equally unreliable</h3><p>I asked ChatGPT questions about lab software like Prism, Geneious, and various molecular dynamics software. Each time, it would give me an answer. Sometimes a very long one! And each time you went to try it out in the software, you’d realize you had received <strong>plausible-sounding but completely made-up instructions.</strong> For Prism, I asked how to format Error Bars and it invented features of the graphical user interface that don’t exist:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*DhB-834FdDxcbODYdzOUsg.png" /><figcaption>ChatGPT invents instructions for using software.</figcaption></figure><p>Prism has neither a “Format” tab nor an “Error Bars” dialog box, nor does it have a “Line” tab. Instead, Prism has a dialog box called “Format Graph”, which ChatGPT did not seem to know about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*AJAK-FtyMGGUXU7xD4ozgg.png" /></figure><blockquote>If you ask ChatGPT instructions to do something it doesn’t know how to do, it will make it up.</blockquote><p>Similarly, when I asked it how to discover conserved motifs across sequences using the Geneious program, ChatGPT gave me an answer that just doesn’t exist in the software. I thought I was going crazy. I even checked multiple versions of software! Alas, just like ChatGPT does not seem to value the sanctity of a scientific article’s text as the bonafide source of information about what is in the article, it also does not place much value on a software’s online documentation — inventing its own instructions based on language regularities.</p><h3>Open-Source Software: Scarily unreliable</h3><p>Similarly to the case with proprietary lab software, ChatGPT will give you some basic ideas of the process you need to carry out to get your results, but when it comes to giving you specific instructions, it will be brazenly incorrect — again. In the case below, ChatGPT <strong>invents a GUI and again conjures up commands that don’t exist.</strong></p><p>I asked ChatGPT to tell me how to use <a href="https://github.com/ChandaLab/DISC">DISC, an open-source software package,</a> to fit a three-state model to FRET data, it responded:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/0*BXhtZZ6aTLW2N8EC" /><figcaption>ChatGPT also invents instructions for using open-source software.</figcaption></figure><p>Sounds plausible, the problem is <strong><em>there are no tabs</em></strong> in the DISC GUI. So I asked it about using the DISC as a command line function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/0*GX152rAUwGLAWLW2" /></figure><p>It seemed to think DISC was a command from a Linux-based operating system, which it’s not. It only runs from the MATLAB command line. When I informed ChatGPT that DISC is a MATLAB package, it returned:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/0*TVixtod5F9lb-kTT" /></figure><p>Unfortunately, neither of these commands are part of the DISC package, nor have they ever been.</p><blockquote>ChatGPT is willing to substitute plausible statements for actual instructions, but would it make up software that doesn’t even exist?</blockquote><p>I asked it about “FRET jump state analysis”, a non-sensical idea which nobody has ever proposed and I can confirm doesn’t exist. What happened next was quite alarming!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/0*pFToc929JNmLbaQ0" /><figcaption>ChatGPT creates GitHub repositories that don’t even exist based on fields of science that don’t exist!</figcaption></figure><p>Of course, the link didn’t work. While there is at least one Köster laboratory that works on computational biology, this is not their GitHub. I asked ChatGPT if it was sure about FRETJumpStateAnalysis and it corrected itself, saying this wasn’t in fact a real package, but then unfortunately suggested other GitHub packages with links that <strong>also don’t exist </strong>(this was a bad day for GPT). Confirming that a webpage exists seems like a simple step ChatGPT could implement that could prevent a lot of headaches and wasted time.</p><h3>Data analysis and coding: Gets there eventually, but no replacement for expertise</h3><p>When it comes to basic Python and MATLAB tasks, ChatGPT3 is good at giving you the code you need to carry out your projects. This can save you a ton of time. Asking for basic things like<strong> “scatter plot with points colored by value” </strong>returns the right answer, and it will even give a full and complete response when asked to write code for carrying out maximum likelihood estimation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BB_gLT8gIgtAIt3DlsWELA.png" /><figcaption>ChatGPT’s reasonable and valid MATLAB code for performing maximum likelihood estimation.</figcaption></figure><p>In this way, ChatGPT3 could be used as a very patient tutor for helping you learn to implement advanced statistics in a scientific computing environment — very good!</p><p>I tried it out with Python to accomplish some basic machine learning tasks, where it needed a bit of nudging to complete the tasks. My query was <strong>“use Python to find a line that best separates two sets of points.”</strong> It told me to use PCA and gave me an example, which isn’t correct. I specified:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdBqvAxjmAx_R1x6N6bElw.png" /><figcaption>Help ChatGPT along by telling it where it went wrong.</figcaption></figure><p>Then it got to a right answer: support vector machines. That’s fine, but it requires labeled training data which I did not say I had. I specified that I needed an unsupervised method. I went back and forth with it for a little while, as it tried a few different approaches but had some errors that could have been due to differences in the versions of sklearn. It eventually solved the problem using a module of sklearn that implements spectral clustering:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ORClCyd3-HEWudqK7_6fvw.png" /><figcaption>ChatGPT correctly divides linearly separable data using spectral clustering.</figcaption></figure><p>This was a good enough solution for the simple case of linearly separable data. But what about the more difficult case, for example, circles. Here was ChatGPT3’s solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rB_6dUKeLOIk4F5dGBh4vA.png" /><figcaption>ChatGPT believes this is a good result…</figcaption></figure><p>I kept telling it that it wasn’t right, and it just gave me the same thing, substituting with a different example dataset each time (it cycled through moons, then blobs, circles again) — never getting it right. Eventually, I asked whether initial parameters might matter for the result. It gave me a long discussion on the choice of parameters and then showed me two examples of spectral clustering with different settings for the “affinity” parameter in spectralclustering (the default settings at left, and using nearest neighbors at right):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XZapUEiA44SGoToBLp-ctA.png" /><figcaption>With some nudging, ChatGPT realizes it needs to use parameters. It shows you what happens without setting the affinity parameter (left) and when it is set to nearestneighbor (right).</figcaption></figure><p>ChatGPT seemed to know a lot about the importance of tuning parameters for spectral clustering, but it didn’t get to it on the first try, or even second or third. ChatGPT needed significant nudging. It felt like I was teaching it what to do, rather than vice versa. To be clear, the instructions for carrying out this task are literally contained — verbatim — on the website documenting sklearn.</p><blockquote>It would have been a lot faster and easier to get an example from the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.cluster.SpectralClustering.html">sklearn documentation</a>, freely available online.</blockquote><p>Next, I gave it a more open-ended query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SibDy3kagl92-H-NLJQSlQ.png" /></figure><p>For this, it provided three reasonable approaches with variable complexity: thresholding, adaptive thresholding, and blob detection. They all relied on the cv2 package. I could say “without cv2” and it gave me a ton of other algorithms for carrying out the task. Whether they would run without some debugging isn’t as important as the fact that ChatGPT gave me a ton of ideas and ways to implement them. Here, I’d say ChatGPT did a great job. So while I’m sure that ChatGPT3 will save me a lot of finger calories when I’m carrying out basic tasks or starting a new project, it clearly doesn’t have the sophisticated knowledge of data analysis that you would need in most areas of scientific computing. I’m not worried about it taking coding work away from anyone in the lab.</p><h3>Practical lab tasks: Suggestions to be checked</h3><p>For practical tasks like preparing solutions or carrying out an experimental method, ChatGPT will offer <strong>understandable </strong>instructions but these should never be a substitute for methods printed in a paper. I emphasize the “understandable”, because while a journal provides methods sections that are exhaustive and comprehensive, the results ChatGPT provides will be readable but not necessarily complete (and potentially inaccurate as we saw above). But unlike a journal, with ChatGPT you can ask follow-up questions, or clarification. I imagine this could especially be useful for non-native English reading scientists who might benefit from having a certain sentence or word rotated out by ChatGPT’s transformation process, and substituted with another word or rewritten in an alternative way.</p><p>I asked ChatGPT about the optimal concentrations for several chemical reagents, and its answers were accurate. It also told me to perform pilot experiments. It also provided me with a recipe for making artificial cerebrospinal fluid, a standard neuroscience lab concoction. It gave me this recipe, which is roughly equivalent to one you’d find at this Cold Spring Harbor website⁴ — which kind of begs the question of why you wouldn’t just use the website since we know Cold Spring Harbor is a reputable source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xwWTUqNJzznN8NhWmN6NPQ.png" /></figure><p>When asked about the best antibodies to use in an experiment, ChatGPT provided an accurate and comprehensive answer. I would say ChatGPT performed well with searching for lab materials. In general, ChatGPT can be effective as a personal shopper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hfm23bSk2pAwFKxxxEXCrQ.png" /></figure><p>There are more obscure antibodies, which it did not tell me about, and I am certain I could have returned these results using Google, although it might have required more clicking through various websites, getting tons of cookie requests, and seeing irrelevant information. In this way, ChatGPT helps you shop.</p><h3>Setting up a laboratory: Comprehensive and correct</h3><p>As you may have already realized, ChatGPT is incredibly chatty. It will give you a lot of details, which can be helpful when you are trying to learn about many aspects of a topic. This makes it useful for answering general questions about setting up a laboratory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WWfg2S9R2QwLVk5U9qA9VQ.png" /><figcaption>Response truncated. It also told me about funding and institutional support.</figcaption></figure><p>ChatGPT also gave me a reasonable estimate for a budget for my hypothetical rodent electrophysiology lab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*bYEe89yQpQ54FVDHT6CCeQ.png" /></figure><p>These are all very realistic, and they even give ideas for specific parts. Again, ChatGPT can be very useful if you have money to burn or you’re writing a hypothetical budget — but this is something which your grants office could probably help you do in a more accurate way. ChatGPT also knows about grants, both public and private, even for specific types of research. When asked about private grants to fund electrophysiology research, it returned Brain &amp; Behavior Research Foundation NARSAD, McKnight, Alzheimer’s Association, Whitehall, and Simon’s as potential funders — nothing made up!</p><h4>Create instructional materials: Excellent</h4><p>Occasionally, researchers will have to design courses, and ChatGPT really shines here. It can easily generate a variety of different syllabi for even specific types of seminars. Here’s one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NQZbgNQHZkdITez3QeAiYg.png" /><figcaption>Truncated the response. In the complete response, it provides 8 weeks of classes.</figcaption></figure><p>When I asked it for readings, it gave me a list of readings for each week, which were actually true. I suspect it has learned a large corpus of syllabi that repeat and it’s assembling fragments of those. I asked it for discussion questions and it provided me this comprehensive list:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oos-GwDoIHyEtQa3k5RFFw.png" /></figure><p>It also gave me answers those questions which were correct. As long as you don’t get too specific about what you want to learn, this could save you a ton of time when you’re teaching a course.</p><h3>Conclusion</h3><p>One of the most important things to keep in mind when using ChatGPT is that <strong>it is not a search engine. </strong>It is <em>creating responses </em>to you based on matching statistical regularities in its “knowledge base”. This is why ChatGPT will invent statements that you will <strong>never actually find</strong> <strong>anywhere</strong> on a website or document. You’ll find it helpful for rephrasing a sentence you’ve written, but not for finding out details about what a specific scientific paper contains. ChatGPT is not fact-checking itself, it’s <strong><em>not even checking its own links</em></strong>, or running code that it gives you. Thus, ChatGPT should be seen as a time-saving assistant for lab tasks, compiling information for you to assess, and never as an authoritative source.</p><h3>What comes next?</h3><p>ChatGPT is nowhere near usurping the skills or knowledge of a scientist or even a coder. The productive attitude toward the use of ChatGPT and future AI in the sciences will be to learn the limitations of the technology and use them in a careful way to free up your time to hone the skills necessary for filling those gaps in the AI technology.</p><p>Even with improvements in technology, subjective measures like beauty, clarity, or utility, remain squarely in the realm of human intelligence. In science, this means that ChatGPT cannot output a highly-memorable and understandable figure, poster or conference presentation, a grant that answers exactly the right question at the right time for your field, or a software package that will be used by many. Additionally, for specific tasks, ChatGPT has minimal reasoning abilities.</p><blockquote>If there are inconsistencies in the literature, ChatGPT will not ponder them, weigh the quality of the evidence, or make logical deductions about how to resolve inconsistencies based on the premises its knowledge contains.</blockquote><p>You can ask it to hypothesize, but it will only recombine the previous literature in predictable (and strange, or even very incorrect) ways. It won’t figure out the solutions to open questions in science. For now, ChatGPT is less like a wise senior scientist and more of an autocompleting Wikipedia that can code a lot of things somewhat well. In other words, a combination of technologies that have recently been developed along with some slight improvements. These are the realities that must be remembered as we familiarize ourselves with this new technology.</p><p>Lastly, here’s a chart to show you how you might think about what tasks are appropriate and safe for ChatGPT to do. You’ll want to use ChatGPT with tasks where there is a high agreement among humans, and where the cost of getting it wrong is low. There are grey areas, where you should exercise care and consult with peers or your supervisor. Then there is the “Danger Zone”, which are tasks that you should certainly consult and work with others to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dn2mjMf3hnS2AaraGmw-gg.png" /></figure><h4>If you made it down here, here is my summary of how ChatGPT fared on a variety of science uses</h4><p>Writing: <strong>5/5</strong><br>Literature search: <strong>1/5</strong> <strong>(avoid!)</strong><br>Proprietary and open-source software: <strong>0/5</strong> <strong>(hard pass!)</strong><br>Data analysis and coding: <strong>4/5</strong><br>Guiding practical lab tasks (including shopping): <strong>3/5</strong><br>Setting up a lab: <strong>4/5</strong><br>Creating course materials: <strong>5/5</strong></p><h3>As there is definitely no substitute for crowdsourcing knowledge, please feel free to use the comments section to add your own uses of ChatGPT or other types of AI inside (and outside) the laboratory!</h3><h4>References:</h4><p>[1] The Bibliomagician. “AI-based citation evaluation tools: good, bad or ugly?” <a href="https://thebibliomagician.wordpress.com/2020/07/23/ai-based-citation-evaluation-tools-good-bad-or-ugly/">https://thebibliomagician.wordpress.com/2020/07/23/ai-based-citation-evaluation-tools-good-bad-or-ugly/</a> (2020).</p><p>[2] Brown, Tom B., et al. “Language models are few-shot learners.” arXiv preprint arXiv:2005.14165 (2020).</p><p>[3] Parnaudeau, Sebastien and O’Neill, Pia-Kelsey, et al. “Inhibition of mediodorsal thalamus disrupts thalamofrontal connectivity and cognition.” <em>Neuron</em> 77.6 (2013): 1151–1162.</p><p>[4] “ACSF Recipe.” <em>Cold Spring Harb Protoc</em> (2007) (<a href="http://cshprotocols.cshlp.org/content/2007/2/pdb.rec10804.full?text_only=true">http://cshprotocols.cshlp.org/content/2007/2/pdb.rec10804.full?text_only=true</a>)</p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12832caf5048" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/how-i-use-chatgpt3-as-a-scientist-12832caf5048">Comprehensive Guide for Scientists Using ChatGPT3</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Out of memory” in MATLAB? Easily cut memory use in half with this tip]]></title>
            <link>https://neuraljojo.medium.com/out-of-memory-in-matlab-easily-cut-memory-use-in-half-with-this-tip-3d718a5a6e4?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d718a5a6e4</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[matlab]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Tue, 30 Aug 2022 22:09:39 GMT</pubDate>
            <atom:updated>2022-08-30T22:09:39.760Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ll demonstrate how simply changing the precision of your numeric data can cut your memory usage by half or even more</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ozY4tKzgZn2ndUJI" /><figcaption>Photo by <a href="https://unsplash.com/@possessedphotography?utm_source=medium&amp;utm_medium=referral">Possessed Photography</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Have you ever ran into an issue with memory in MATLAB and you went looking online?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*mDiPHEcSI3uFst9DHfba1w.png" /><figcaption>Google Search result for what to do if you run out of memory.</figcaption></figure><p>Turns out #7 is the simplest solution in many cases and can cut your memory use by up to 90%. This method comes with the <strong>added benefit that if you save your variables, they will also take up less hard drive space and load much more quickly</strong>.</p><p>Let’s dive right in to understand how we can reduce memory usage (the concepts apply to other programming environments, but I’ll focus on MATLAB):</p><p><strong>Critical commands discussed in this post:</strong></p><p><strong>whos </strong><strong>single </strong><strong>double </strong><strong>int16 </strong><strong>format</strong></p><p>When it comes to numerical data, you can specify the type using single or double but people almost never do! What does that mean for your data? Let’s take a look with some random examples:</p><pre>mydata = rand(1000,1000); % Do not omit this semi-colon!<br>whos mydata</pre><p>You’ll see a table that says your variable <strong>mydata </strong>has a value of Bytes equal to: 8 000 000. That’s 8 megabytes. Stored in your memory.</p><p>Next to bytes, it will say “double”. MATLAB will, by default, create a very wasteful double precision variable. Try this:</p><pre>mydata = single( rand(1000,1000) ); % Do not omit this semi-colon!<br>whos mydata</pre><p>What do we lose by going from double to single?</p><pre>mydata_double = rand(1000,1000);<br>mydata_single = single( mydata_double );<br>mean(mydata_double,&#39;all&#39;) % No semi-colons here<br>mean(mydata_single,&#39;all&#39;)</pre><p>You’ll see that the result for the double precision variable is “0.500049032047244”, whereas the result for the single precision variable is “0.5000491”. If the difference between 5,000,490 and 5,000,491 will have a substantial effect on your data, then <strong>do not convert to single</strong>.</p><h3>Can we do even better with integers?</h3><p>Yes. With caveats. If you are happy with 50%, you can stop reading here. However, if you’re willing to go the extra mile, continue. You should only do this if you test your results with single and then with the integer formats int8 or int16— and you confirm they’re comparable. Working with integers is complicated but can be worth it in some cases.</p><ul><li><strong>Caveat 1:</strong> Data should either already be integers in double precision format, or round-able to integers without much loss of information (e.g. 1.001, 2.0045, etc), or you should be prepared to transform your data back and forth.</li></ul><p>To see how this caveat can hurt you if you aren’t careful — try this:</p><pre>mydata = rand(10,10);<br>mydata = int8( mydata );<br>mydata</pre><p>❌ You’ll get a bunch of 0’s and 1’s, because your data has been rounded.</p><p>How can we avoid this? You can multiply by each number by 10, 100, or 1000 and then subtract later if necessary.</p><pre>% Scale up the data and scale it back down<br>to_integer = @(x) x*1000<br>to_decimal = @(x) double(x)/1000</pre><pre>% Convert to integer<br>mydata = int16( to_integer( rand(10,10) ) );</pre><pre>% And back to double!<br>mydata = to_decimal( mydata );</pre><ul><li><strong>Caveat 2:</strong> Your integers must have a set minimum and maximum value that you won’t exceed (and it will be rounded down to that value, if you do).</li></ul><p>If you are converting using int16 that means -2¹⁶ to 2¹⁶, so you could potentially store values between -65536 and +65536 — or -6.5536 and +6.5536 to have the most precision in your decimals — or between -65.636 and +65.636. See how that works?</p><blockquote>What is the maximum value of an int8 matrix? Scroll to the bottom for the answer.</blockquote><p><strong>Caveat 3:</strong> You’ll need to convert to double or single if you want to do certain kinds of processing on your matrix. For example, you’ll be able to take the mean or standard deviation of your matrix, but you won’t be able to use inv to compute the matrix inverse, for example.</p><h3>Let’s check out the savings from each type of data</h3><p>We’ll try with the assumption that our data will be less than 256 and we can tolerate losing some decimals (you should always check this works for your data).</p><pre>to_integer = @(x) ( x*100 )<br>to_decimal = @(x) ( double(x) /100 )</pre><pre>mydata = rand(1000,1000); % 8 megabyte double<br>mydata_to_integer = to_integer(mydata); % Still 8 megabytes (double)<br>mydata_to_integer = single( mydata_to_integer ); % Now 4 megabytes</pre><pre>% Check if the min and max are between -2^16 and 2^16<br>[ min(mydata_to_integer,[],&#39;all&#39;),max(mydata_to_integer,[],&#39;all&#39;) ]</pre><pre><br>mydata_to_integer = int16( mydata_to_integer ); % Now 2 megabytes</pre><pre>% Since the max is 100, we can go down to int8</pre><pre>mydata_to_integer = int8( mydata_to_integer ); % 1 megabyte!</pre><p>Now let’s try to recover our original data and see how accurate it is by computing the mean:</p><pre>format long<br>mean( mydata, &#39;all&#39; ) % Original data<br>to_decimal( mean( mydata_to_integer,&#39;all&#39; ) ) % Converted back data</pre><p>0.500068458752177 versus 0.500070540000000. Let’s compute the standard deviation as well.</p><pre>std(mydata,[],&#39;all&#39;)<br>std( to_decimal( double( mydata_to_integer ) ),[],&#39;all&#39; )</pre><p>0.288641641801215 versus 0.288675051151625 — we can live with it.</p><h3>Conclusion</h3><p>To save memory (RAM and hard drive storage), you should use single(yourvariable). This is the easiest and most straightforward way to cut your memory use in half. If you want to cut your memory use by up to 90%, you should carefully experiment with int16 or int8 (which stores values between -256 and +256, as the answer to the question above).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d718a5a6e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can you transition from MATLAB to Python without classes?]]></title>
            <link>https://medium.com/analytics-vidhya/can-you-transition-from-matlab-to-python-without-classes-38a5c7ab0fed?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/38a5c7ab0fed</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[matlab]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[neuroscience]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Sun, 31 Jul 2022 14:01:22 GMT</pubDate>
            <atom:updated>2022-09-09T14:19:16.237Z</atom:updated>
            <content:encoded><![CDATA[<p>Academic scientists steeped in MATLAB will need to be comfortable with classes to successfully transition to Python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FX5HWqO0aZ2qPHIe" /><figcaption>Classes can be useful for trying to educate ourselves— but in computer science, classes are programmatic instructions for simply achieving sophisticated computational objectives in a self-contained way.</figcaption></figure><p>When is the last time you sat down to your MATLAB console and typed out something like <strong>mydata.myfunction()</strong>? Right now you might be saying <strong><em>“That’s not MATLAB”</em></strong>. Especially if, like me, you learned MATLAB because you needed a way to analyze your data (or a collaborator needed a way for you to analyze <em>their </em>data).</p><h3>Why does <strong>mydata.myfunction() </strong>look so odd to us?</h3><h4>Because programming with classes and objects is not common for MATLAB users. But if you want to transition from MATLAB to Python, you absolutely need to be familiar with classes and objects.</h4><blockquote>How do you get started? If you have ever written a MATLAB function, you will be able to write and understand classes.</blockquote><p>This post is intended for intermediate-level MATLAB users to get familiar with the vocabulary and implementation of <strong>classes and objects, an essential step in transitioning to more industry-oriented tech positions.</strong></p><p>Let’s cover three basic concepts and jump into an example:</p><ul><li><strong>Classes: </strong>Basically, a blueprint for a set of functions and a structure that will store data for those functions <strong>(stored in MATLAB as a .m file)</strong></li><li><strong>Objects: </strong>Imagine taking a blueprint and turning it into a real thing: an <strong>object</strong>. But the <strong>object </strong>goes further than the blueprint<em>: Not only does it have instructions for building the house — but once you built it, it also has instructions for later painting the walls, switching out the carpet, putting up holiday decorations, etc </em><strong>(objects are created in your workspace and each object is treated as its own variable)</strong></li><li><strong>Methods: </strong>A way to execute functions that are specific to your object <strong>(methods are written into the class file and used on objects)</strong></li></ul><p>Basically, <strong>objects </strong>are like variables with a limitless repertoire of things they can do to themselves — they store data but they can also alter that data (hence, the methods). <strong>Methods </strong>are functions specific to your object — defined by the <strong>class</strong>. Additionally, I will show examples of two other concepts: <strong>class instances </strong>and <strong>overloaded functions </strong>(not a bad thing!)</p><h3>You may already use classes in MATLAB without knowing it</h3><p>One example of a <strong>class </strong>that MATLAB users rely on is <strong>gmdistribution, </strong>a <strong>class </strong>for creating a Gaussian Mixture distribution.</p><pre><strong>mygm = gmdistribution([0,10],[1,2])</strong></pre><p>MATLAB will, rather stoically, return:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eSyLlg1J-c6ynweOBGap9w.png" /></figure><p>Is this the end of the object <strong>mygm</strong>? Typically, with a function, you enter it and what you get is an output and you move on. But <strong>mygm </strong>is an <strong>object of class</strong> <strong>gmdistribution </strong>and creating it was just the first step.</p><p>Like for any <strong>object</strong>, you can get help by typing: <strong>help(mygm)</strong>. You can also look at the type of <strong>class </strong>(blueprint) that built the object by typing: <strong>class(mygm)</strong>.</p><p>Let’s type your first command using dot notation:</p><pre><strong>two_random_numbers = mygm.random()</strong></pre><p>If you were to just type “.random()” on a variable that wasn’t a <strong>gmdistribution</strong>, you’d get an error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*do5wOxWYbW6pXkEUgVNKGA.png" /></figure><p>That’s because <strong>methods </strong>are <strong><em>specific to classes</em>. </strong>Now, let’s explore some properties of <strong>mygm</strong>. Remember that a <strong>class </strong>not only has functions (which we refer to as <strong>methods</strong>) but also information about your data, which we call <strong>properties or attributes</strong>.</p><p>By typing <strong>help(mygm)</strong> you’ll see a list of <strong>properties/attributes</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zULJudFWigPz-z1dDPJNpw.png" /></figure><p>You can retrieve any of these <strong>properties </strong>using dot notation.<strong> </strong>Just type one of them after <strong>mygm</strong> and a dot — just like you did for the<strong> function/method</strong>.</p><pre><strong>mygm.NumVariables<br>mygm.DistributionName<br>mygm.mu<br>mygm.Sigma</strong></pre><pre><strong>%etc...</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BxoY55QAS_4iLCFUjs8AMg.png" /></figure><h3>videoWriter, another example, of a MATLAB class</h3><p>Ever tried to write a video in MATLAB? It involves <strong>VideoWriter</strong>, a <strong>class</strong>.</p><pre><strong>video1 = VideoWriter(‘myfile.avi’ , ‘MPEG-4’); % House building part</strong></pre><p>Remember that you can build a green house and then later decide to decorate the walls. Objects can be designed for you to modify their properties even once they’re created! The <strong>VideoWriter </strong>class enables this.</p><pre><strong>video1.Quality = 95; % Decorating the walls part<br>video1.FrameRate = 20;</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*LnzYyZSqvh6RwBAIrMJNjQ.png" /></figure><p>You’ll next write a single frame to your video:</p><pre><strong>a = rand(300);</strong><br><strong>video1.open();<br>video1.writeVideo(a);<br>video1.close();</strong></pre><p>Typically with a function, you call the function and that’s the end of it. With an <strong>object</strong>, there can be a process with a set of steps that involve setting <strong>properties </strong>and invoking <strong>methods</strong>, as we did above with the Quality, FrameRate, and then ultimately, writing the frame. Clearly, <strong>objects</strong> are more complex than functions alone, but they’re also far more self-contained, logical, and the basis for more advanced programming environments (like Python).</p><h3>Now let’s write our own custom class</h3><p>In MATLAB, a <strong>class </strong>is created using the editor and saved as a “.m” file, just like a function. Copy this one into the MATLAB editor and save it as <strong>fileclass.m</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95c4fe7986a23c6c1c279f5176e00ef7/href">https://medium.com/media/95c4fe7986a23c6c1c279f5176e00ef7/href</a></iframe><p>This <strong>class </strong>will allow you to store information about a file, summarize it, and compare the sizes of files (it won’t modify or do anything to the files themselves).</p><p><strong>Constructor method </strong>The most important method in any <strong>class</strong> is the <strong>constructor method</strong>. Best practice is to have the file name of the class, the classdef and the constructor named the same way. The constructor method can take arguments (in our case, inputfile) or not, just like a typical function.</p><p>Now let’s give our example class a test run. Copy and paste the code that applies to you from below. Browse to a directory and take a look at the files located there by doing as follows:</p><pre>% Windows example<br>cd(&#39;C:\Windows\&#39;)<br>dir(&#39;*.exe&#39;)</pre><pre>% Mac example<br>cd(&#39;\bin&#39;)<br>dir()</pre><p><strong>New vocabulary word alert — instance: </strong>I’m going to create two <strong>instances</strong> of our fileclass <strong>object </strong>by typing the lines below:</p><pre>% Windows example<br>file1 = fileclass(&#39;writer.exe&#39;);<br>file2 = fileclass(&#39;notepad.exe&#39;);</pre><pre>% Mac example<br>file1 = fileclass(&#39;bash&#39;);<br>file2 = fileclass(&#39;sh&#39;);</pre><p>Try out the <strong>method </strong>we created called <strong>summary. </strong>If you aren’t sure how, just glance up at the work we’ve done in this post so far.</p><p>The other <strong>method </strong>of fileclass we wrote is <strong>lt</strong>, which means “less than”. However, because this is considered a “built-in” function and an operator of MATLAB, the presence of <strong>lt </strong>in the class is understood as an <strong>overloaded function</strong>. Why? Because you are creating a unique additional role for a function that already existed! Go ahead and try it:</p><pre>file1 &lt; file2</pre><p>This might seem like a trivial use — but imagine a situation where you have created your own class, let’s class it “eventclass” and it defines when an event occurs and what kind of event it was — you can use the same approach taken here to tell you whether one event happened before the other by simply writing event_x&lt;event_y!</p><p>You may be wondering how <strong>functions</strong> differ from <strong>methods </strong>— and hopefully the above example shows you that methods are far more extensible (a programming term meaning it enables more use cases). For example, you could never write a stand-alone <strong>function</strong> that is the less than comparator above. There are many other wonderful ways to extend methods for a variety of uses — which is at the core of much programming.</p><h3>Now how does this help me with Python?</h3><p>Nearly everything in Python relies on <strong>objects </strong>and <strong>classes</strong>. Even a numpy array is an <strong>object</strong>, which is different from a pandas dataframe <strong>object</strong>. Which means they have different <strong>methods</strong>!</p><p>Let’s go ahead and reproduce the <strong>class </strong>we just made in MATLAB, this time in Python. I recommend using Python from within Google Colab for a carefree prototyping experience. Simply type or paste the code below into a Python cell. Execute it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc989099f3c9f9a7a5e999b281c04d91/href">https://medium.com/media/fc989099f3c9f9a7a5e999b281c04d91/href</a></iframe><p>Can you tell what the constructor function here is? Answer below.</p><p>Now, if you are in the Google colab environment, you can browse to the sample data files, otherwise find some files on your local computer:</p><pre>import os<br>os.chdir(&#39;/content/sample_data&#39;)<br>os.listdir()</pre><pre># Output [&#39;README.md&#39;,  &#39;anscombe.json&#39;,  &#39;mnist_train_small.csv&#39;,  # &#39;california_housing_train.csv&#39;,  &#39;california_housing_test.csv&#39;,  <br># &#39;mnist_test.csv&#39;]</pre><p>Now in the next cell, try out the <strong>summary </strong>method. (Hint: This will be done using dot notation!) — then try the less than operator, just like you had in MATLAB. You should see the clear parallels between using classes in MATLAB and Python, and now you are capable of understanding object-oriented programming in both environments!</p><blockquote>Answer: The constructor method for a Python class is always “__init__”, short for initialize.</blockquote><p>Here is a quick table that you can refer to for remembering how classes, objects, and methods work together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCheAmYmBFY2wWcsT2cMuA.png" /></figure><h3>I hope this post has helped you see how to bridge MATLAB and Python. Hopefully you can simultaneously advance your skills in both to accomplish your objectives.</h3><p>There’s lots more to learn about classes in both MATLAB and Python, and I’ll be discussing these topics in future posts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38a5c7ab0fed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/can-you-transition-from-matlab-to-python-without-classes-38a5c7ab0fed">Can you transition from MATLAB to Python without classes?</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A few MATLAB table tricks I had to learn the hard way]]></title>
            <link>https://neuraljojo.medium.com/a-few-matlab-table-tricks-i-had-to-learn-the-hard-way-ed5cfa876793?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed5cfa876793</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[matlab]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Sat, 12 Mar 2022 21:23:02 GMT</pubDate>
            <atom:updated>2022-08-09T01:09:13.075Z</atom:updated>
            <content:encoded><![CDATA[<p>Tables are very versatile which means there isn’t a single document explaining the uses you might have for them, which is why I created this tutorial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*_hRpr-oKGffvYGcV" /><figcaption>Tables! They’re everywhere but we can waste hours of time if we aren’t using them to their full potential.</figcaption></figure><h3>In this tutorial, I will outline best practices for <strong>making </strong>a table, filtering a table, and displaying table contents.</h3><p>In MATLAB, nearly every entry point for a thorough exploratory data analysis or machine learning application is a table. Sure, you can do these things by cobbling together matrices, arrays, cells, structures — but your code can get extremely long and indecipherable. Bad for replicability! Here are some tips for getting started with tables.</p><p>These are not strategies that MATLAB documentation will outline for you — they are very specific pathways through the table variable type which I have had to discover through many hours of painstaking trial and error. MATLAB tables are one of the most opaque concepts that the average MATLAB user will encounter: replete with non-intuitive conventions, sparsely documented properties or functions, and cryptic error messages. I hope this tutorial will help speed up your learning process.</p><h3>Four quick ways to make a table:</h3><p>Here are four of the most common ways to generate a table: <strong>table</strong>( var1,var2,var3,etc…), <strong>array2table</strong>( matrix), <strong>readtable</strong>(‘file.csv’), and <strong>dot notation</strong>.</p><h4>1) table( <strong>var1,var2,var3,etc… )</strong></h4><p>Let’s create some data in the form of <strong>x</strong>, <strong>y</strong>, and <strong>score</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f494fb0ccc915f993cb72959d64a95ca/href">https://medium.com/media/f494fb0ccc915f993cb72959d64a95ca/href</a></iframe><p>Here’s how you would make it into a table using <strong>table()</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c54c7241f963eb9f822b68c34656372/href">https://medium.com/media/3c54c7241f963eb9f822b68c34656372/href</a></iframe><p>If you do not specify a ‘VariableNames’ input to the table function, the columns will be named what you passed to the function, “x”, “y” and “score”.</p><h4>2) <strong>array2table( matrix )</strong></h4><p>One of the most annoying things that you’ll encounter with tables (and a mistake you’ll surely make even as a TablePro™) is trying to just put a matrix into the table command above. MATLAB will allow it, but this is almost never what you want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*Uc5QOnNWByi7uGky_lab_Q.png" /><figcaption>Don’t make this table!</figcaption></figure><p>If you are trying to convert a matrix into a table, use <strong>array2table() </strong>instead.</p><p>This will make each column in your matrix into its own column in the table. Much better!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*Gr00drneIYkfezlQCgsofQ.png" /></figure><p><strong>3) readtable( ‘file.csv’ )</strong></p><p>This is pretty self-explanatory and it’s compatible with Microsoft Excel (as long as you have Microsoft Excel also installed). I strongly recommend getting all your data in a cleaned-up CSV, which you can do by opening your Excel Spreadsheet in Google and exporting it. This is much easier than using an XLS file.</p><p><strong>4) You can build a table using dot notation, with some caveats.</strong></p><p>Generate an empty table:</p><pre>mytable = table()</pre><p>Now try to add data to it:</p><pre>mytable.Stuff = [1:3]&#39;</pre><p>Cool! But once you have added your first column using dot notation, you have defined your table’s size and there’s no real going back!</p><p>Here’s what I mean: try adding a column of zeros.</p><pre>mytable.Zeros = 0</pre><p>No. This won’t work! Here is what would work:</p><pre>mytable.Zeros = repmat(0,size(mytable,1),1)</pre><blockquote>What about a column of text?</blockquote><h3>You will <strong><em>always, always want to enclose text in cells</em></strong> when adding it to a table.</h3><pre>mytable.Country = {&#39;Canada&#39;;&#39;US&#39;;&#39;Mexico&#39;}</pre><p>Or</p><pre>mytable.TextColumn = repmat( {&#39;To fill later&#39;}, size(mytable,1), 1 )</pre><p><strong><em>(For advanced users)</em></strong> You can use <strong>rowfun</strong> to make a new table from an existing table, but if your newly formed table is a column of text, you need to enclose every element in brackets:</p><pre>mytable.TextColumn = rowfun(@(text1,text2) ...   {sprintf(&#39;%s_%s&#39;,text1,text2)}, my_other_table )</pre><blockquote><strong>Bonus Tip: </strong>If you have a table and you want to get, set, or change its column headings, you have to do it like so:</blockquote><pre>mytable.Properties.VariableNames = {&#39;Col1&#39;,&#39;Col2&#39;,&#39;etc&#39;}</pre><p><strong>Another Bonus Tip: </strong><em>It’s no secret that every row in a matrix must have the same number of elements — the same is true for tables — so you cannot make a table like this:</em></p><pre>table( [[1:10];[1:5]] )</pre><p>This will give you a concatenation error. <strong>However,</strong> this will work and you should use it:</p><pre>table( [{1:10};{1:5}] )</pre><h3>Filtering a table based on various criteria</h3><p>One of the most common operations you want to do in a table is filter it. Run the sample code below to get a table that contains some text, labels, integer data, and time series data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/890182b96b4a909352ef818139e3e36c/href">https://medium.com/media/890182b96b4a909352ef818139e3e36c/href</a></iframe><p><strong>I will go through and show how you would filter the table based on any of the columns.</strong></p><p><strong><em>CellNumber</em></strong></p><p>Notice the code below — it doesn’t work — and you will get back a totally cryptic error talking about three or more subscripts or something like that:</p><pre><strong>filteredtable = mytable( mytable.CellNumber==1 )</strong></pre><p><em>You must always include a comma and semi-colon after your filtering condition.</em></p><pre>%You need to do: <br><strong>filteredtable = mytable( mytable.CellNumber==1, : )</strong></pre><p>You might also want to try and filter a table based on whether the CellNumber is within a set that you’re interested in — here’s how using the <strong>ismember </strong>function:</p><pre>filteredtable = mytable( ismember(mytable.CellNumber,[1,3,4]),:)</pre><p>This is something that you would do if you wanted to isolate the effect of certain variables in your table.</p><p><strong><em>Means</em></strong></p><p>Similar story here. Let’s say you want rows with means greater than 0.5. You’ll use <strong>mytable( mytable.Means&gt;0.5, : )</strong></p><p>Now what about that NaN in the <strong>Means </strong>column? MATLAB does not make it straightforward to delete rows containing NaN. But no worries. Here’s how you’d do it:</p><pre>filteredtable = mytable( ~isnan(mytable.Means),: )</pre><p>Basically, you’re just finding the values where the <strong>Means </strong>column is not a NaN. This is not the same as replacing all NaN values in your table, which is a more complicated task and I recommend using a custom script or function.</p><p><strong><em>Label</em></strong></p><p>Let’s say you want to filter by text in a column. Seems easy, right? Again, the intuitive solution doesn’t work —</p><pre>filteredtable = mytable( mytable.Label==&#39;Open&#39;,: )</pre><p>This is because you cannot filter on text columns. You can only filter on categorical variables. So enclose mytable.Label inside a <strong>categorical( ) </strong>type conversion.</p><pre>filteredtable = mytable( categorical(mytable.Label)==&#39;Open&#39;,: )</pre><p>Or better yet, convert the whole column to categorical:</p><pre>mytable.Label = categorical( mytable.Label )<br>filteredtable = mytable( mytable.Label==&#39;Open&#39;,: )</pre><p>If you want to revert from categorical back to the cell string array you had before, you can use the <strong>cellstr() </strong>function:</p><pre>mytable.Label = cellstr( mytable.Label )</pre><p>You’d want to do this if you are planning to use contains — which only works with cell arrays and not categorical.</p><p><strong><em>Date</em></strong></p><p>Suppose you want to make a filtered table that contains only those rows of data that were collected in “Jan” (short for January). How to do it?</p><p>Here’s a slight exception to the rule above. If you use the MATLAB function <strong>contains, </strong>you can filter a cell array of character vectors. But this is slightly slower and it also works on patterns of text, so you should use it carefully.</p><p>Here’s how I would use it:</p><pre>filteredtable = mytable( contains(mytable.Date,&#39;Jan&#39;),: )</pre><p>Using <strong>contains</strong>, you can find all rows that match (even partially) any of a set of different items. So if you wanted everything in your table from Jan or Feb, you’d just do:</p><pre>filteredtable = mytable( contains(mytable.Date,{&#39;Jan&#39;,&#39;Feb&#39;}),: )</pre><p><strong><em>Tseries</em></strong></p><p>This is where the power of tables really comes through. You can filter your table based on data inside of a time series! Data that you cannot even see from the table. Here are a couple of ways you might choose to do that.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47a4644b8bedd08afa24b610a78d05c3/href">https://medium.com/media/47a4644b8bedd08afa24b610a78d05c3/href</a></iframe><p><strong><em>A combination of columns</em></strong></p><p>There is no straightforward way to do this. You can try chaining together a bunch of conditional statements but this can get ugly with more than a couple columns. What I recommend instead is combining your columns into a new column using <strong>rowfun(), </strong>which will take data from multiple columns in a single row and allow you to work with them at once.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9998f124bd8c5260caac45e8f064075/href">https://medium.com/media/b9998f124bd8c5260caac45e8f064075/href</a></iframe><p>Two important things to notice here.</p><p>First, you <strong>always need to take the first element of “text” data when you wish to use it</strong>, thus lbl{1} and date{1}. Text data in your table is, after all, a cell string. (Unless you’ve turned it into a categorical — but we won’t go there right now).</p><p>Second, a more subtle point is that when you use rowfun, every variable you are placing into rowfun must match exactly with the table you are inputting. So for example,</p><pre>rowfun(@(cellnumber) sprintf(&#39;%i&#39;,cellnumber), mytable ) </pre><p>This won’t work, because mytable has multiple columns and the rowfun — as it is set up to only take ‘cellnumber’, will throw an error. You must index your table’s columns for rowfun, unless you plan to put every column into your rowfun, like so:</p><pre>rowfun(@(cellnum,means,lbl,date,tseries) somefxn(), mytable )</pre><p>But this is rarely what you’ll want, so you almost always need to specify certain columns of:</p><pre>rowfun(@(lbl,date) somefxn(lbl,date), mytable(:,{&#39;Label&#39;,&#39;Date&#39;} )</pre><p>Also notice that the variables going into rowfun are not verbatim the names of the columns — they could be — but they don’t have to be. However, they do need to be in order. So for example,</p><pre>rowfun(@(date,lbl) somefxn(lbl,date), mytable(:,{&#39;Label&#39;,&#39;Date&#39;} )</pre><p>Would give you a result that won’t be an error but it may not be what you intend, since you are sending in the Label column and calling it “date” in rowfun. Definitely takes some practice, but once you have run the code to create the CombinedLabel column, you can search your table based on a variety of strings, for example:</p><pre>mytable( contains(mytable.CombinedLabel,&#39;Open&#39;), : )<br>mytable( contains(mytable.CombinedLabel,&#39;Jan1&#39;), : )<br>mytable( contains(mytable.CombinedLabel,&#39;Cell2&#39;), : )</pre><p>There are also uses for this if you want to combine two tables using a common key, an advanced (and valuable) use for tables.</p><blockquote><strong>Bonus Tip:</strong> Find groups in your data using <strong>findgroups</strong>() function</blockquote><p>While not specifically a table-related function, <strong>findgroups </strong>works really well with tables, especially when you want to do exploratory data analysis.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/257c0cf743ca9081f2cf957eb196000c/href">https://medium.com/media/257c0cf743ca9081f2cf957eb196000c/href</a></iframe><h3>Displaying your table data</h3><ol><li><strong>Imagesc </strong>(heatmap for time series)</li></ol><p>There are several massive advantages to having your data in a table. Here’s one of line of code that makes me the most grateful I learned tables:</p><pre>figure(); imagesc( cell2mat( mytable.Tseries ) )</pre><p>Why? Because it allows you to view subsets of your data, filtered or transformed in any way that you choose.</p><p>Here’s one way I’d use this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86e2210fbd3409f50c788f9ff89ae100/href">https://medium.com/media/86e2210fbd3409f50c788f9ff89ae100/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*wiO2JdraZOS3whmq7UNE3Q.png" /><figcaption>Ready for a figure panel!</figcaption></figure><p>No towering for loops, no arbitrary variables clogging up the workspace, and all in a couple of lines — thanks to tables.</p><p>2. <strong>Plot() </strong>(lines)</p><p>If you’re a neuroscientist, you’ve surely admired those cool Joy Division looking plots. With tables, it’s easy to make your own using the cell number and time series columns.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/225ae707ee16a68853c22cab079c5b46/href">https://medium.com/media/225ae707ee16a68853c22cab079c5b46/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*IKMvE1GdlKGw0Pcha-3zqg.png" /></figure><p>3. <strong>Heatmap() (</strong>heatmap for data other than time series)</p><p>This is a pretty finnicky function but it has some concrete uses. Use the command below to get a heatmap that shows the average value of table elements that correspond to those x,y values.</p><pre>figure(&#39;color&#39;,&#39;w&#39;); heatmap( mytable, &#39;Label&#39;, &#39;CellNumber&#39;, &#39;ColorVariable&#39;, &#39;Means&#39; )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*lrsG9eOchssUZOBPRAyfdw.png" /></figure><blockquote>Another use of heatmap is to place values at specific x,y locations.</blockquote><p>Suppose you want to examine some spatial dependency of a phenomenon (for example, the average activity of neurons at a specific location). Here’s some code and what your figure might look like.</p><p>Let’s generate some artificial data for that.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5bf475bf35581d0960a3f24bf675397/href">https://medium.com/media/c5bf475bf35581d0960a3f24bf675397/href</a></iframe><p>Now try to get this figure using the tip from above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*fk9TXxzt4GMIZquA4aJHjw.png" /></figure><p>Here’s how you would do it:</p><pre>heatmap( mytable2, &#39;x&#39;,&#39;y&#39;,&#39;colorvariable&#39;,&#39;value&#39; )</pre><blockquote>Bonus tip: There is no way to remove the text labels, unfortunately. Here’s a workaround that gets you a close approximation of what you see above:</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/437b60f0a5b5d22b3a3fde660575d3b2/href">https://medium.com/media/437b60f0a5b5d22b3a3fde660575d3b2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*rc5Pe5LGN72yW2r16UXclA.png" /></figure><p>To read about more advanced uses of tables, check out <a href="https://neuraljojo.medium.com/day-21-varfun-is-the-most-fun-way-to-use-tables-in-matlab-a6c468c8abd3">my post on varfun</a>, another one <a href="https://neuraljojo.medium.com/day-9-finding-fun-parallels-between-matlab-and-python-fcdf497e8a9c">about rowfun</a>, as well as a post on <a href="https://neuraljojo.medium.com/day-29-join-your-tables-to-make-a-sorted-boxplot-56a75cd79db0">how to join multiple tables</a>. Stay tuned for more posts from me about how to fully harness the potential of tables!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed5cfa876793" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bimodal or trimodal data? Try a Gaussian Mixture Model]]></title>
            <link>https://medium.com/analytics-vidhya/easily-cluster-and-display-data-using-a-gaussian-mixture-model-in-matlab-68bdbeb8cd3f?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/68bdbeb8cd3f</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[matlab]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[computational-biology]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Sun, 06 Feb 2022 21:13:52 GMT</pubDate>
            <atom:updated>2022-02-25T16:05:55.280Z</atom:updated>
            <content:encoded><![CDATA[<p>If you are using MATLAB to analyze data containing two or more normally distributed populations, this tutorial could help you sort them out</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*_6CS289E7f8tJpjETnwcaw.png" /><figcaption>Let’s hope that your data will cluster this cleanly!</figcaption></figure><p>Let’s begin by creating a hypothetical dataset containing data from three random distributions.</p><pre>rng(5) % Initialize the random number generator</pre><pre>mu = [5,10,15]; % Choose mean and standard deviation values<br>sigma = [.5,2,3];<br>Nsamples = [100,100,100]; % Note this %</pre><pre>data = cell2mat( arrayfun(@(i) ...<br>random(&#39;Normal&#39;,mu(i),sigma(i),Nsamples(i),1), [1:3],...<br>&#39;UniformOutput&#39;, false )&#39; );</pre><pre>figure; hist(data)</pre><p>Creating the figure will get you something that looks like this. It’s hard to tell how many Gaussians you put in there using the automatic binning by MATLAB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*9RnLAT7hpr0aWoYad_4mmQ.png" /></figure><p>We can use the AIC estimator to check the optimal number of Gaussians in your data. Of course, we know the answer is three for our toy dataset, but this won’t be the case out in the wild.</p><pre>rng(5)</pre><pre>Nguess = 5; % Guess the data has at most 5 Gaussians</pre><pre>options = statset(); options.MaxIter = 500;</pre><pre>% Fit Gaussian Mixture Models for each of the five models<br>gmmodels = arrayfun( @(x) fitgmdist( data, x, &#39;Options&#39;, options ),[1:Nguess],...<br>&#39;UniformOutput&#39;, false );</pre><pre>% Extract the AIC values for each model<br>AIC_values = cellfun( @(x) x.AIC, gmmodels );</pre><pre>% Plot AIC values to see the number that gives you the smallest AIC<br>figure; ax=axes(&#39;NextPlot&#39;,&#39;add&#39;); plot( ax, [1:5], AIC_values );[a,b] = min(AIC_values);<br>plot(b,a,&#39;ro&#39;)</pre><p>This will show you a shallow plateau with a minimum at 3, which is great because that’s how many we put into the artificial data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*3-fTTO-3FoAXjUa9-f7jOw.png" /></figure><p>Now, we’ll want to try and cluster the data based on the Gaussian Mixture Models we created above using <strong>arrayfun </strong>(if you need to brush up on your <strong>fun</strong>ctions check out my post on <a href="https://medium.com/analytics-vidhya/day-1-making-matlab-fun-ad850eaffbde">Making MATLAB “Fun”</a>).</p><pre>Idx = cluster(gmmodels{3},data);</pre><pre>% We will create artificially spaced bins<br>Nbins=100;<br>bins = linspace(0,max(data),Nbins);</pre><pre>% Create a Kernel Density Function for Each of the 3 Gaussians<br>% Note the multiplication by the normalization factor<br>% (sum(Idx==x))/numel(Idx) <br>% Entries roughly should match up with Nsamples/sum(Nsamples)<br>%<br>f = arrayfun( @(x) (sum(Idx==x)/numel(Idx))*ksdensity( data(Idx==x), bins, &#39;BandWidth&#39;, 1 ), [1:3], &#39;UniformOutput&#39;, false );</pre><pre>figure; ax=axes(&#39;nextplot&#39;,&#39;add&#39;);</pre><pre>arrayfun( @(f_) plot(bins,f_{1}), f );</pre><p>Now there are two areas here that you may need to adjust when using it with your own custom data.</p><p><strong>(1) Binning.</strong> You may want to try a more principled approach, such as the <a href="https://en.wikipedia.org/wiki/Freedman%E2%80%93Diaconis_rule">Freedman-Diaconis rule</a> (it’s very straightforward).</p><p><strong>(2) Bandwidth. </strong>The parameter within the KS density function is arbitrarily set to 1. This is something you can play around with, or check out this <a href="https://en.wikipedia.org/wiki/Kernel_density_estimation#A_rule-of-thumb_bandwidth_estimator">formula</a>.</p><p>Once you display the result, it should look pretty crisp and a huge improvement visually from the histogram we started out with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*qbbpnziamKBaCZBk-VdKMA.png" /></figure><p>Lastly, you’ll want to get that beautiful aesthetic that’s possible through using the legendary <strong>patch </strong>function (more <strong>patch </strong>magic <a href="https://neuraljojo.medium.com/day-28-accentuate-your-data-using-best-fit-lines-and-shading-4d6994e72425">in this post</a>).</p><pre>% I use two anonymous functions to create the x and y input to patch<br>% This way the Kernel Density Function is generated new for each<br>% of the three Gaussians<br>patchup_ksf_y = @(ksf) [ksf,zeros(1,numel(ksf)),ksf(1)]&#39;;<br>patchup_ksf_x = @(ksf) [bins,fliplr(bins),bins(1)]&#39;;</pre><pre>% Create a figure without lines or axes<br>% You can edit here if you wish to preserve any of the plot pieces<br>figure(&#39;color&#39;,&#39;w&#39;); ax=axes(&#39;nextplot&#39;,&#39;add&#39;,&#39;xcolor&#39;,&#39;w&#39;,&#39;ycolor&#39;,&#39;w&#39;,&#39;box&#39;,&#39;off&#39;);</pre><pre>% Create an array that contains three patches<br>patches = cellfun( @(ksf) patch( patchup_ksf_x(ksf),patchup_ksf_y(ksf),...<br>&#39;k&#39;,&#39;EdgeAlpha&#39;,0,&#39;facealpha&#39;,0.5,&#39;parent&#39;,ax ), f )</pre><pre>% A custom palette for the display<br>palette=lines(3);</pre><pre>% Re-coloring the patches<br>arrayfun( @(i) set(patches(i),&#39;facecolor&#39;,palette(i,:)), [1:3] )</pre><p>Note that this isn’t the same as fitting a Gaussian to your data. Here’s how you would do that.</p><pre>gm = arrayfun( @(x) gmdistribution( gmmodels{3}.mu(x), gmmodels{3}.Sigma(x) ), [1:3], &#39;UniformOutput&#39;, false )</pre><pre>figure(&#39;color&#39;,&#39;w&#39;);<br>ax = axes(&#39;nextplot&#39;,&#39;add&#39;,&#39;xlim&#39;,[min(bins),max(bins)]);</pre><pre>arrayfun( @(x) plot( bins, (sum(Idx==x)/numel(Idx))*pdf( gm{x}, bins&#39; ) ), [1:3] )</pre><p>Let’s take a look at the data, the Kernel Density plot, and the Gaussian fits together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0szsEcMR_d006ToOF0q6fA.png" /><figcaption>The raw data (left) is clustered using the <strong>cluster </strong>function and a Gaussian Mixture Model — giving us three colors. These data can then be smoothed over using <strong>ksdensity</strong>, or alternately, three Gaussian <strong>pdf</strong>s<strong>.</strong></figcaption></figure><p>That’s all it takes to produce a nice graphic showing the diversity of your dataset. For another use of the <strong>ksdensity </strong>function, check out my post on <a href="https://neuraljojo.medium.com/use-matlab-to-create-beautiful-custom-violin-plots-c972358fa97a">creating a half-violin plot in MATLAB</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68bdbeb8cd3f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/easily-cluster-and-display-data-using-a-gaussian-mixture-model-in-matlab-68bdbeb8cd3f">Bimodal or trimodal data? Try a Gaussian Mixture Model</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use MATLAB to create beautiful, custom violin plots]]></title>
            <link>https://medium.com/analytics-vidhya/use-matlab-to-create-beautiful-custom-violin-plots-c972358fa97a?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c972358fa97a</guid>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[matlab]]></category>
            <category><![CDATA[biology]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Thu, 03 Feb 2022 20:07:32 GMT</pubDate>
            <atom:updated>2022-02-25T23:59:50.028Z</atom:updated>
            <content:encoded><![CDATA[<h4>One of the most desired plot types, a favorite of R users, is the violin or half-violin. In this post I show how to accomplish this in MATLAB using two built-in functions: <strong>ksdensity</strong> and <strong>patch</strong>.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*50PiklSxxrSs8zAKCP1xgw.png" /><figcaption>After reading this post, you will be able to understand the magic behind these graphs and make one yourself!</figcaption></figure><h4>Has this ever happened to you?</h4><p>You have some great data for a boxplot, but you also want to view the distributions. Your supervisor suggests something called a violin, but you’re a MATLAB native and the last time you saw a violin was in your high school orchestra section. So you go looking:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5a5WgbnpsxCMWbp6iLuzag.png" /></figure><p>There are some homemade solutions on the MathWorks page, but in general they don’t have the same aesthetic as what you’d see from an R package.</p><h3>Here I will guide you through making a very basic half-violin, and a more complex example follows</h3><p><strong>Step 1:</strong> Create some fake data. We’ll create data that is a combination of two Gaussian distributions, with means at 0 and 10, and standard deviations of 5 and 1.</p><pre>rng(5)</pre><pre>mu = [0,10];<br>sigma = [5,0;0,1];<br>gm = gmdistribution( mu,sigma );<br>data = random(gm,30);<br>data = data(:);</pre><p><strong>Step 2:</strong> Create a kernel density function from your data, which is at the heart of those lovely half-violins.</p><pre>xvalues = linspace( prctile(data,1), prctile(data,99), 100 );</pre><pre>[f,xi] = ksdensity( data(:),xvalues,&#39;Bandwidth&#39;,1,&#39;BoundaryCorrection&#39;,&#39;reflection&#39;);</pre><p><strong>Step 3: </strong>Use <strong>patch</strong> for displaying it, and viola (or should we say, violin!)</p><pre>figure; patch( 0-[f,zeros(1,numel(xi),1),0],[xi,fliplr(xi),xi(1)],&#39;r&#39; )</pre><p><strong>Tip: If your graph looks like it’s filling up the whole plot box, try adjusting the value of ‘Bandwidth’ in the ksdensity command.</strong></p><p>By resizing the window, you can actually configure how the plot looks (which is one of the biggest advantages of plotting with MATLAB over R or Jupyter, in my opinion).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*QDQ6dpNVDYcAy-ZYwYIL0w.png" /></figure><p>Now you understand the basics of plotting the violin using <strong>ksdensity</strong> and <strong>patch. </strong>In the gist below, I lay out how to generate the graph in the top panel. If you want to fully understand all the code I recommend you check out my other posts about <a href="https://medium.com/analytics-vidhya/day-7-where-does-my-data-go-and-matlabs-most-useful-hidden-plotting-property-d00d9d12045e">how to grab objects in a figure</a>, as well as <a href="https://medium.com/analytics-vidhya/day-1-making-matlab-fun-ad850eaffbde">arrayfun</a> and <a href="https://neuraljojo.medium.com/day-18-an-underutilized-matlab-trick-to-help-you-patch-up-your-plots-7ae145a440e9">patch</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/320be541e00db04b6f83b9f5cf4962b9/href">https://medium.com/media/320be541e00db04b6f83b9f5cf4962b9/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c972358fa97a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/use-matlab-to-create-beautiful-custom-violin-plots-c972358fa97a">Use MATLAB to create beautiful, custom violin plots</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to make an after-hours market scanner with Selenium and Yahoo]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-make-a-pre-and-post-market-scanner-with-selenium-and-yahoo-b9182c80bbc3?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9182c80bbc3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[selenium]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Wed, 02 Feb 2022 19:42:28 GMT</pubDate>
            <atom:updated>2022-02-02T19:43:42.808Z</atom:updated>
            <content:encoded><![CDATA[<p>This article describes how Selenium in Python can be used to easily obtain pre- and post-market quotes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BUKkPCm_cHCKluPAxv1WOA.jpeg" /><figcaption>Selenium is not only essential to a balanced diet, it should also be part of every data scientist’s toolkit.</figcaption></figure><p>These days, keeping track of pre- and post-market prices can mean the difference between an appreciable gain and a fantastic one. For example, when AMD announced their earnings on February 1st, the post-market price reached above $135, but opened at $129.89 the next day.</p><p>In this article, I will go step by step through a Python notebook that automates the extraction of pre- and post-market stock quotes. Stay tuned for a future Medium Post from me that combines this pre-/post-market scanner with e-mail notifications and a visual dashboard.</p><blockquote>This post will show you how to <strong>retrieve stock market data from before or after hours, storing the information locally and watching trends as they appear.</strong></blockquote><p><em>Below is all the code you need to get this done from within a Jupyter Notebook on your local computer. I have not tried this on any cloud-based notebook services.</em></p><h3>As with most Python projects, the most difficult piece will be making sure you have all the packages</h3><p>Here is what you’ll need to be able to do without errors:</p><pre>from selenium import webdriver<br>from webdriver_manager.chrome import ChromeDriverManager<br>from selenium.webdriver.support.ui import WebDriverWait<br>from selenium.webdriver.chrome.options import Options<br>from selenium.webdriver.common.by import By</pre><pre>driver = webdriver.Chrome(ChromeDriverManager().install())</pre><p>Getting a warning about versions is fine, but no error message. If you have issues here, you should look around on online and do the necessary pip install and so on to get to the right place.</p><h3>If you are already running Selenium without issue, the rest is three easy steps.</h3><h4>I’ll lead you through how I came up with the code (so you can customize) or just skip to the bottom for all the code so you can run it immediately.</h4><p>With any web scraping project, you’ll want to find out how you can locate the elements you want to extract. I am breaking this down into three steps (get the generic container, hone in on the target text, output price changes to a dataframe).</p><p>0. Send the Selenium driver to the page containing your ticker of interest. (This is not really a step!)</p><pre>ticker=&#39;TSLA&#39;<br>driver.get(f&#39;<a href="https://finance.yahoo.com/quote/%7Bticker%7D&#39;">https://finance.yahoo.com/quote/{ticker}&#39;</a>)</pre><ol><li>Locate the category of thing that contains the pre-market price.</li></ol><pre># Find all of the fin-streamer elements on the page (there are many)<br>fs = driver.find_elements( By.TAG_NAME , &#39;fin-streamer&#39; )</pre><p>The output of this (<strong>fs</strong>), is a list. Each item within that list is a webdriver element.</p><blockquote>By is an incredibly useful module of Selenium. Unfortunately, there isn’t a lot of documentation about it so if you want to see what else it can do I recommend typing <strong>help(By) </strong>and running it. I may decide to write a tutorial at some point about it!</blockquote><p>2. Hone in on the exact field by finding something specific to the pre-market price, in this case, where ‘data-field’ is equal to ‘preMarketPrice’ or ‘postMarketPrice’. Using the same approach, we’ll extract the price change field.</p><pre>price_field = [ c for c,x in enumerate(fs) if x.get_attribute(&#39;data-field&#39;)==&#39;preMarketPrice&#39; or x.get_attribute(&#39;data-field&#39;)==&#39;postMarketPrice&#39; ]<br><br>change_field = [ c for c,x in enumerate(fs) if x.get_attribute(&#39;data-field&#39;)==&#39;preMarketChangePercent&#39; or <br>x.get_attribute(&#39;data-field&#39;)==&#39;postMarketChangePercent&#39; ]</pre><p>I am using a <strong>list comprehension</strong> and <strong>enumerate </strong>to get the numerical index for the list item corresponding to the price (either pre- or post-market, it will always be one or the other).</p><p>3. I use a little trick to only log when the price changes. A lot of people may already have a solution for this that’s more sophisticated (like an SQL database). Here is what I do to save all the changes in a list called values_list:</p><pre>from datetime import datetime<br>import time</pre><pre>price0,change0=[],[],[]<br>values_list = list()</pre><pre>while 1:<br>    fs = driver.find_elements( By.TAG_NAME , &#39;fin-streamer&#39; )</pre><pre>    price1  = fs[price_field[0]].text<br>    change1 = fs[change_field[0]].text<br>    if price0!=price1: # Detects a change in price<br>        localtime = datetime.now().strftime(&#39;%H:%M:%S %m-%d-%y&#39;)<br>        values_list.append( [localtime,price1,change1] )<br>    price0,change0 = price1,change1<br>    time.sleep(1)</pre><p>Afterwards, <strong>values_list</strong> can be converted to a dataframe using Pandas, written to a text file, used to create an alert, or show up in a graph.</p><p>Good luck to all scalping those earnings pops in the after hours!</p><h3>I already know Selenium, just give me the code</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/491276993ecf34ef58256410b23dee0e/href">https://medium.com/media/491276993ecf34ef58256410b23dee0e/href</a></iframe><h4>Here’s an advanced version that uses dictionaries so you can track multiple tickers simultaneously, and also operates without opening windows.</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3931cc56910d9426c044a5fe5e859817/href">https://medium.com/media/3931cc56910d9426c044a5fe5e859817/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9182c80bbc3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-make-a-pre-and-post-market-scanner-with-selenium-and-yahoo-b9182c80bbc3">How to make an after-hours market scanner with Selenium and Yahoo</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yet Another Way Colormaps Can Be Used to Enhance Your Data Visualization]]></title>
            <link>https://medium.com/swlh/yet-another-way-colormaps-can-be-used-to-enhance-your-data-visualization-2bc1863654b5?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bc1863654b5</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[matlab]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Thu, 18 Feb 2021 18:38:05 GMT</pubDate>
            <atom:updated>2021-02-18T22:11:29.031Z</atom:updated>
            <content:encoded><![CDATA[<p>For most scientists, there is a ton of utility in switching between colormaps. Here I describe a less commonly-used method to get even more from your colormap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pet8MZLHKxRgnBZ7ezBYYQ.png" /></figure><p>If you’re like me and you learn by reading code, feel free to skip to the end for a code snippet from MATLAB.</p><h3>When would you want to do this?</h3><p>You have data you want to display, but most of the values are much smaller than the overall range of values. This can give you the sense that there’s “nothing there” in your image when actually there may be interesting variability that you’re not seeing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*poH2G7jzRskWNxvXu10qqA.png" /><figcaption>The before image, with the colorbar shown to the right.</figcaption></figure><p>In many cases, people will go to their image adjustment program of choice (ImageJ, Photoshop, etc) and crank up the contrast. This is of course not efficient or good for reproducibility, since people rarely keep track of the exact adjustments they make. And if you have any text on your image, you’ll be adjusting the contrast of that too.</p><blockquote>For reproducibility and efficiency, use code for all your data visualization and adjustments (as opposed to manual tweaking in GUIs like ImageJ).</blockquote><p>Below I’ll explore two alternative solutions and compare them.</p><h3>Convert your data to a <strong>log scale</strong></h3><p>This is one approach that is often taken and <em>does not rely on changing your colormap</em>, and it can also help you emphasize variability at the lower end of your data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EX1mS2MnwQkZq7EYih290w.png" /><figcaption>The before image from above, with the logarithm values of the data displayed.</figcaption></figure><p>There is a lot more visible here, but there are several downsides to this approach. Here are some:</p><ul><li><strong>Not intuitive. </strong>Your data is now in log units, which is not intuitive for 99% of people, especially non-scientists.</li><li><strong>Lost values. </strong>If your data contains negative values, you will lose these values since negative numbers do not have real logarithms.</li><li><strong>Hard to tweak the amount of contrast</strong> you’re gaining. You could do it by changing the base of your logarithm, but who wants to do that?</li></ul><h3>Make your colormap non-linear</h3><p>To understand this approach, you first have to de-mystify colormaps. I’m going to explain these concepts using MATLAB but it’s in general true.</p><p>A colormap is basically just a list of colors, in the example below, they go from color 1 to color 50. The dimmer objects in your image get assigned lower colors and the brighter objects get higher colors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dPuPZPjgAWJNR-drybZmQg.png" /></figure><p>Now imagine picking a subset of the colors in a precise way, where you’d want dim objects to range from blue to green to yellow, so you can see the differences in their intensity.</p><p>Here’s what you might do with your colormap:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mX0IBP796qD7HnPYNsss9g.png" /></figure><p>And now if you looked at the same image from above with your new “hand-picked” colormap, it would look more like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NLn8aF2pfVsrTEeN6I9ezQ.png" /></figure><h3>Am I really going to pick those colors every time I make a figure?</h3><p>No. That’s what math and code are for!</p><ol><li>Pick a color map — this means a palette and also how many colors it contains (pick a large number — much larger than the number of different objects you want to be able to see — I usually go with 1000).</li><li>Pick a mathematical function (for example, an exponential) and obtain its output values f(x) over a range of x’s large enough to show the behavior you want.</li><li>Rescale the outputs into integers from 1 to the number of colors in your color map.</li><li>Use the rescaled outputs to specify a <em>new </em>colormap that is a subset of the original.</li></ol><h3>Here is how I would implement this in MATLAB</h3><p>The code below would easily be adapted to any coding language, the main idea is just reindexing your colormaps using something other than a linear progression.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/738838b95c02e8ab1e939a4ee9d3e4b6/href">https://medium.com/media/738838b95c02e8ab1e939a4ee9d3e4b6/href</a></iframe><p>You can see the result below, with images and colormaps above the function used to reindex the colormap. You’ll notice that the colorbars on the right side are all the same, which is because you have not changed your data — only the way it’s displayed!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vf5afVXdTOR4x0aZFtFy5w.png" /></figure><p>The nice thing is that you can get creative and change the shape of the function however you want. You could even use a sine wave if you wanted to!</p><p><em>Please feel free to @ or DM me on Twitter (neurojojo) with any cool images you generate using this approach, or questions you have!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bc1863654b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/yet-another-way-colormaps-can-be-used-to-enhance-your-data-visualization-2bc1863654b5">Yet Another Way Colormaps Can Be Used to Enhance Your Data Visualization</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 29: Join your tables to make a sorted boxplot]]></title>
            <link>https://neuraljojo.medium.com/day-29-join-your-tables-to-make-a-sorted-boxplot-56a75cd79db0?source=rss-71417ba6ebf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/56a75cd79db0</guid>
            <category><![CDATA[datatables]]></category>
            <category><![CDATA[matlab]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[sociology]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jozsef Meszaros]]></dc:creator>
            <pubDate>Fri, 29 May 2020 23:40:09 GMT</pubDate>
            <atom:updated>2020-05-29T23:40:09.761Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sK4bge72n2KJdw8AVOPuDw.png" /></figure><p>Joining tables is an art and the benefits and consequences of joining are not always be clear. Here, I’ll describe a simple case that works well even for a beginner learning to use tables.</p><h3>Joining for enhanced boxplots</h3><p>Here, I’ll use the generic <em>join </em>to spruce up a boxplot and accentuate a pattern in the data. First, load up some data from MATLAB’s standard library.</p><pre>load census1994<br><br>adultdata_subset = adultdata( find(adultdata.capital_gain&gt;0),:);</pre><p>I created a subset of the data that only contains entries where individuals reported any capital gain at all. Next, graph it sideways with some data limits imposed and clipping the outliers.</p><pre>figure(&#39;color&#39;,&#39;w&#39;); boxplot( adultdata_subset.capital_gain, adultdata_subset.race, &#39;datalim&#39;, [0,30000], &#39;extrememode&#39;, &#39;clip&#39; )<br>ylabel(&#39;Capital gain ($)&#39;)<br>camroll(-90)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*4enMoa8xXaCxqWIxr44BKA.png" /></figure><p>There’s clearly a disparity in the capital gain amount based on race, but we can really make this result appear if we sorted the boxes.</p><h4>Ordering a boxplot</h4><p>To improve the presentation of this data, let’s try to show the boxplot with the lowest medians near the top and increasing medians as you go down.</p><p>This requires five easy steps:</p><ol><li>Create a separate table <strong>newtable </strong>using <em>varfun </em>that will contain the median value for each race.</li><li>Sort <strong>newtable </strong>by the median value, and store the sorting order in a separate column.</li><li>Create a ranking of the sorting order by using the sort-twice trick.</li><li>Innerjoin the original data table with <strong>newtable</strong>.</li><li>Plot and add the sorted column of ‘race’ as x-tick labels.</li></ol><p>Here’s the full process:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a9b176433f07138ae79d56a773f31fc/href">https://medium.com/media/0a9b176433f07138ae79d56a773f31fc/href</a></iframe><p>The <em>join </em>function basically distributes the values from <strong>ranking_table </strong>into <strong>adultdata_subset</strong> to matching entries of the “race” variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ghRqD0MS9ljgwdMGkUzxeg.png" /></figure><p>By appending the “rank” column, we are able to, in essence, create a new grouping for our data and MATLAB’s <em>boxplot </em>function leverages that to create an ordered display.</p><h4>When things get complicated</h4><p>Often tables are joined which have different columns, unmatched rows, and other idiosyncrasies. Getting good at joining tables can take weeks to months and there are dozens of ways to use the additional <em>innerjoin </em>and <em>outerjoin </em>functions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56a75cd79db0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>